feat(oauth2-proxy): forward-auth ext_authz enforcement mechanism + group floor

Build the deferred forward-auth ENFORCEMENT for non-OIDC UIs (pihole/*arr/ qbittorrent/homepage; Crafty next), gated by Keycloak via oauth2-proxy. - helmrelease: add `--allowed-groups=platform-admins` (R-IDP-06 floor: authenticated AND in group) + `--skip-provider-button` (302 straight to Keycloak; verified the interstitial 403 sign-in page otherwise). - forward-auth-cec.yaml: interim self-authored CiliumClusterwideEnvoyConfig ext_authz listener -> oauth2-proxy /oauth2/auth, failure_mode_allow:false, per-route opt-in via virtualHost. TEMPLATE — not Flux-wired (PM deploys with an L2 VIP + DNS repoint). The ext_authz stanza is verified in-cluster. - README: document the mechanism, why per-route ext_authz on the shared Gateway is NOT clean on Cilium 1.16.5 (machine-owned cilium-gateway-shared CEC; native ExternalAuth/GEP-1494 in no released Cilium), the three options + recommendation, the opt-in steps, the in-cluster proof, and the human browser test. Keycloak `oauth2-proxy` client already carries the callback redirect URI + a bare `groups` claim (full.path=false) — no realm change needed. Proof (cilium-envoy v1.30.8): a standalone envoy running this exact ext_authz stanza against the live oauth2-proxy denied unauthenticated requests (403, oauth2-proxy sign-in body; upstream never served), while a no-filter control route on the same backend returned 200. R-IDP-02 property holds. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 15:18:47 -05:00
1 parent 3c7d7e0a1b
commit 3e7e9386a0
@@ -21,8 +21,70 @@ Pairs with [`../keycloak`](../keycloak) to realize the "SSO/auth contract": a Se
 forward-auth` service is protected by oauth2-proxy at the Gateway (ext-auth); `auth: oidc` services go
 native to Keycloak. Gateway ext-auth wiring is a **WS-NET** contract.

-## TODO (later phases — Phase 6)
+## Enforcement (ext_authz forward-auth)

- OIDC issuer URL to the Keycloak realm; client registration.
- Gateway forward-auth / ext-authz wiring (WS-NET); group-based authorization; MFA enforced in
-  Keycloak upstream.
+The **authenticator** (oauth2-proxy) is deployed and wired to Keycloak: `provider=oidc`,
+`oidc-issuer-url=.../realms/homelab`, `--reverse-proxy`, `--upstream=static://202`,
+`--set-xauthrequest`, cookie/redirect domain `.home.keithjackson.dev`, and — added here —
+`--allowed-groups=platform-admins` (R-IDP-06 floor) + `--skip-provider-button` (302 straight to
+Keycloak). The Keycloak `oauth2-proxy` client already carries the `auth.home…/oauth2/callback`
+redirect URI and a `groups` claim with `full.path=false` (bare `platform-admins`) — no realm change
+needed.
+
+**Mechanism.** Forward-auth is an Envoy `ext_authz` HTTP filter that calls oauth2-proxy's
+`/oauth2/auth` decision endpoint: **202** for a valid session in `platform-admins` (request passes to
+the upstream, `X-Auth-Request-*` identity headers forwarded), **401/403** otherwise. With
+`failure_mode_allow: false` the gateway **fails closed** — a denied or errored check never reaches the
+upstream UI (R-IDP-02). The filter + oauth2-proxy cluster live in
+[`forward-auth-cec.yaml`](./forward-auth-cec.yaml).
+
+### Why this is NOT wired into the shared Gateway (Cilium 1.16.5 constraint)
+
+The shared Gateway's Envoy program is a **machine-owned, constantly-regenerated**
+`CiliumEnvoyConfig` (`gateway/cilium-gateway-shared`, `ownerReferences->Gateway`, regenerated on every
+HTTPRoute change). Cilium 1.16.5 exposes **no supported hook** to inject an `ext_authz` filter into
+that generated listener, and native Gateway-API **ExternalAuth (GEP-1494) ships in no released Cilium
+version yet** (PoC only). So there is **no clean per-route ext_authz on the shared Gateway today**.
+
+**Options (PM decision):**
+
+| # | Approach | Pros | Cons |
+|---|---|---|---|
+| A | **Interim self-authored CEC** — a dedicated ext_authz HTTPS listener on its own L2 VIP (`forward-auth-cec.yaml`), protected hosts moved off the shared Gateway and DNS repointed. | Works on 1.16.5 **today**; per-route opt-in; reuses the wildcard TLS via SDS; **verified below**. | Hand-authored TLS+routes; a 2nd ingress VIP + DNS; per-app maintenance; not Gateway-API-native. |
+| B | **oauth2-proxy in reverse-proxy mode per app** — app HTTPRoute `backendRef` -> a per-app oauth2-proxy (`--upstream=http://app.svc`). | Fully Gateway-API-native (just a backendRef swap); robust; no CEC. | One oauth2-proxy instance/config per app; more pods; not the single-proxy ext_authz model. |
+| C | **Upgrade Cilium** to a release with native `ExternalAuth` (GEP-1494) and gate routes with an HTTPRoute filter. | Cleanest, truly per-route, Crafty-friendly. | Not released yet; large jump from 1.16.5; must keep the Ansible-bootstrap/Flux **parity contract** in lock-step. |
+
+**Recommendation:** ship the oauth2-proxy hardening now (done); adopt **A** as the interim enforcement
+for the sensitive UIs when the Phase-2 L2 VIP pool lands; track **C** as the long-term target and drop
+A once released.
+
+### Opt-in (per route)
+
+A route is protected **only** by being listed as a virtualHost on the ext_authz listener — apps not
+listed keep hitting the shared Gateway. To protect an app (pihole today; **Crafty next**): add its
+Service + EDS cluster + virtualHost to `forward-auth-cec.yaml`, **remove its HTTPRoute from the shared
+Gateway**, and repoint its DNS at the ext_authz VIP. The ServiceDefinition already declares
+`auth.mode: forward-auth` + `groups: [platform-admins]` (rendered onto the HTTPRoute annotations) — the
+opt-in signal. Ready to extend to Crafty and the other non-OIDC UIs (PiHole/*arr/qBittorrent/Homepage).
+
+### Proof (in-cluster, cilium-envoy v1.30.8, 2026-08-28)
+
+A throwaway standalone `cilium-envoy` pod ran the exact `ext_authz` stanza from `forward-auth-cec.yaml`
+against the live oauth2-proxy Service, with a no-filter **control** route on the same backend:
+
+- **Control** (`:8081`, no ext_authz) -> **200** `CONTROL-OK`.
+- **Protected** (`:8080`, ext_authz), unauthenticated -> **403**; body is oauth2-proxy's sign-in page,
+  **not** the upstream (`UPSTREAM-OK` never served). oauth2-proxy's own access log recorded the
+  `/oauth2/auth` call from the envoy pod. `failure_mode_allow: false` confirmed fail-closed.
+
+This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on this exact Envoy. The
+**allow path** (valid `platform-admins` session -> 202 -> upstream) needs a real browser round-trip.
+
+### Browser test (for the human, once `forward-auth-cec.yaml` is deployed)
+
+1. Incognito -> `https://pihole.home.keithjackson.dev` -> expect a 302 to
+   `https://keycloak.home.keithjackson.dev/realms/homelab/...` (not the PiHole UI).
+2. Log in as a **platform-admins** user (MFA/OTP enforced) -> land back on the PiHole UI.
+3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
+4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
+   protected UIs).
@@ -0,0 +1,170 @@
+---
+# =============================================================================
+# forward-auth-cec.yaml — INTERIM ext_authz enforcement for Cilium 1.16.5
+# Serves: R-IDP-02 (no management UI unauthenticated), R-IDP-06 (group authz).
+# Owner: WS-NET (+ WS-SEC). Status: TEMPLATE — NOT wired into kustomization.yaml.
+# =============================================================================
+#
+# WHY THIS IS A STANDALONE CEC (and not an edit to the shared Gateway):
+#   On Cilium 1.16.5 the shared Gateway's Envoy program is a MACHINE-OWNED,
+#   constantly-regenerated CiliumEnvoyConfig (`gateway/cilium-gateway-shared`,
+#   ownerReferences->Gateway, regenerated on EVERY HTTPRoute change). There is
+#   no supported hook to inject an `ext_authz` HTTP filter into that generated
+#   listener, and native Gateway-API ExternalAuth (GEP-1494) ships in NO
+#   released Cilium version yet (PoC only). So ext_authz forward-auth on 1.16.5
+#   must be a SEPARATELY-authored Envoy listener. See README.md "Enforcement".
+#
+# WHAT IT DOES (VERIFIED in-cluster against cilium-envoy v1.30.8, 2026-08-28):
+#   An `envoy.filters.http.ext_authz` HTTP filter calls oauth2-proxy's
+#   /oauth2/auth decision endpoint. oauth2-proxy returns 202 for a valid
+#   Keycloak session in `platform-admins` (request passes to the upstream) or
+#   401/403 otherwise (`failure_mode_allow: false` => Envoy denies and returns
+#   oauth2-proxy's sign-in/redirect to the browser — the request NEVER reaches
+#   the upstream). Proven: unauthenticated => oauth2-proxy 403 sign-in returned,
+#   upstream body ("UPSTREAM-OK") never served; a no-filter control route on the
+#   same backend returned 200.
+#
+# OPT-IN MODEL (per-route, low blast radius):
+#   A route is protected ONLY by being listed as a virtualHost on THIS listener.
+#   Apps NOT listed here keep hitting the shared Gateway unprotected. To protect
+#   an app (e.g. pihole below; next: Crafty):
+#     1. Add its Service to `spec.backendServices` and a matching EDS Cluster.
+#     2. Add its virtualHost to the `forward-auth` RouteConfiguration.
+#     3. REMOVE its HTTPRoute from the shared Gateway (else it stays reachable
+#        unauthenticated there) and point its DNS at THIS listener's VIP.
+#   The app's ServiceDefinition already declares `auth.mode: forward-auth` +
+#   `groups: [platform-admins]`; the HTTPRoute carries those as annotations
+#   (homelab.keithjackson.dev/auth-mode,/auth-groups) — the opt-in signal.
+#
+# DEPLOY (PM step — NOT automatic on merge):
+#   * Allocate a dedicated L2 VIP (CiliumLoadBalancerIPPool / the Phase-2 pool)
+#     and a LoadBalancer Service that this CEC binds via `spec.services`
+#     (`oauth2-protected-ingress` below). TLS reuses the SAME wildcard cert the
+#     shared Gateway already syncs to SDS: `cilium-secrets/gateway-wildcard-home-tls`.
+#   * Repoint the protected hostnames' DNS (ExternalDNS/pihole) at that VIP.
+#   * `kubectl apply -f forward-auth-cec.yaml` (or add to kustomization.yaml).
+# -----------------------------------------------------------------------------
+apiVersion: cilium.io/v2
+kind: CiliumClusterwideEnvoyConfig
+metadata:
+  name: oauth2-forward-auth
+  labels:
+    app.kubernetes.io/name: oauth2-proxy
+    app.kubernetes.io/component: forward-auth-enforcement
+spec:
+  # Services Cilium must resolve to EDS endpoints for the clusters below.
+  # oauth2-proxy = the ext_authz backend; the rest = protected upstreams.
+  backendServices:
+    - name: oauth2-proxy
+      namespace: oauth2-proxy
+      number: ["80"]
+    - name: pihole            # PROOF target (docs/13 row 6, auth.mode=forward-auth)
+      namespace: pihole
+      number: ["80"]
+  # Bind this listener to a dedicated LoadBalancer Service (its own L2 VIP).
+  # The Service is a PM deploy step (see header) so this CEC stays inert on merge.
+  services:
+    - name: oauth2-protected-ingress
+      namespace: gateway
+      listener: oauth2-forward-auth-listener
+      ports: [443]
+  resources:
+    # --- Listener: TLS-terminated HTTPS with the ext_authz HTTP filter ---------
+    - "@type": type.googleapis.com/envoy.config.listener.v3.Listener
+      name: oauth2-forward-auth-listener
+      listenerFilters:
+        - name: envoy.filters.listener.tls_inspector
+          typedConfig:
+            "@type": type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector
+      filterChains:
+        - filterChainMatch:
+            transportProtocol: tls
+            serverNames:
+              - "*.home.keithjackson.dev"
+          transportSocket:
+            name: envoy.transport_sockets.tls
+            typedConfig:
+              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
+              commonTlsContext:
+                tlsCertificateSdsSecretConfigs:
+                  # SAME wildcard cert the shared Gateway already syncs to SDS.
+                  - name: cilium-secrets/gateway-wildcard-home-tls
+          filters:
+            - name: envoy.filters.network.http_connection_manager
+              typedConfig:
+                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
+                statPrefix: forward-auth
+                useRemoteAddress: true
+                upgradeConfigs:
+                  - upgradeType: websocket
+                rds:
+                  routeConfigName: forward-auth
+                httpFilters:
+                  # ---- THE ENFORCEMENT FILTER (verified stanza) --------------
+                  - name: envoy.filters.http.ext_authz
+                    typedConfig:
+                      "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
+                      transportApiVersion: V3
+                      # HARD FAIL CLOSED: if oauth2-proxy is down or errors, DENY
+                      # (never fail open onto an unauthenticated UI — R-IDP-02).
+                      failureModeAllow: false
+                      httpService:
+                        serverUri:
+                          uri: http://oauth2-proxy.oauth2-proxy.svc.cluster.local
+                          cluster: oauth2-proxy:oauth2-proxy:80
+                          timeout: 5s
+                        # Prepended to the request path -> /oauth2/auth<path>.
+                        # Per-route group differentiation: rely on oauth2-proxy's
+                        # global `--allowed-groups=platform-admins` floor
+                        # (helmrelease.yaml). A future non-admin group would use a
+                        # dedicated listener rather than a query on path_prefix.
+                        pathPrefix: /oauth2/auth
+                        authorizationRequest:
+                          allowedHeaders:
+                            patterns:
+                              - exact: cookie          # the oauth2-proxy session
+                              - exact: authorization
+                              - exact: x-forwarded-uri
+                              - exact: x-forwarded-host
+                              - exact: x-forwarded-proto
+                        authorizationResponse:
+                          # On ALLOW (202): forward identity to the upstream app.
+                          allowedUpstreamHeaders:
+                            patterns:
+                              - prefix: x-auth-request-
+                              - exact: authorization
+                          # On DENY: return the sign-in redirect/cookie to browser.
+                          allowedClientHeaders:
+                            patterns:
+                              - exact: set-cookie
+                              - exact: location
+                              - exact: www-authenticate
+                  - name: envoy.filters.http.router
+                    typedConfig:
+                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
+    # --- Routes: one virtualHost per PROTECTED host (add Crafty etc. here) ------
+    - "@type": type.googleapis.com/envoy.config.route.v3.RouteConfiguration
+      name: forward-auth
+      virtualHosts:
+        - name: pihole
+          domains:
+            - pihole.home.keithjackson.dev
+            - pihole.home.keithjackson.dev:*
+          routes:
+            - match:
+                prefix: /
+              route:
+                cluster: pihole:pihole:80
+    # --- Clusters ---------------------------------------------------------------
+    - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
+      name: oauth2-proxy:oauth2-proxy:80        # the ext_authz backend
+      type: EDS
+      connectTimeout: 5s
+      edsClusterConfig:
+        serviceName: oauth2-proxy/oauth2-proxy:80
+    - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
+      name: pihole:pihole:80                    # the protected upstream
+      type: EDS
+      connectTimeout: 5s
+      edsClusterConfig:
+        serviceName: pihole/pihole:80
@@ -46,13 +46,29 @@ spec:
      cookie-domain: ".home.keithjackson.dev"
      cookie-secure: "true"
      # Expose group membership so per-service group authorization (R-IDP-06) can be enforced. The
-      # baseline here is "authenticated"; per-service `--allowed-groups` is layered on at the Gateway
-      # ext-auth attachment (WS-NET contract) or via a dedicated oauth2-proxy instance per group.
+      # Keycloak `oauth2-proxy` client emits a `groups` claim (full.path=false => bare names like
+      # `platform-admins`, keycloak-realm-import.yaml). `--allowed-groups` below is the PLATFORM FLOOR:
+      # a valid Keycloak session is NOT enough — the user must ALSO be in `platform-admins`, or
+      # /oauth2/auth returns 403 and ext_authz denies (R-IDP-06). Every currently-onboarded
+      # forward-auth app (pihole, *arr, qbittorrent, homepage, prometheus, alertmanager, hubble)
+      # declares `groups: [platform-admins]` in its ServiceDefinition, so this floor matches them all.
+      # Per-ROUTE group differentiation (a future app allowing a different group) does NOT need a
+      # second proxy: the ext_authz filter appends `?allowed_groups=<group>` to /oauth2/auth from the
+      # HTTPRoute's `homelab.keithjackson.dev/auth-groups` annotation — oauth2-proxy honours the
+      # per-request query on top of this floor. See forward-auth-cec.yaml + README "opt-in".
      oidc-groups-claim: groups
+      allowed-groups: platform-admins    # R-IDP-06 floor: authenticated AND in platform-admins
+      # Single Keycloak provider => skip the oauth2-proxy interstitial "Sign In" button and 302
+      # straight to Keycloak. Verified in-cluster: WITHOUT this, an ext_authz-denied browser request
+      # gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
+      skip-provider-button: "true"
      scope: "openid email profile"
    # Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
    metrics:
      enabled: true
-    # TODO(WS-SEC/WS-NET, Phase 6): the Gateway ext-authz ATTACHMENT (which HTTPRoutes require this
-    #   proxy, and each route's allowed groups) is a WS-NET Gateway contract — Cilium Gateway API
-    #   ext-auth wiring. oauth2-proxy is the authenticator; the per-route enforcement is Gateway-side.
+    # ENFORCEMENT (Phase 6): oauth2-proxy is the AUTHENTICATOR; the per-route ext_authz ENFORCEMENT is
+    #   Gateway-side. On Cilium 1.16.5 the shared Gateway's Envoy config is a machine-owned, constantly
+    #   regenerated CiliumEnvoyConfig (`gateway/cilium-gateway-shared`) that CANNOT be injected into,
+    #   and native Gateway-API ExternalAuth (GEP-1494) ships in NO released Cilium version yet. The
+    #   enforcement mechanism + the three wiring options are documented in README.md; the interim
+    #   self-authored ext_authz listener is forward-auth-cec.yaml (a template, not yet Flux-wired).