# Evaluation: Cilium 1.20.1 native Gateway API ExternalAuth vs. the oauth2-proxy design **Status:** Research + plan only — NO live change proposed for tonight. Reviewable evaluation. **Author:** WS-NET/WS-SEC investigation (Claude Opus 4.8), 2026-08-31. **Scope:** Whether Cilium 1.20.1's native Gateway API `ExternalAuth` filter (GEP-1494) can replace (a) the hand-authored, **un-wired** forward-auth `CiliumClusterwideEnvoyConfig` (`platform/oauth2-proxy/forward-auth-cec.yaml`) and (b) Crafty's per-pod oauth2-proxy **sidecar**. **Answers the open item** from DECISIONS-LOG (2026-08-30): *"the Cilium ext_authz/forward-auth workaround relevance under 1.20.1."* --- ## TL;DR — Verdict: **DEFER (partial-feasible, not viable as-is tonight)** Native `ExternalAuth` **exists in the Cilium 1.20.1 codebase** and *could* eventually replace both mechanisms with a single, Gateway-API-native, per-route filter. **But it is NOT usable on this cluster in its current state**, and adopting it is a non-trivial, elevated-risk change to the exact shared Gateway that SSO was just stabilized on. Recommendation: **keep the current design, do not adopt now.** Track native ExternalAuth as the long-term target and revisit after the blocking conditions below clear. Three hard blockers, in order of severity: 1. **The cluster's Gateway API CRDs are the `standard` channel — the `ExternalAuth` HTTPRoute filter is not even installable.** Verified live: the `httproutes.gateway.networking.k8s.io` CRD is `gateway.networking.k8s.io/channel: standard`, bundle `v1.6.1`, and its `filters[].type` enum is exactly `[RequestHeaderModifier, ResponseHeaderModifier, RequestMirror, RequestRedirect, URLRewrite, ExtensionRef, CORS]` — **`ExternalAuth` is absent**. GEP-1494 is an **experimental** Gateway API feature; using it requires swapping the cluster-wide Gateway API CRDs to the **experimental channel** bundle. That is a breaking, cluster-wide CRD change, and it must be kept in lock-step with the Ansible-bootstrap ↔ Flux **parity contract**. 2. **The feature is young and shipped with a fail-**open** security bug.** `ExternalAuth` landed in Cilium **1.20.0** (2026-07-29, PR #45739). Issue **#47178 "Basic externalauth setup fails open"** was filed against 1.20.0-rc.0 — the ext_authz filter silently produced no Envoy entry and let **unauthenticated traffic through**. The fix (PR #47265) was backported to the 1.20 branch (~2026-07-21) and should be in 1.20.0/1.20.1, but a fail-**open** regression in a brand-new security-critical filter is exactly the failure mode R-IDP-02 forbids. Confidence that today's filter is robust is low, and documentation is thin (Cilium ships **no** `external-auth` docs page in 1.20 — the dedicated doc URL 404s; the stable Gateway API doc page makes no mention of it). 3. **`enforce_policy_on_l7lb` (the CVE-2025-30162 hairpin behavior) will bite the ext_authz hop too.** Commit `68690e5` just fixed cluster-wide SSO breakage: after the 1.16.5→1.20.1 jump, the shared Gateway's L7 proxy re-evaluates a pod's **egress policy against the *real backend* identity** when it hairpins (`enforce_policy_on_l7lb`), so every NetworkPolicy'd app needed an explicit `toEndpoints` egress allow to the Keycloak backend. A Gateway-level `ExternalAuth` filter makes the Gateway's Envoy hairpin to the **oauth2-proxy** backend cluster for the auth check — the identical class of policy re-evaluation. Expect to need new egress/identity allowances for the ext_authz path, and to re-validate it does not regress the freshly-fixed Keycloak egress. Given SSO "was just painfully fixed," the cost/benefit does not favor touching it now. --- ## Current state (verified — live cluster + repo) **Cluster:** Cilium `v1.20.1` (`quay.io/cilium/cilium:v1.20.1`), Gateway API CRDs bundle `v1.6.1` **standard** channel, k3s v1.36.4. Shared Gateway `gateway/shared` (class `cilium`, VIP `192.168.1.53`, Programmed), backed by the machine-owned `CiliumEnvoyConfig gateway/cilium-gateway-shared` (the only CEC/CCEC on the cluster). `cilium-config` has `enable-gateway-api: true` and **no** external-auth flag (`mesh-auth-*` is the unrelated SPIFFE mutual-auth feature and is disabled). **Auth design today (three mechanisms, one of them dead):** | Mechanism | Where | Apps | Status | |---|---|---|---| | **Native OIDC** (`auth.mode: oidc`) | in-app OIDC client → Keycloak | gitea, nextcloud, semaphore, grafana(native) | **Live, working.** Not in scope to change. | | **Native/other** | app-specific | seerr (own OIDC), plex | Live. Out of scope. | | **forward-auth CEC** (`auth.mode: forward-auth`) | `platform/oauth2-proxy/forward-auth-cec.yaml` — hand-authored `CiliumClusterwideEnvoyConfig` with an `ext_authz` HTTP filter → oauth2-proxy `/oauth2/auth` | jackett, pihole, qbittorrent, radarr, sonarr | **NOT DEPLOYED.** A TEMPLATE, **not** referenced by `platform/oauth2-proxy/kustomization.yaml`. Proven in a throwaway envoy pod (2026-08-28) but never wired. These 5 UIs are currently reachable on the shared Gateway **un-gated**. | | **oauth2-proxy sidecar** (per-pod reverse-proxy, "Option B") | `platform/crafty/deployment.yaml` container `oauth2-proxy` (`v7.8.1`, `:4180` → `https://127.0.0.1:8443`) | crafty | **Live, working.** In-data-path fail-closed gate. | **The renderer contract (`catalog/render.py`):** `auth.mode ∈ {oidc, forward-auth, native, none}` (`AUTH_MODES`, line 53); default is `forward-auth` for any app with `ingressRoutes` else `none` (lines 76-77). `_auth_annotations()` (line 605) stamps `homelab.keithjackson.dev/auth-mode` + `/auth-groups` onto the HTTPRoute for `oidc`/`forward-auth`, and pods carry an `auth-mode` label enforced by a Kyverno `require-auth-mode-label` policy (R-IDP-02). Since commit `68690e5`, the renderer also emits a `toEndpoints` egress allow to the Keycloak backend for `oidc`/`forward-auth` apps (line 539). **No renderer field today expresses an ext_authz `backendRef`** — the forward-auth annotations are consumed by the (un-wired) platform enforcement, not by a Gateway-API filter. **Why the CEC exists / why it's not on the Gateway (per its own header + README):** on Cilium 1.16.5 there was no supported hook to inject `ext_authz` into the machine-regenerated shared-Gateway Envoy program, and native ExternalAuth "ships in NO released Cilium version yet (PoC only)." That premise is now **partially stale** — see below — which is precisely why this evaluation exists. --- ## Does Cilium 1.20.1 support native Gateway API ExternalAuth? (research) **Yes, in the code — as an experimental Gateway API feature that is not enabled on this cluster.** - **Shipped in 1.20.0.** The v1.20.0 release notes list: *"🔐 External Authorization: HTTPRoute requests can now be authenticated and authorized through an external service before they reach the application, using the Gateway API `ExternalAuth` filter from GEP-1494."* (PR [cilium/cilium#45739](https://github.com/cilium/cilium/pull/45739)). Isovalent's 1.20 launch post headlines "Gateway API ExternalAuth." - **Shape (GEP-1494 experimental filter).** An HTTPRoute rule gains a filter `type: ExternalAuth` with `externalAuth.protocol: GRPC|HTTP`, a `backendRef` to the auth service, and `grpc.allowedHeaders` / `http.{path,allowedHeaders,allowedResponseHeaders}` / `forwardBody`. It maps onto Envoy's `ext_authz` protocol — the **same** mechanism the hand-authored CEC uses, but declared per-route on the standard shared Gateway instead of a separate listener. Example: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute spec: rules: - filters: - type: ExternalAuth externalAuth: protocol: GRPC # or HTTP (oauth2-proxy /oauth2/auth is HTTP) backendRef: { kind: Service, name: oauth2-proxy, port: 80 } http: { path: /oauth2/auth, allowedHeaders: [cookie, authorization] } backendRefs: [{ name: , port: }] ``` - **It is EXPERIMENTAL-channel.** GEP-1494 status is **Experimental** ([gateway-api.sigs.k8s.io/geps/gep-1494](https://gateway-api.sigs.k8s.io/geps/gep-1494/)). Its fields live only in the Gateway API **experimental** CRD bundle. **This cluster runs the `standard` channel** (verified), so the filter type is not present in the HTTPRoute schema — a manifest using it would be rejected at admission today. - **Maturity risk.** Fail-open bug [#47178](https://github.com/cilium/cilium/issues/47178) → fix [#47265](https://github.com/cilium/cilium/pull/47265) (backported to 1.20, ~2026-07-21). Cilium ships **no dedicated ExternalAuth documentation page** in 1.20 (the `/en/v1.20/.../external-auth/` and `/en/latest/.../external-auth/` URLs both 404; the stable Gateway API support page does not mention it). Feature is real but young and thinly documented. **Bottom line:** the CEC/sidecar READMEs' claim that native ExternalAuth "ships in no released Cilium version" is now outdated — it ships in 1.20.x — **but** it is experimental, off by default here (wrong CRD channel), and not battle-tested. --- ## Would it cleanly replace the two mechanisms? **(a) The un-wired forward-auth CEC — YES, in principle, and more cleanly.** This is the strongest case for native ExternalAuth: the CEC is a hand-authored TLS listener + routes + EDS clusters on a *separate* VIP that requires moving hosts off the shared Gateway and repointing DNS. A native `ExternalAuth` filter would instead be one filter block on each app's **existing** HTTPRoute on the **existing** shared Gateway/VIP/DNS — no second ingress, no DNS churn, no hand-authored TLS. Since the CEC is not deployed, there is **no rip-out risk** here: adopting native ExternalAuth for the 5 forward-auth apps (jackett/pihole/qbittorrent/radarr/sonarr) would *complete* an unfinished story rather than replace a working one. The renderer already carries the `auth-mode`/`auth-groups` signal; it would need a new emitter that turns `auth.mode: forward-auth` into the HTTPRoute `ExternalAuth` filter (see plan). **(b) Crafty's sidecar — PARTIAL, and lower value.** Native ExternalAuth could drop the per-pod oauth2-proxy container and gate `crafty.home` at the Gateway like the others. **But** Crafty's UI is HTTPS **self-signed on :8443**; the sidecar terminates that in-pod (`--ssl-upstream-insecure-skip- verify`). Moving the gate to the Gateway means the shared Gateway's Envoy must speak HTTPS to Crafty's self-signed `:8443` upstream — requiring a `BackendTLSPolicy` with `insecureSkipVerify` (or a real cert), and the `crafty` Service would target `:8443` directly. That is net-new Gateway plumbing to remove a small, **working**, fail-closed sidecar. Low priority; do Crafty last, if at all. Native ExternalAuth **cannot** and **should not** touch the `oidc` apps (gitea/nextcloud/semaphore/ grafana) — those use in-app OIDC and are out of scope. --- ## Proposed migration plan (do NOT execute now — for review) Phased, reversible, lowest-blast-radius first. **Precondition gates P1-P3 must clear before ANY cluster change.** Each step names the files it touches. ### Phase 0 — Preconditions (research/verify only, no cluster change) - **P1. Confirm the fail-open fix is in 1.20.1.** Verify PR #47265 shipped in the running `v1.20.1` (changelog / `git tag --contains`), and reproduce a **fail-closed** ext_authz on a throwaway route (the 2026-08-28 CEC proof, redone with the *native filter* on a scratch HTTPRoute) BEFORE trusting it for R-IDP-02. - **P2. Decide the Gateway API channel swap.** Adopting GEP-1494 requires the **experimental** CRD bundle cluster-wide. Assess blast radius (every Gateway/HTTPRoute revalidated against experimental schemas) and reconcile the **Ansible-bootstrap ↔ Flux parity contract** — the Ansible bootstrap that installs Gateway API CRDs must install the *same* channel. Files: wherever the Gateway API CRD bundle is declared for both bootstrap and Flux (search `gateway.networking.k8s.io` CRD source; `platform/gateway/` + the Ansible bootstrap CRD step). This is the single biggest change and its own reviewable sub-plan. - **P3. Confirm the Cilium enablement path.** Determine whether Cilium honors the `ExternalAuth` filter on experimental-CRD presence alone or needs an agent/helm flag; if a flag exists it is a `[parity]` value in `platform/cilium/helmrelease.yaml` and the Ansible Cilium values. ### Phase 1 — One canary app (pihole), additive, reversible - Install experimental Gateway API CRDs (P2) + any Cilium flag (P3). - Add the `ExternalAuth` filter to **pihole's** HTTPRoute only, `backendRef` → `oauth2-proxy` Service, `http.path: /oauth2/auth`, header allow-list mirroring the CEC's proven stanza. Keep `failureMode`/deny semantics fail-**closed**. Files: `catalog/pihole.yaml` (unchanged intent) + a new renderer emitter in `catalog/render.py` (`_httproute()` path) that renders the filter from `auth.mode: forward-auth`; re-render pihole golden. Add the ext_authz **egress allow** to the oauth2-proxy backend for the app (the `enforce_policy_on_l7lb` mitigation, mirroring the Keycloak fix in `render.py`). - **Browser test** (the human): incognito `pihole.home` → 302 to Keycloak → login `platform-admins` → PiHole UI; non-`platform-admins` → 403; oauth2-proxy down → **denied**, never the bare UI. - **Rollback:** delete the filter block (revert the HTTPRoute) → back to un-gated shared Gateway. No data, no DNS, no VIP involved. ### Phase 2 — Roll out to the remaining forward-auth apps - Repeat for jackett, qbittorrent, radarr, sonarr via the renderer (they already declare `auth.mode: forward-auth`; the emitter does it uniformly). Re-render 4 goldens. - **Delete `platform/oauth2-proxy/forward-auth-cec.yaml`** and its README section describing the separate-VIP interim — now obsolete. (It was never wired, so this is doc/template cleanup.) ### Phase 3 — Crafty (optional, last) - Add `ExternalAuth` to `crafty`'s HTTPRoute; point the `crafty` Service at `:8443`; add a `BackendTLSPolicy` for the self-signed upstream; **remove the oauth2-proxy sidecar** from `platform/crafty/deployment.yaml` and its SOPS creds. Verify game-port L4 path (192.168.1.55) untouched. Higher effort for marginal gain — defer unless consolidating. --- ## Risks & rollback | Risk | Severity | Mitigation / Rollback | |---|---|---| | **Fail-open regression** (a new/edge ext_authz bug lets traffic through un-authed — R-IDP-02 violation) | **High** | P1 reproduction on a scratch route before any real app; canary (pihole) first with explicit browser deny-path test; keep un-gated-but-internal apps behind LAN until proven. | | **Experimental-channel CRD swap breaks existing Gateways/HTTPRoutes** | **High** | Treat P2 as its own reviewed change; snapshot/backup CRDs; parity with Ansible bootstrap; validate all 16 existing HTTPRoutes still Programmed after the swap. | | **`enforce_policy_on_l7lb` blocks the ext_authz hop** (Gateway hairpins to oauth2-proxy backend; NetworkPolicy'd apps 403 the auth check — the CVE-2025-30162 pattern from commit `68690e5`) | **Med-High** | Emit a `toEndpoints` egress allow to the oauth2-proxy backend for forward-auth apps, exactly as done for Keycloak; re-validate the Keycloak egress is not regressed. | | **Parity-contract drift** (Flux vs Ansible disagree on CRD channel / Cilium flag) | **Med** | Land CRD-channel + Cilium-flag changes in both bootstrap and Flux in one PR; gate (`make -C ci gate`). | | **Thin docs / churn** (experimental API fields change across Gateway API/Cilium minors) | **Med** | Pin bundle version; Renovate watch; keep the CEC template in git history as a fallback recipe. | | **Regressing freshly-fixed SSO** | **High** | Do nothing to `oidc` apps; canary + reversible filter-only changes; never touch the shared Gateway CEC directly. | **Global rollback:** every proposed change is additive at the HTTPRoute level and reverts by removing the filter block. The nuclear fallback if native ExternalAuth proves flaky is the **already-proven Option B** (per-app oauth2-proxy reverse-proxy, as Crafty uses today) — Gateway-API-native, no CRD channel swap, just a `backendRef` to a per-app proxy. --- ## Recommendation **DEFER.** Native ExternalAuth is the right long-term target and would let us delete both the dead CEC template and (eventually) Crafty's sidecar in favor of one uniform, Gateway-API-native filter. But **tonight it is not viable**: the enabling CRD channel isn't installed, the feature is experimental and recently fail-open-buggy, it is undocumented in 1.20, and it would perturb the shared Gateway and the just-fixed `enforce_policy_on_l7lb` egress that SSO now depends on. Concretely: - **Do NOT** adopt now. **Keep** Crafty's working sidecar and leave the 5 forward-auth UIs as-is. - **When revisiting** (next maintenance window, not tonight): execute **Phase 0** preconditions, then a **pihole-only canary** (Phase 1). Only after a clean fail-closed browser proof, roll out Phase 2 and retire the CEC template. - **Cheaper interim if gating the 5 UIs becomes urgent before then:** adopt **Option B** (per-app oauth2-proxy reverse-proxy, the Crafty pattern) — no experimental CRDs, fully reversible — rather than the separate-VIP CEC or a rushed native-ExternalAuth switch. This resolves the DECISIONS-LOG open item: **native ExternalAuth is relevant and code-present under 1.20.1, but blocked on an experimental CRD-channel swap + maturity/fail-open concerns; the interim workaround stays.** --- ## Sources - Cilium v1.20.0 release notes — "External Authorization" / GEP-1494 (PR #45739): https://github.com/cilium/cilium/releases/tag/v1.20.0 - Cilium v1.20.1 release: https://github.com/cilium/cilium/releases/tag/v1.20.1 - Isovalent — "Cilium 1.20: Gateway API ExternalAuth…": https://isovalent.com/blog/post/cilium-1-20/ - GEP-1494 (HTTP Auth in Gateway API), status **Experimental**: https://gateway-api.sigs.k8s.io/geps/gep-1494/ - Issue #47178 "Basic externalauth setup fails open": https://github.com/cilium/cilium/issues/47178 - Fix PR #47265 (backported to 1.20): https://github.com/cilium/cilium/pull/47265 - Issue #45704 (GEP-1494 filter support, closed as dup of the implementing PR): https://github.com/cilium/cilium/issues/45704 - Cilium 1.20.1 Gateway API support page (no ExternalAuth mention — feature undocumented in 1.20): https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/ - Live cluster (read-only): Cilium `v1.20.1`; HTTPRoute CRD `channel: standard`, bundle `v1.6.1`, `filters[].type` enum lacks `ExternalAuth`; `cilium-config` has no external-auth flag. - Repo: `platform/oauth2-proxy/forward-auth-cec.yaml` (+ README), `platform/crafty/deployment.yaml` (+ README), `catalog/render.py` (`AUTH_MODES`, `_auth_annotations`, keycloak-egress emitter), commit `68690e5` (CVE-2025-30162 / `enforce_policy_on_l7lb` egress fix).