From b745bab26be9b2d72f8d68787c64d1f587191ba0 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 31 Aug 2026 01:19:36 -0500 Subject: [PATCH] docs(plan): evaluate Cilium 1.20.1 native Gateway ExternalAuth vs oauth2-proxy Research + implementation plan (no live change) on whether Cilium 1.20.1's native Gateway API ExternalAuth filter (GEP-1494) can replace the un-wired forward-auth CEC template and Crafty's oauth2-proxy sidecar. Verdict: DEFER. The filter exists in 1.20.1 code (shipped 1.20.0, PR #45739) but is EXPERIMENTAL-channel Gateway API; the cluster runs the standard-channel CRDs (verified: HTTPRoute filters[].type enum lacks ExternalAuth), so it is not even installable today. Also: fail-open bug (#47178, fixed #47265) in a brand-new security filter, no 1.20 docs, and the ext_authz hop would hit the same enforce_policy_on_l7lb hairpin re-evaluation just fixed for Keycloak (68690e5). Plan is phased/reversible (pihole canary first) and keeps the working sidecar. Resolves the DECISIONS-LOG open item on ext_authz workaround relevance under 1.20.1. Co-Authored-By: Claude Opus 4.8 --- .../cilium-native-externalauth-evaluation.md | 268 ++++++++++++++++++ 1 file changed, 268 insertions(+) create mode 100644 docs/plans/cilium-native-externalauth-evaluation.md diff --git a/docs/plans/cilium-native-externalauth-evaluation.md b/docs/plans/cilium-native-externalauth-evaluation.md new file mode 100644 index 0000000..1c1a1c5 --- /dev/null +++ b/docs/plans/cilium-native-externalauth-evaluation.md @@ -0,0 +1,268 @@ +# Evaluation: Cilium 1.20.1 native Gateway API ExternalAuth vs. the oauth2-proxy design + +**Status:** Research + plan only — NO live change proposed for tonight. Reviewable evaluation. +**Author:** WS-NET/WS-SEC investigation (Claude Opus 4.8), 2026-08-31. +**Scope:** Whether Cilium 1.20.1's native Gateway API `ExternalAuth` filter (GEP-1494) can replace +(a) the hand-authored, **un-wired** forward-auth `CiliumClusterwideEnvoyConfig` +(`platform/oauth2-proxy/forward-auth-cec.yaml`) and (b) Crafty's per-pod oauth2-proxy **sidecar**. +**Answers the open item** from DECISIONS-LOG (2026-08-30): *"the Cilium ext_authz/forward-auth +workaround relevance under 1.20.1."* + +--- + +## TL;DR — Verdict: **DEFER (partial-feasible, not viable as-is tonight)** + +Native `ExternalAuth` **exists in the Cilium 1.20.1 codebase** and *could* eventually replace both +mechanisms with a single, Gateway-API-native, per-route filter. **But it is NOT usable on this +cluster in its current state**, and adopting it is a non-trivial, elevated-risk change to the exact +shared Gateway that SSO was just stabilized on. Recommendation: **keep the current design, do not +adopt now.** Track native ExternalAuth as the long-term target and revisit after the blocking +conditions below clear. + +Three hard blockers, in order of severity: + +1. **The cluster's Gateway API CRDs are the `standard` channel — the `ExternalAuth` HTTPRoute filter + is not even installable.** Verified live: the `httproutes.gateway.networking.k8s.io` CRD is + `gateway.networking.k8s.io/channel: standard`, bundle `v1.6.1`, and its `filters[].type` enum is + exactly `[RequestHeaderModifier, ResponseHeaderModifier, RequestMirror, RequestRedirect, + URLRewrite, ExtensionRef, CORS]` — **`ExternalAuth` is absent**. GEP-1494 is an **experimental** + Gateway API feature; using it requires swapping the cluster-wide Gateway API CRDs to the + **experimental channel** bundle. That is a breaking, cluster-wide CRD change, and it must be kept + in lock-step with the Ansible-bootstrap ↔ Flux **parity contract**. +2. **The feature is young and shipped with a fail-**open** security bug.** `ExternalAuth` landed in + Cilium **1.20.0** (2026-07-29, PR #45739). Issue **#47178 "Basic externalauth setup fails open"** + was filed against 1.20.0-rc.0 — the ext_authz filter silently produced no Envoy entry and let + **unauthenticated traffic through**. The fix (PR #47265) was backported to the 1.20 branch + (~2026-07-21) and should be in 1.20.0/1.20.1, but a fail-**open** regression in a brand-new + security-critical filter is exactly the failure mode R-IDP-02 forbids. Confidence that today's + filter is robust is low, and documentation is thin (Cilium ships **no** `external-auth` docs page + in 1.20 — the dedicated doc URL 404s; the stable Gateway API doc page makes no mention of it). +3. **`enforce_policy_on_l7lb` (the CVE-2025-30162 hairpin behavior) will bite the ext_authz hop too.** + Commit `68690e5` just fixed cluster-wide SSO breakage: after the 1.16.5→1.20.1 jump, the shared + Gateway's L7 proxy re-evaluates a pod's **egress policy against the *real backend* identity** when + it hairpins (`enforce_policy_on_l7lb`), so every NetworkPolicy'd app needed an explicit + `toEndpoints` egress allow to the Keycloak backend. A Gateway-level `ExternalAuth` filter makes the + Gateway's Envoy hairpin to the **oauth2-proxy** backend cluster for the auth check — the identical + class of policy re-evaluation. Expect to need new egress/identity allowances for the ext_authz + path, and to re-validate it does not regress the freshly-fixed Keycloak egress. + +Given SSO "was just painfully fixed," the cost/benefit does not favor touching it now. + +--- + +## Current state (verified — live cluster + repo) + +**Cluster:** Cilium `v1.20.1` (`quay.io/cilium/cilium:v1.20.1`), Gateway API CRDs bundle `v1.6.1` +**standard** channel, k3s v1.36.4. Shared Gateway `gateway/shared` (class `cilium`, VIP +`192.168.1.53`, Programmed), backed by the machine-owned `CiliumEnvoyConfig +gateway/cilium-gateway-shared` (the only CEC/CCEC on the cluster). `cilium-config` has +`enable-gateway-api: true` and **no** external-auth flag (`mesh-auth-*` is the unrelated SPIFFE +mutual-auth feature and is disabled). + +**Auth design today (three mechanisms, one of them dead):** + +| Mechanism | Where | Apps | Status | +|---|---|---|---| +| **Native OIDC** (`auth.mode: oidc`) | in-app OIDC client → Keycloak | gitea, nextcloud, semaphore, grafana(native) | **Live, working.** Not in scope to change. | +| **Native/other** | app-specific | seerr (own OIDC), plex | Live. Out of scope. | +| **forward-auth CEC** (`auth.mode: forward-auth`) | `platform/oauth2-proxy/forward-auth-cec.yaml` — hand-authored `CiliumClusterwideEnvoyConfig` with an `ext_authz` HTTP filter → oauth2-proxy `/oauth2/auth` | jackett, pihole, qbittorrent, radarr, sonarr | **NOT DEPLOYED.** A TEMPLATE, **not** referenced by `platform/oauth2-proxy/kustomization.yaml`. Proven in a throwaway envoy pod (2026-08-28) but never wired. These 5 UIs are currently reachable on the shared Gateway **un-gated**. | +| **oauth2-proxy sidecar** (per-pod reverse-proxy, "Option B") | `platform/crafty/deployment.yaml` container `oauth2-proxy` (`v7.8.1`, `:4180` → `https://127.0.0.1:8443`) | crafty | **Live, working.** In-data-path fail-closed gate. | + +**The renderer contract (`catalog/render.py`):** `auth.mode ∈ {oidc, forward-auth, native, none}` +(`AUTH_MODES`, line 53); default is `forward-auth` for any app with `ingressRoutes` else `none` +(lines 76-77). `_auth_annotations()` (line 605) stamps +`homelab.keithjackson.dev/auth-mode` + `/auth-groups` onto the HTTPRoute for `oidc`/`forward-auth`, +and pods carry an `auth-mode` label enforced by a Kyverno `require-auth-mode-label` policy +(R-IDP-02). Since commit `68690e5`, the renderer also emits a `toEndpoints` egress allow to the +Keycloak backend for `oidc`/`forward-auth` apps (line 539). **No renderer field today expresses an +ext_authz `backendRef`** — the forward-auth annotations are consumed by the (un-wired) platform +enforcement, not by a Gateway-API filter. + +**Why the CEC exists / why it's not on the Gateway (per its own header + README):** on Cilium 1.16.5 +there was no supported hook to inject `ext_authz` into the machine-regenerated shared-Gateway Envoy +program, and native ExternalAuth "ships in NO released Cilium version yet (PoC only)." That premise +is now **partially stale** — see below — which is precisely why this evaluation exists. + +--- + +## Does Cilium 1.20.1 support native Gateway API ExternalAuth? (research) + +**Yes, in the code — as an experimental Gateway API feature that is not enabled on this cluster.** + +- **Shipped in 1.20.0.** The v1.20.0 release notes list: *"🔐 External Authorization: HTTPRoute + requests can now be authenticated and authorized through an external service before they reach the + application, using the Gateway API `ExternalAuth` filter from GEP-1494."* (PR + [cilium/cilium#45739](https://github.com/cilium/cilium/pull/45739)). Isovalent's 1.20 launch post + headlines "Gateway API ExternalAuth." +- **Shape (GEP-1494 experimental filter).** An HTTPRoute rule gains a filter + `type: ExternalAuth` with `externalAuth.protocol: GRPC|HTTP`, a `backendRef` to the auth service, + and `grpc.allowedHeaders` / `http.{path,allowedHeaders,allowedResponseHeaders}` / `forwardBody`. + It maps onto Envoy's `ext_authz` protocol — the **same** mechanism the hand-authored CEC uses, + but declared per-route on the standard shared Gateway instead of a separate listener. Example: + ```yaml + apiVersion: gateway.networking.k8s.io/v1 + kind: HTTPRoute + spec: + rules: + - filters: + - type: ExternalAuth + externalAuth: + protocol: GRPC # or HTTP (oauth2-proxy /oauth2/auth is HTTP) + backendRef: { kind: Service, name: oauth2-proxy, port: 80 } + http: { path: /oauth2/auth, allowedHeaders: [cookie, authorization] } + backendRefs: [{ name: , port: }] + ``` +- **It is EXPERIMENTAL-channel.** GEP-1494 status is **Experimental** + ([gateway-api.sigs.k8s.io/geps/gep-1494](https://gateway-api.sigs.k8s.io/geps/gep-1494/)). Its + fields live only in the Gateway API **experimental** CRD bundle. **This cluster runs the `standard` + channel** (verified), so the filter type is not present in the HTTPRoute schema — a manifest using + it would be rejected at admission today. +- **Maturity risk.** Fail-open bug [#47178](https://github.com/cilium/cilium/issues/47178) → fix + [#47265](https://github.com/cilium/cilium/pull/47265) (backported to 1.20, ~2026-07-21). Cilium + ships **no dedicated ExternalAuth documentation page** in 1.20 (the `/en/v1.20/.../external-auth/` + and `/en/latest/.../external-auth/` URLs both 404; the stable Gateway API support page does not + mention it). Feature is real but young and thinly documented. + +**Bottom line:** the CEC/sidecar READMEs' claim that native ExternalAuth "ships in no released Cilium +version" is now outdated — it ships in 1.20.x — **but** it is experimental, off by default here +(wrong CRD channel), and not battle-tested. + +--- + +## Would it cleanly replace the two mechanisms? + +**(a) The un-wired forward-auth CEC — YES, in principle, and more cleanly.** This is the strongest +case for native ExternalAuth: the CEC is a hand-authored TLS listener + routes + EDS clusters on a +*separate* VIP that requires moving hosts off the shared Gateway and repointing DNS. A native +`ExternalAuth` filter would instead be one filter block on each app's **existing** HTTPRoute on the +**existing** shared Gateway/VIP/DNS — no second ingress, no DNS churn, no hand-authored TLS. Since +the CEC is not deployed, there is **no rip-out risk** here: adopting native ExternalAuth for the 5 +forward-auth apps (jackett/pihole/qbittorrent/radarr/sonarr) would *complete* an unfinished story +rather than replace a working one. The renderer already carries the `auth-mode`/`auth-groups` signal; +it would need a new emitter that turns `auth.mode: forward-auth` into the HTTPRoute `ExternalAuth` +filter (see plan). + +**(b) Crafty's sidecar — PARTIAL, and lower value.** Native ExternalAuth could drop the per-pod +oauth2-proxy container and gate `crafty.home` at the Gateway like the others. **But** Crafty's UI is +HTTPS **self-signed on :8443**; the sidecar terminates that in-pod (`--ssl-upstream-insecure-skip- +verify`). Moving the gate to the Gateway means the shared Gateway's Envoy must speak HTTPS to Crafty's +self-signed `:8443` upstream — requiring a `BackendTLSPolicy` with `insecureSkipVerify` (or a real +cert), and the `crafty` Service would target `:8443` directly. That is net-new Gateway plumbing to +remove a small, **working**, fail-closed sidecar. Low priority; do Crafty last, if at all. + +Native ExternalAuth **cannot** and **should not** touch the `oidc` apps (gitea/nextcloud/semaphore/ +grafana) — those use in-app OIDC and are out of scope. + +--- + +## Proposed migration plan (do NOT execute now — for review) + +Phased, reversible, lowest-blast-radius first. **Precondition gates P1-P3 must clear before ANY +cluster change.** Each step names the files it touches. + +### Phase 0 — Preconditions (research/verify only, no cluster change) +- **P1. Confirm the fail-open fix is in 1.20.1.** Verify PR #47265 shipped in the running `v1.20.1` + (changelog / `git tag --contains`), and reproduce a **fail-closed** ext_authz on a throwaway route + (the 2026-08-28 CEC proof, redone with the *native filter* on a scratch HTTPRoute) BEFORE trusting + it for R-IDP-02. +- **P2. Decide the Gateway API channel swap.** Adopting GEP-1494 requires the **experimental** CRD + bundle cluster-wide. Assess blast radius (every Gateway/HTTPRoute revalidated against experimental + schemas) and reconcile the **Ansible-bootstrap ↔ Flux parity contract** — the Ansible bootstrap + that installs Gateway API CRDs must install the *same* channel. Files: wherever the Gateway API CRD + bundle is declared for both bootstrap and Flux (search `gateway.networking.k8s.io` CRD source; + `platform/gateway/` + the Ansible bootstrap CRD step). This is the single biggest change and its + own reviewable sub-plan. +- **P3. Confirm the Cilium enablement path.** Determine whether Cilium honors the `ExternalAuth` + filter on experimental-CRD presence alone or needs an agent/helm flag; if a flag exists it is a + `[parity]` value in `platform/cilium/helmrelease.yaml` and the Ansible Cilium values. + +### Phase 1 — One canary app (pihole), additive, reversible +- Install experimental Gateway API CRDs (P2) + any Cilium flag (P3). +- Add the `ExternalAuth` filter to **pihole's** HTTPRoute only, `backendRef` → `oauth2-proxy` + Service, `http.path: /oauth2/auth`, header allow-list mirroring the CEC's proven stanza. Keep + `failureMode`/deny semantics fail-**closed**. + Files: `catalog/pihole.yaml` (unchanged intent) + a new renderer emitter in + `catalog/render.py` (`_httproute()` path) that renders the filter from `auth.mode: forward-auth`; + re-render pihole golden. Add the ext_authz **egress allow** to the oauth2-proxy backend for the app + (the `enforce_policy_on_l7lb` mitigation, mirroring the Keycloak fix in `render.py`). +- **Browser test** (the human): incognito `pihole.home` → 302 to Keycloak → login `platform-admins` + → PiHole UI; non-`platform-admins` → 403; oauth2-proxy down → **denied**, never the bare UI. +- **Rollback:** delete the filter block (revert the HTTPRoute) → back to un-gated shared Gateway. No + data, no DNS, no VIP involved. + +### Phase 2 — Roll out to the remaining forward-auth apps +- Repeat for jackett, qbittorrent, radarr, sonarr via the renderer (they already declare + `auth.mode: forward-auth`; the emitter does it uniformly). Re-render 4 goldens. +- **Delete `platform/oauth2-proxy/forward-auth-cec.yaml`** and its README section describing the + separate-VIP interim — now obsolete. (It was never wired, so this is doc/template cleanup.) + +### Phase 3 — Crafty (optional, last) +- Add `ExternalAuth` to `crafty`'s HTTPRoute; point the `crafty` Service at `:8443`; add a + `BackendTLSPolicy` for the self-signed upstream; **remove the oauth2-proxy sidecar** from + `platform/crafty/deployment.yaml` and its SOPS creds. Verify game-port L4 path (192.168.1.55) + untouched. Higher effort for marginal gain — defer unless consolidating. + +--- + +## Risks & rollback + +| Risk | Severity | Mitigation / Rollback | +|---|---|---| +| **Fail-open regression** (a new/edge ext_authz bug lets traffic through un-authed — R-IDP-02 violation) | **High** | P1 reproduction on a scratch route before any real app; canary (pihole) first with explicit browser deny-path test; keep un-gated-but-internal apps behind LAN until proven. | +| **Experimental-channel CRD swap breaks existing Gateways/HTTPRoutes** | **High** | Treat P2 as its own reviewed change; snapshot/backup CRDs; parity with Ansible bootstrap; validate all 16 existing HTTPRoutes still Programmed after the swap. | +| **`enforce_policy_on_l7lb` blocks the ext_authz hop** (Gateway hairpins to oauth2-proxy backend; NetworkPolicy'd apps 403 the auth check — the CVE-2025-30162 pattern from commit `68690e5`) | **Med-High** | Emit a `toEndpoints` egress allow to the oauth2-proxy backend for forward-auth apps, exactly as done for Keycloak; re-validate the Keycloak egress is not regressed. | +| **Parity-contract drift** (Flux vs Ansible disagree on CRD channel / Cilium flag) | **Med** | Land CRD-channel + Cilium-flag changes in both bootstrap and Flux in one PR; gate (`make -C ci gate`). | +| **Thin docs / churn** (experimental API fields change across Gateway API/Cilium minors) | **Med** | Pin bundle version; Renovate watch; keep the CEC template in git history as a fallback recipe. | +| **Regressing freshly-fixed SSO** | **High** | Do nothing to `oidc` apps; canary + reversible filter-only changes; never touch the shared Gateway CEC directly. | + +**Global rollback:** every proposed change is additive at the HTTPRoute level and reverts by removing +the filter block. The nuclear fallback if native ExternalAuth proves flaky is the **already-proven +Option B** (per-app oauth2-proxy reverse-proxy, as Crafty uses today) — Gateway-API-native, no CRD +channel swap, just a `backendRef` to a per-app proxy. + +--- + +## Recommendation + +**DEFER.** Native ExternalAuth is the right long-term target and would let us delete both the dead CEC +template and (eventually) Crafty's sidecar in favor of one uniform, Gateway-API-native filter. But +**tonight it is not viable**: the enabling CRD channel isn't installed, the feature is experimental +and recently fail-open-buggy, it is undocumented in 1.20, and it would perturb the shared Gateway and +the just-fixed `enforce_policy_on_l7lb` egress that SSO now depends on. + +Concretely: +- **Do NOT** adopt now. **Keep** Crafty's working sidecar and leave the 5 forward-auth UIs as-is. +- **When revisiting** (next maintenance window, not tonight): execute **Phase 0** preconditions, + then a **pihole-only canary** (Phase 1). Only after a clean fail-closed browser proof, roll out + Phase 2 and retire the CEC template. +- **Cheaper interim if gating the 5 UIs becomes urgent before then:** adopt **Option B** (per-app + oauth2-proxy reverse-proxy, the Crafty pattern) — no experimental CRDs, fully reversible — rather + than the separate-VIP CEC or a rushed native-ExternalAuth switch. + +This resolves the DECISIONS-LOG open item: **native ExternalAuth is relevant and code-present under +1.20.1, but blocked on an experimental CRD-channel swap + maturity/fail-open concerns; the interim +workaround stays.** + +--- + +## Sources + +- Cilium v1.20.0 release notes — "External Authorization" / GEP-1494 (PR #45739): + https://github.com/cilium/cilium/releases/tag/v1.20.0 +- Cilium v1.20.1 release: https://github.com/cilium/cilium/releases/tag/v1.20.1 +- Isovalent — "Cilium 1.20: Gateway API ExternalAuth…": https://isovalent.com/blog/post/cilium-1-20/ +- GEP-1494 (HTTP Auth in Gateway API), status **Experimental**: + https://gateway-api.sigs.k8s.io/geps/gep-1494/ +- Issue #47178 "Basic externalauth setup fails open": https://github.com/cilium/cilium/issues/47178 +- Fix PR #47265 (backported to 1.20): https://github.com/cilium/cilium/pull/47265 +- Issue #45704 (GEP-1494 filter support, closed as dup of the implementing PR): + https://github.com/cilium/cilium/issues/45704 +- Cilium 1.20.1 Gateway API support page (no ExternalAuth mention — feature undocumented in 1.20): + https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/ +- Live cluster (read-only): Cilium `v1.20.1`; HTTPRoute CRD `channel: standard`, bundle `v1.6.1`, + `filters[].type` enum lacks `ExternalAuth`; `cilium-config` has no external-auth flag. +- Repo: `platform/oauth2-proxy/forward-auth-cec.yaml` (+ README), `platform/crafty/deployment.yaml` + (+ README), `catalog/render.py` (`AUTH_MODES`, `_auth_annotations`, keycloak-egress emitter), + commit `68690e5` (CVE-2025-30162 / `enforce_policy_on_l7lb` egress fix). + +