Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
380ea98a38c84273af178a0b30e2069904d576ed
home-network-k3s-impl/docs/plans/cilium-native-externalauth-evaluation.md
T
jakman85andClaude Opus 4.8 b745bab26b docs(plan): evaluate Cilium 1.20.1 native Gateway ExternalAuth vs oauth2-proxy 
Research + implementation plan (no live change) on whether Cilium 1.20.1's
native Gateway API ExternalAuth filter (GEP-1494) can replace the un-wired
forward-auth CEC template and Crafty's oauth2-proxy sidecar.

Verdict: DEFER. The filter exists in 1.20.1 code (shipped 1.20.0, PR #45739) but
is EXPERIMENTAL-channel Gateway API; the cluster runs the standard-channel CRDs
(verified: HTTPRoute filters[].type enum lacks ExternalAuth), so it is not even
installable today. Also: fail-open bug (#47178, fixed #47265) in a brand-new
security filter, no 1.20 docs, and the ext_authz hop would hit the same
enforce_policy_on_l7lb hairpin re-evaluation just fixed for Keycloak (68690e5).

Plan is phased/reversible (pihole canary first) and keeps the working sidecar.
Resolves the DECISIONS-LOG open item on ext_authz workaround relevance under 1.20.1.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-31 01:19:36 -05:00

19 KiB
Raw Blame History

Evaluation: Cilium 1.20.1 native Gateway API ExternalAuth vs. the oauth2-proxy design

Status: Research + plan only — NO live change proposed for tonight. Reviewable evaluation. Author: WS-NET/WS-SEC investigation (Claude Opus 4.8), 2026-08-31. Scope: Whether Cilium 1.20.1's native Gateway API ExternalAuth filter (GEP-1494) can replace (a) the hand-authored, un-wired forward-auth CiliumClusterwideEnvoyConfig (platform/oauth2-proxy/forward-auth-cec.yaml) and (b) Crafty's per-pod oauth2-proxy sidecar. Answers the open item from DECISIONS-LOG (2026-08-30): "the Cilium ext_authz/forward-auth workaround relevance under 1.20.1."

TL;DR — Verdict: DEFER (partial-feasible, not viable as-is tonight)

Native ExternalAuth exists in the Cilium 1.20.1 codebase and could eventually replace both mechanisms with a single, Gateway-API-native, per-route filter. But it is NOT usable on this cluster in its current state, and adopting it is a non-trivial, elevated-risk change to the exact shared Gateway that SSO was just stabilized on. Recommendation: keep the current design, do not adopt now. Track native ExternalAuth as the long-term target and revisit after the blocking conditions below clear.

Three hard blockers, in order of severity:

  1. The cluster's Gateway API CRDs are the standard channel — the ExternalAuth HTTPRoute filter is not even installable. Verified live: the httproutes.gateway.networking.k8s.io CRD is gateway.networking.k8s.io/channel: standard, bundle v1.6.1, and its filters[].type enum is exactly [RequestHeaderModifier, ResponseHeaderModifier, RequestMirror, RequestRedirect, URLRewrite, ExtensionRef, CORS] — ExternalAuth is absent. GEP-1494 is an experimental Gateway API feature; using it requires swapping the cluster-wide Gateway API CRDs to the experimental channel bundle. That is a breaking, cluster-wide CRD change, and it must be kept in lock-step with the Ansible-bootstrap ↔ Flux parity contract.
  2. The feature is young and shipped with a fail-open security bug. ExternalAuth landed in Cilium 1.20.0 (2026-07-29, PR #45739). Issue #47178 "Basic externalauth setup fails open" was filed against 1.20.0-rc.0 — the ext_authz filter silently produced no Envoy entry and let unauthenticated traffic through. The fix (PR #47265) was backported to the 1.20 branch (~2026-07-21) and should be in 1.20.0/1.20.1, but a fail-open regression in a brand-new security-critical filter is exactly the failure mode R-IDP-02 forbids. Confidence that today's filter is robust is low, and documentation is thin (Cilium ships no external-auth docs page in 1.20 — the dedicated doc URL 404s; the stable Gateway API doc page makes no mention of it).
  3. enforce_policy_on_l7lb (the CVE-2025-30162 hairpin behavior) will bite the ext_authz hop too. Commit 68690e5 just fixed cluster-wide SSO breakage: after the 1.16.5→1.20.1 jump, the shared Gateway's L7 proxy re-evaluates a pod's egress policy against the real backend identity when it hairpins (enforce_policy_on_l7lb), so every NetworkPolicy'd app needed an explicit toEndpoints egress allow to the Keycloak backend. A Gateway-level ExternalAuth filter makes the Gateway's Envoy hairpin to the oauth2-proxy backend cluster for the auth check — the identical class of policy re-evaluation. Expect to need new egress/identity allowances for the ext_authz path, and to re-validate it does not regress the freshly-fixed Keycloak egress.

Given SSO "was just painfully fixed," the cost/benefit does not favor touching it now.

Current state (verified — live cluster + repo)

Cluster: Cilium v1.20.1 (quay.io/cilium/cilium:v1.20.1), Gateway API CRDs bundle v1.6.1 standard channel, k3s v1.36.4. Shared Gateway gateway/shared (class cilium, VIP 192.168.1.53, Programmed), backed by the machine-owned CiliumEnvoyConfig gateway/cilium-gateway-shared (the only CEC/CCEC on the cluster). cilium-config has enable-gateway-api: true and no external-auth flag (mesh-auth-* is the unrelated SPIFFE mutual-auth feature and is disabled).

Auth design today (three mechanisms, one of them dead):

Mechanism Where Apps Status
Native OIDC (auth.mode: oidc) in-app OIDC client → Keycloak gitea, nextcloud, semaphore, grafana(native) Live, working. Not in scope to change.
Native/other app-specific seerr (own OIDC), plex Live. Out of scope.
forward-auth CEC (auth.mode: forward-auth) platform/oauth2-proxy/forward-auth-cec.yaml — hand-authored CiliumClusterwideEnvoyConfig with an ext_authz HTTP filter → oauth2-proxy /oauth2/auth jackett, pihole, qbittorrent, radarr, sonarr NOT DEPLOYED. A TEMPLATE, not referenced by platform/oauth2-proxy/kustomization.yaml. Proven in a throwaway envoy pod (2026-08-28) but never wired. These 5 UIs are currently reachable on the shared Gateway un-gated.
oauth2-proxy sidecar (per-pod reverse-proxy, "Option B") platform/crafty/deployment.yaml container oauth2-proxy (v7.8.1, :4180 → https://127.0.0.1:8443) crafty Live, working. In-data-path fail-closed gate.

The renderer contract (catalog/render.py): auth.mode ∈ {oidc, forward-auth, native, none} (AUTH_MODES, line 53); default is forward-auth for any app with ingressRoutes else none (lines 76-77). _auth_annotations() (line 605) stamps homelab.keithjackson.dev/auth-mode + /auth-groups onto the HTTPRoute for oidc/forward-auth, and pods carry an auth-mode label enforced by a Kyverno require-auth-mode-label policy (R-IDP-02). Since commit 68690e5, the renderer also emits a toEndpoints egress allow to the Keycloak backend for oidc/forward-auth apps (line 539). No renderer field today expresses an ext_authz backendRef — the forward-auth annotations are consumed by the (un-wired) platform enforcement, not by a Gateway-API filter.

Why the CEC exists / why it's not on the Gateway (per its own header + README): on Cilium 1.16.5 there was no supported hook to inject ext_authz into the machine-regenerated shared-Gateway Envoy program, and native ExternalAuth "ships in NO released Cilium version yet (PoC only)." That premise is now partially stale — see below — which is precisely why this evaluation exists.

Does Cilium 1.20.1 support native Gateway API ExternalAuth? (research)

Yes, in the code — as an experimental Gateway API feature that is not enabled on this cluster.

  • Shipped in 1.20.0. The v1.20.0 release notes list: "🔐 External Authorization: HTTPRoute requests can now be authenticated and authorized through an external service before they reach the application, using the Gateway API ExternalAuth filter from GEP-1494." (PR cilium/cilium#45739). Isovalent's 1.20 launch post headlines "Gateway API ExternalAuth."
  • Shape (GEP-1494 experimental filter). An HTTPRoute rule gains a filter type: ExternalAuth with externalAuth.protocol: GRPC|HTTP, a backendRef to the auth service, and grpc.allowedHeaders / http.{path,allowedHeaders,allowedResponseHeaders} / forwardBody. It maps onto Envoy's ext_authz protocol — the same mechanism the hand-authored CEC uses, but declared per-route on the standard shared Gateway instead of a separate listener. Example: 
    apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
spec:
  rules:
    - filters:
        - type: ExternalAuth
          externalAuth:
            protocol: GRPC            # or HTTP (oauth2-proxy /oauth2/auth is HTTP)
            backendRef: { kind: Service, name: oauth2-proxy, port: 80 }
            http: { path: /oauth2/auth, allowedHeaders: [cookie, authorization] }
      backendRefs: [{ name: <app>, port: <port> }]
  • It is EXPERIMENTAL-channel. GEP-1494 status is Experimental (gateway-api.sigs.k8s.io/geps/gep-1494). Its fields live only in the Gateway API experimental CRD bundle. This cluster runs the standard channel (verified), so the filter type is not present in the HTTPRoute schema — a manifest using it would be rejected at admission today.
  • Maturity risk. Fail-open bug #47178 → fix #47265 (backported to 1.20, ~2026-07-21). Cilium ships no dedicated ExternalAuth documentation page in 1.20 (the /en/v1.20/.../external-auth/ and /en/latest/.../external-auth/ URLs both 404; the stable Gateway API support page does not mention it). Feature is real but young and thinly documented.

Bottom line: the CEC/sidecar READMEs' claim that native ExternalAuth "ships in no released Cilium version" is now outdated — it ships in 1.20.x — but it is experimental, off by default here (wrong CRD channel), and not battle-tested.

Would it cleanly replace the two mechanisms?

(a) The un-wired forward-auth CEC — YES, in principle, and more cleanly. This is the strongest case for native ExternalAuth: the CEC is a hand-authored TLS listener + routes + EDS clusters on a separate VIP that requires moving hosts off the shared Gateway and repointing DNS. A native ExternalAuth filter would instead be one filter block on each app's existing HTTPRoute on the existing shared Gateway/VIP/DNS — no second ingress, no DNS churn, no hand-authored TLS. Since the CEC is not deployed, there is no rip-out risk here: adopting native ExternalAuth for the 5 forward-auth apps (jackett/pihole/qbittorrent/radarr/sonarr) would complete an unfinished story rather than replace a working one. The renderer already carries the auth-mode/auth-groups signal; it would need a new emitter that turns auth.mode: forward-auth into the HTTPRoute ExternalAuth filter (see plan).

(b) Crafty's sidecar — PARTIAL, and lower value. Native ExternalAuth could drop the per-pod oauth2-proxy container and gate crafty.home at the Gateway like the others. But Crafty's UI is HTTPS self-signed on :8443; the sidecar terminates that in-pod (--ssl-upstream-insecure-skip- verify). Moving the gate to the Gateway means the shared Gateway's Envoy must speak HTTPS to Crafty's self-signed :8443 upstream — requiring a BackendTLSPolicy with insecureSkipVerify (or a real cert), and the crafty Service would target :8443 directly. That is net-new Gateway plumbing to remove a small, working, fail-closed sidecar. Low priority; do Crafty last, if at all.

Native ExternalAuth cannot and should not touch the oidc apps (gitea/nextcloud/semaphore/ grafana) — those use in-app OIDC and are out of scope.

Proposed migration plan (do NOT execute now — for review)

Phased, reversible, lowest-blast-radius first. Precondition gates P1-P3 must clear before ANY cluster change. Each step names the files it touches.

Phase 0 — Preconditions (research/verify only, no cluster change)

  • P1. Confirm the fail-open fix is in 1.20.1. Verify PR #47265 shipped in the running v1.20.1 (changelog / git tag --contains), and reproduce a fail-closed ext_authz on a throwaway route (the 2026-08-28 CEC proof, redone with the native filter on a scratch HTTPRoute) BEFORE trusting it for R-IDP-02.
  • P2. Decide the Gateway API channel swap. Adopting GEP-1494 requires the experimental CRD bundle cluster-wide. Assess blast radius (every Gateway/HTTPRoute revalidated against experimental schemas) and reconcile the Ansible-bootstrap ↔ Flux parity contract — the Ansible bootstrap that installs Gateway API CRDs must install the same channel. Files: wherever the Gateway API CRD bundle is declared for both bootstrap and Flux (search gateway.networking.k8s.io CRD source; platform/gateway/ + the Ansible bootstrap CRD step). This is the single biggest change and its own reviewable sub-plan.
  • P3. Confirm the Cilium enablement path. Determine whether Cilium honors the ExternalAuth filter on experimental-CRD presence alone or needs an agent/helm flag; if a flag exists it is a [parity] value in platform/cilium/helmrelease.yaml and the Ansible Cilium values.

Phase 1 — One canary app (pihole), additive, reversible

  • Install experimental Gateway API CRDs (P2) + any Cilium flag (P3).
  • Add the ExternalAuth filter to pihole's HTTPRoute only, backendRef → oauth2-proxy Service, http.path: /oauth2/auth, header allow-list mirroring the CEC's proven stanza. Keep failureMode/deny semantics fail-closed. Files: catalog/pihole.yaml (unchanged intent) + a new renderer emitter in catalog/render.py (_httproute() path) that renders the filter from auth.mode: forward-auth; re-render pihole golden. Add the ext_authz egress allow to the oauth2-proxy backend for the app (the enforce_policy_on_l7lb mitigation, mirroring the Keycloak fix in render.py).
  • Browser test (the human): incognito pihole.home → 302 to Keycloak → login platform-admins → PiHole UI; non-platform-admins → 403; oauth2-proxy down → denied, never the bare UI.
  • Rollback: delete the filter block (revert the HTTPRoute) → back to un-gated shared Gateway. No data, no DNS, no VIP involved.

Phase 2 — Roll out to the remaining forward-auth apps

  • Repeat for jackett, qbittorrent, radarr, sonarr via the renderer (they already declare auth.mode: forward-auth; the emitter does it uniformly). Re-render 4 goldens.
  • Delete platform/oauth2-proxy/forward-auth-cec.yaml and its README section describing the separate-VIP interim — now obsolete. (It was never wired, so this is doc/template cleanup.)

Phase 3 — Crafty (optional, last)

  • Add ExternalAuth to crafty's HTTPRoute; point the crafty Service at :8443; add a BackendTLSPolicy for the self-signed upstream; remove the oauth2-proxy sidecar from platform/crafty/deployment.yaml and its SOPS creds. Verify game-port L4 path (192.168.1.55) untouched. Higher effort for marginal gain — defer unless consolidating.

Risks & rollback

Risk Severity Mitigation / Rollback
Fail-open regression (a new/edge ext_authz bug lets traffic through un-authed — R-IDP-02 violation) High P1 reproduction on a scratch route before any real app; canary (pihole) first with explicit browser deny-path test; keep un-gated-but-internal apps behind LAN until proven.
Experimental-channel CRD swap breaks existing Gateways/HTTPRoutes High Treat P2 as its own reviewed change; snapshot/backup CRDs; parity with Ansible bootstrap; validate all 16 existing HTTPRoutes still Programmed after the swap.
enforce_policy_on_l7lb blocks the ext_authz hop (Gateway hairpins to oauth2-proxy backend; NetworkPolicy'd apps 403 the auth check — the CVE-2025-30162 pattern from commit 68690e5) Med-High Emit a toEndpoints egress allow to the oauth2-proxy backend for forward-auth apps, exactly as done for Keycloak; re-validate the Keycloak egress is not regressed.
Parity-contract drift (Flux vs Ansible disagree on CRD channel / Cilium flag) Med Land CRD-channel + Cilium-flag changes in both bootstrap and Flux in one PR; gate (make -C ci gate).
Thin docs / churn (experimental API fields change across Gateway API/Cilium minors) Med Pin bundle version; Renovate watch; keep the CEC template in git history as a fallback recipe.
Regressing freshly-fixed SSO High Do nothing to oidc apps; canary + reversible filter-only changes; never touch the shared Gateway CEC directly.

Global rollback: every proposed change is additive at the HTTPRoute level and reverts by removing the filter block. The nuclear fallback if native ExternalAuth proves flaky is the already-proven Option B (per-app oauth2-proxy reverse-proxy, as Crafty uses today) — Gateway-API-native, no CRD channel swap, just a backendRef to a per-app proxy.

Recommendation

DEFER. Native ExternalAuth is the right long-term target and would let us delete both the dead CEC template and (eventually) Crafty's sidecar in favor of one uniform, Gateway-API-native filter. But tonight it is not viable: the enabling CRD channel isn't installed, the feature is experimental and recently fail-open-buggy, it is undocumented in 1.20, and it would perturb the shared Gateway and the just-fixed enforce_policy_on_l7lb egress that SSO now depends on.

Concretely:

  • Do NOT adopt now. Keep Crafty's working sidecar and leave the 5 forward-auth UIs as-is.
  • When revisiting (next maintenance window, not tonight): execute Phase 0 preconditions, then a pihole-only canary (Phase 1). Only after a clean fail-closed browser proof, roll out Phase 2 and retire the CEC template.
  • Cheaper interim if gating the 5 UIs becomes urgent before then: adopt Option B (per-app oauth2-proxy reverse-proxy, the Crafty pattern) — no experimental CRDs, fully reversible — rather than the separate-VIP CEC or a rushed native-ExternalAuth switch.

This resolves the DECISIONS-LOG open item: native ExternalAuth is relevant and code-present under 1.20.1, but blocked on an experimental CRD-channel swap + maturity/fail-open concerns; the interim workaround stays.

Sources

Reference in new issue View Git Blame Copy Permalink