home-network-k3s-impl/docs/plans/cilium-native-externalauth-evaluation.md

# Evaluation: Cilium 1.20.1 native Gateway API ExternalAuth vs. the oauth2-proxy design

**Status:** Research + plan only — NO live change proposed for tonight. Reviewable evaluation.
**Author:** WS-NET/WS-SEC investigation (Claude Opus 4.8), 2026-08-31.
**Scope:** Whether Cilium 1.20.1's native Gateway API `ExternalAuth` filter (GEP-1494) can replace
(a) the hand-authored, **un-wired** forward-auth `CiliumClusterwideEnvoyConfig`
(`platform/oauth2-proxy/forward-auth-cec.yaml`) and (b) Crafty's per-pod oauth2-proxy **sidecar**.
**Answers the open item** from DECISIONS-LOG (2026-08-30): *"the Cilium ext_authz/forward-auth
workaround relevance under 1.20.1."*

---

## TL;DR — Verdict: **DEFER (partial-feasible, not viable as-is tonight)**

Native `ExternalAuth` **exists in the Cilium 1.20.1 codebase** and *could* eventually replace both
mechanisms with a single, Gateway-API-native, per-route filter. **But it is NOT usable on this
cluster in its current state**, and adopting it is a non-trivial, elevated-risk change to the exact
shared Gateway that SSO was just stabilized on. Recommendation: **keep the current design, do not
adopt now.** Track native ExternalAuth as the long-term target and revisit after the blocking
conditions below clear.

Three hard blockers, in order of severity:

1. **The cluster's Gateway API CRDs are the `standard` channel — the `ExternalAuth` HTTPRoute filter
   is not even installable.** Verified live: the `httproutes.gateway.networking.k8s.io` CRD is
   `gateway.networking.k8s.io/channel: standard`, bundle `v1.6.1`, and its `filters[].type` enum is
   exactly `[RequestHeaderModifier, ResponseHeaderModifier, RequestMirror, RequestRedirect,
   URLRewrite, ExtensionRef, CORS]` — **`ExternalAuth` is absent**. GEP-1494 is an **experimental**
   Gateway API feature; using it requires swapping the cluster-wide Gateway API CRDs to the
   **experimental channel** bundle. That is a breaking, cluster-wide CRD change, and it must be kept
   in lock-step with the Ansible-bootstrap ↔ Flux **parity contract**.
2. **The feature is young and shipped with a fail-**open** security bug.** `ExternalAuth` landed in
   Cilium **1.20.0** (2026-07-29, PR #45739). Issue **#47178 "Basic externalauth setup fails open"**
   was filed against 1.20.0-rc.0 — the ext_authz filter silently produced no Envoy entry and let
   **unauthenticated traffic through**. The fix (PR #47265) was backported to the 1.20 branch
   (~2026-07-21) and should be in 1.20.0/1.20.1, but a fail-**open** regression in a brand-new
   security-critical filter is exactly the failure mode R-IDP-02 forbids. Confidence that today's
   filter is robust is low, and documentation is thin (Cilium ships **no** `external-auth` docs page
   in 1.20 — the dedicated doc URL 404s; the stable Gateway API doc page makes no mention of it).
3. **`enforce_policy_on_l7lb` (the CVE-2025-30162 hairpin behavior) will bite the ext_authz hop too.**
   Commit `68690e5` just fixed cluster-wide SSO breakage: after the 1.16.5→1.20.1 jump, the shared
   Gateway's L7 proxy re-evaluates a pod's **egress policy against the *real backend* identity** when
   it hairpins (`enforce_policy_on_l7lb`), so every NetworkPolicy'd app needed an explicit
   `toEndpoints` egress allow to the Keycloak backend. A Gateway-level `ExternalAuth` filter makes the
   Gateway's Envoy hairpin to the **oauth2-proxy** backend cluster for the auth check — the identical
   class of policy re-evaluation. Expect to need new egress/identity allowances for the ext_authz
   path, and to re-validate it does not regress the freshly-fixed Keycloak egress.

Given SSO "was just painfully fixed," the cost/benefit does not favor touching it now.

---

## Current state (verified — live cluster + repo)

**Cluster:** Cilium `v1.20.1` (`quay.io/cilium/cilium:v1.20.1`), Gateway API CRDs bundle `v1.6.1`
**standard** channel, k3s v1.36.4. Shared Gateway `gateway/shared` (class `cilium`, VIP
`192.168.1.53`, Programmed), backed by the machine-owned `CiliumEnvoyConfig
gateway/cilium-gateway-shared` (the only CEC/CCEC on the cluster). `cilium-config` has
`enable-gateway-api: true` and **no** external-auth flag (`mesh-auth-*` is the unrelated SPIFFE
mutual-auth feature and is disabled).

**Auth design today (three mechanisms, one of them dead):**

| Mechanism | Where | Apps | Status |
|---|---|---|---|
| **Native OIDC** (`auth.mode: oidc`) | in-app OIDC client → Keycloak | gitea, nextcloud, semaphore, grafana(native) | **Live, working.** Not in scope to change. |
| **Native/other** | app-specific | seerr (own OIDC), plex | Live. Out of scope. |
| **forward-auth CEC** (`auth.mode: forward-auth`) | `platform/oauth2-proxy/forward-auth-cec.yaml` — hand-authored `CiliumClusterwideEnvoyConfig` with an `ext_authz` HTTP filter → oauth2-proxy `/oauth2/auth` | jackett, pihole, qbittorrent, radarr, sonarr | **NOT DEPLOYED.** A TEMPLATE, **not** referenced by `platform/oauth2-proxy/kustomization.yaml`. Proven in a throwaway envoy pod (2026-08-28) but never wired. These 5 UIs are currently reachable on the shared Gateway **un-gated**. |
| **oauth2-proxy sidecar** (per-pod reverse-proxy, "Option B") | `platform/crafty/deployment.yaml` container `oauth2-proxy` (`v7.8.1`, `:4180` → `https://127.0.0.1:8443`) | crafty | **Live, working.** In-data-path fail-closed gate. |

**The renderer contract (`catalog/render.py`):** `auth.mode ∈ {oidc, forward-auth, native, none}`
(`AUTH_MODES`, line 53); default is `forward-auth` for any app with `ingressRoutes` else `none`
(lines 76-77). `_auth_annotations()` (line 605) stamps
`homelab.keithjackson.dev/auth-mode` + `/auth-groups` onto the HTTPRoute for `oidc`/`forward-auth`,
and pods carry an `auth-mode` label enforced by a Kyverno `require-auth-mode-label` policy
(R-IDP-02). Since commit `68690e5`, the renderer also emits a `toEndpoints` egress allow to the
Keycloak backend for `oidc`/`forward-auth` apps (line 539). **No renderer field today expresses an
ext_authz `backendRef`** — the forward-auth annotations are consumed by the (un-wired) platform
enforcement, not by a Gateway-API filter.

**Why the CEC exists / why it's not on the Gateway (per its own header + README):** on Cilium 1.16.5
there was no supported hook to inject `ext_authz` into the machine-regenerated shared-Gateway Envoy
program, and native ExternalAuth "ships in NO released Cilium version yet (PoC only)." That premise
is now **partially stale** — see below — which is precisely why this evaluation exists.

---

## Does Cilium 1.20.1 support native Gateway API ExternalAuth? (research)

**Yes, in the code — as an experimental Gateway API feature that is not enabled on this cluster.**

- **Shipped in 1.20.0.** The v1.20.0 release notes list: *"🔐 External Authorization: HTTPRoute
  requests can now be authenticated and authorized through an external service before they reach the
  application, using the Gateway API `ExternalAuth` filter from GEP-1494."* (PR
  [cilium/cilium#45739](https://github.com/cilium/cilium/pull/45739)). Isovalent's 1.20 launch post
  headlines "Gateway API ExternalAuth."
- **Shape (GEP-1494 experimental filter).** An HTTPRoute rule gains a filter
  `type: ExternalAuth` with `externalAuth.protocol: GRPC|HTTP`, a `backendRef` to the auth service,
  and `grpc.allowedHeaders` / `http.{path,allowedHeaders,allowedResponseHeaders}` / `forwardBody`.
  It maps onto Envoy's `ext_authz` protocol — the **same** mechanism the hand-authored CEC uses,
  but declared per-route on the standard shared Gateway instead of a separate listener. Example:
  ```yaml
  apiVersion: gateway.networking.k8s.io/v1
  kind: HTTPRoute
  spec:
    rules:
      - filters:
          - type: ExternalAuth
            externalAuth:
              protocol: GRPC            # or HTTP (oauth2-proxy /oauth2/auth is HTTP)
              backendRef: { kind: Service, name: oauth2-proxy, port: 80 }
              http: { path: /oauth2/auth, allowedHeaders: [cookie, authorization] }
        backendRefs: [{ name: <app>, port: <port> }]
  ```
- **It is EXPERIMENTAL-channel.** GEP-1494 status is **Experimental**
  ([gateway-api.sigs.k8s.io/geps/gep-1494](https://gateway-api.sigs.k8s.io/geps/gep-1494/)). Its
  fields live only in the Gateway API **experimental** CRD bundle. **This cluster runs the `standard`
  channel** (verified), so the filter type is not present in the HTTPRoute schema — a manifest using
  it would be rejected at admission today.
- **Maturity risk.** Fail-open bug [#47178](https://github.com/cilium/cilium/issues/47178) → fix
  [#47265](https://github.com/cilium/cilium/pull/47265) (backported to 1.20, ~2026-07-21). Cilium
  ships **no dedicated ExternalAuth documentation page** in 1.20 (the `/en/v1.20/.../external-auth/`
  and `/en/latest/.../external-auth/` URLs both 404; the stable Gateway API support page does not
  mention it). Feature is real but young and thinly documented.

**Bottom line:** the CEC/sidecar READMEs' claim that native ExternalAuth "ships in no released Cilium
version" is now outdated — it ships in 1.20.x — **but** it is experimental, off by default here
(wrong CRD channel), and not battle-tested.

---

## Would it cleanly replace the two mechanisms?

**(a) The un-wired forward-auth CEC — YES, in principle, and more cleanly.** This is the strongest
case for native ExternalAuth: the CEC is a hand-authored TLS listener + routes + EDS clusters on a
*separate* VIP that requires moving hosts off the shared Gateway and repointing DNS. A native
`ExternalAuth` filter would instead be one filter block on each app's **existing** HTTPRoute on the
**existing** shared Gateway/VIP/DNS — no second ingress, no DNS churn, no hand-authored TLS. Since
the CEC is not deployed, there is **no rip-out risk** here: adopting native ExternalAuth for the 5
forward-auth apps (jackett/pihole/qbittorrent/radarr/sonarr) would *complete* an unfinished story
rather than replace a working one. The renderer already carries the `auth-mode`/`auth-groups` signal;
it would need a new emitter that turns `auth.mode: forward-auth` into the HTTPRoute `ExternalAuth`
filter (see plan).

**(b) Crafty's sidecar — PARTIAL, and lower value.** Native ExternalAuth could drop the per-pod
oauth2-proxy container and gate `crafty.home` at the Gateway like the others. **But** Crafty's UI is
HTTPS **self-signed on :8443**; the sidecar terminates that in-pod (`--ssl-upstream-insecure-skip-
verify`). Moving the gate to the Gateway means the shared Gateway's Envoy must speak HTTPS to Crafty's
self-signed `:8443` upstream — requiring a `BackendTLSPolicy` with `insecureSkipVerify` (or a real
cert), and the `crafty` Service would target `:8443` directly. That is net-new Gateway plumbing to
remove a small, **working**, fail-closed sidecar. Low priority; do Crafty last, if at all.

Native ExternalAuth **cannot** and **should not** touch the `oidc` apps (gitea/nextcloud/semaphore/
grafana) — those use in-app OIDC and are out of scope.

---

## Proposed migration plan (do NOT execute now — for review)

Phased, reversible, lowest-blast-radius first. **Precondition gates P1-P3 must clear before ANY
cluster change.** Each step names the files it touches.

### Phase 0 — Preconditions (research/verify only, no cluster change)
- **P1. Confirm the fail-open fix is in 1.20.1.** Verify PR #47265 shipped in the running `v1.20.1`
  (changelog / `git tag --contains`), and reproduce a **fail-closed** ext_authz on a throwaway route
  (the 2026-08-28 CEC proof, redone with the *native filter* on a scratch HTTPRoute) BEFORE trusting
  it for R-IDP-02.
- **P2. Decide the Gateway API channel swap.** Adopting GEP-1494 requires the **experimental** CRD
  bundle cluster-wide. Assess blast radius (every Gateway/HTTPRoute revalidated against experimental
  schemas) and reconcile the **Ansible-bootstrap ↔ Flux parity contract** — the Ansible bootstrap
  that installs Gateway API CRDs must install the *same* channel. Files: wherever the Gateway API CRD
  bundle is declared for both bootstrap and Flux (search `gateway.networking.k8s.io` CRD source;
  `platform/gateway/` + the Ansible bootstrap CRD step). This is the single biggest change and its
  own reviewable sub-plan.
- **P3. Confirm the Cilium enablement path.** Determine whether Cilium honors the `ExternalAuth`
  filter on experimental-CRD presence alone or needs an agent/helm flag; if a flag exists it is a
  `[parity]` value in `platform/cilium/helmrelease.yaml` and the Ansible Cilium values.

### Phase 1 — One canary app (pihole), additive, reversible
- Install experimental Gateway API CRDs (P2) + any Cilium flag (P3).
- Add the `ExternalAuth` filter to **pihole's** HTTPRoute only, `backendRef` → `oauth2-proxy`
  Service, `http.path: /oauth2/auth`, header allow-list mirroring the CEC's proven stanza. Keep
  `failureMode`/deny semantics fail-**closed**.
  Files: `catalog/pihole.yaml` (unchanged intent) + a new renderer emitter in
  `catalog/render.py` (`_httproute()` path) that renders the filter from `auth.mode: forward-auth`;
  re-render pihole golden. Add the ext_authz **egress allow** to the oauth2-proxy backend for the app
  (the `enforce_policy_on_l7lb` mitigation, mirroring the Keycloak fix in `render.py`).
- **Browser test** (the human): incognito `pihole.home` → 302 to Keycloak → login `platform-admins`
  → PiHole UI; non-`platform-admins` → 403; oauth2-proxy down → **denied**, never the bare UI.
- **Rollback:** delete the filter block (revert the HTTPRoute) → back to un-gated shared Gateway. No
  data, no DNS, no VIP involved.

### Phase 2 — Roll out to the remaining forward-auth apps
- Repeat for jackett, qbittorrent, radarr, sonarr via the renderer (they already declare
  `auth.mode: forward-auth`; the emitter does it uniformly). Re-render 4 goldens.
- **Delete `platform/oauth2-proxy/forward-auth-cec.yaml`** and its README section describing the
  separate-VIP interim — now obsolete. (It was never wired, so this is doc/template cleanup.)

### Phase 3 — Crafty (optional, last)
- Add `ExternalAuth` to `crafty`'s HTTPRoute; point the `crafty` Service at `:8443`; add a
  `BackendTLSPolicy` for the self-signed upstream; **remove the oauth2-proxy sidecar** from
  `platform/crafty/deployment.yaml` and its SOPS creds. Verify game-port L4 path (192.168.1.55)
  untouched. Higher effort for marginal gain — defer unless consolidating.

---

## Risks & rollback

| Risk | Severity | Mitigation / Rollback |
|---|---|---|
| **Fail-open regression** (a new/edge ext_authz bug lets traffic through un-authed — R-IDP-02 violation) | **High** | P1 reproduction on a scratch route before any real app; canary (pihole) first with explicit browser deny-path test; keep un-gated-but-internal apps behind LAN until proven. |
| **Experimental-channel CRD swap breaks existing Gateways/HTTPRoutes** | **High** | Treat P2 as its own reviewed change; snapshot/backup CRDs; parity with Ansible bootstrap; validate all 16 existing HTTPRoutes still Programmed after the swap. |
| **`enforce_policy_on_l7lb` blocks the ext_authz hop** (Gateway hairpins to oauth2-proxy backend; NetworkPolicy'd apps 403 the auth check — the CVE-2025-30162 pattern from commit `68690e5`) | **Med-High** | Emit a `toEndpoints` egress allow to the oauth2-proxy backend for forward-auth apps, exactly as done for Keycloak; re-validate the Keycloak egress is not regressed. |
| **Parity-contract drift** (Flux vs Ansible disagree on CRD channel / Cilium flag) | **Med** | Land CRD-channel + Cilium-flag changes in both bootstrap and Flux in one PR; gate (`make -C ci gate`). |
| **Thin docs / churn** (experimental API fields change across Gateway API/Cilium minors) | **Med** | Pin bundle version; Renovate watch; keep the CEC template in git history as a fallback recipe. |
| **Regressing freshly-fixed SSO** | **High** | Do nothing to `oidc` apps; canary + reversible filter-only changes; never touch the shared Gateway CEC directly. |

**Global rollback:** every proposed change is additive at the HTTPRoute level and reverts by removing
the filter block. The nuclear fallback if native ExternalAuth proves flaky is the **already-proven
Option B** (per-app oauth2-proxy reverse-proxy, as Crafty uses today) — Gateway-API-native, no CRD
channel swap, just a `backendRef` to a per-app proxy.

---

## Recommendation

**DEFER.** Native ExternalAuth is the right long-term target and would let us delete both the dead CEC
template and (eventually) Crafty's sidecar in favor of one uniform, Gateway-API-native filter. But
**tonight it is not viable**: the enabling CRD channel isn't installed, the feature is experimental
and recently fail-open-buggy, it is undocumented in 1.20, and it would perturb the shared Gateway and
the just-fixed `enforce_policy_on_l7lb` egress that SSO now depends on.

Concretely:
- **Do NOT** adopt now. **Keep** Crafty's working sidecar and leave the 5 forward-auth UIs as-is.
- **When revisiting** (next maintenance window, not tonight): execute **Phase 0** preconditions,
  then a **pihole-only canary** (Phase 1). Only after a clean fail-closed browser proof, roll out
  Phase 2 and retire the CEC template.
- **Cheaper interim if gating the 5 UIs becomes urgent before then:** adopt **Option B** (per-app
  oauth2-proxy reverse-proxy, the Crafty pattern) — no experimental CRDs, fully reversible — rather
  than the separate-VIP CEC or a rushed native-ExternalAuth switch.

This resolves the DECISIONS-LOG open item: **native ExternalAuth is relevant and code-present under
1.20.1, but blocked on an experimental CRD-channel swap + maturity/fail-open concerns; the interim
workaround stays.**

---

## Sources

- Cilium v1.20.0 release notes — "External Authorization" / GEP-1494 (PR #45739):
  https://github.com/cilium/cilium/releases/tag/v1.20.0
- Cilium v1.20.1 release: https://github.com/cilium/cilium/releases/tag/v1.20.1
- Isovalent — "Cilium 1.20: Gateway API ExternalAuth…": https://isovalent.com/blog/post/cilium-1-20/
- GEP-1494 (HTTP Auth in Gateway API), status **Experimental**:
  https://gateway-api.sigs.k8s.io/geps/gep-1494/
- Issue #47178 "Basic externalauth setup fails open": https://github.com/cilium/cilium/issues/47178
- Fix PR #47265 (backported to 1.20): https://github.com/cilium/cilium/pull/47265
- Issue #45704 (GEP-1494 filter support, closed as dup of the implementing PR):
  https://github.com/cilium/cilium/issues/45704
- Cilium 1.20.1 Gateway API support page (no ExternalAuth mention — feature undocumented in 1.20):
  https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/
- Live cluster (read-only): Cilium `v1.20.1`; HTTPRoute CRD `channel: standard`, bundle `v1.6.1`,
  `filters[].type` enum lacks `ExternalAuth`; `cilium-config` has no external-auth flag.
- Repo: `platform/oauth2-proxy/forward-auth-cec.yaml` (+ README), `platform/crafty/deployment.yaml`
  (+ README), `catalog/render.py` (`AUTH_MODES`, `_auth_annotations`, keycloak-egress emitter),
  commit `68690e5` (CVE-2025-30162 / `enforce_policy_on_l7lb` egress fix).
</content>
</invoke>