jakman85andClaude Opus 5
fc7ec01628
Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.
The reason nothing alerted turns out not to be a missing rule. Measured on prod:
count by (__name__) ({__name__=~"oauth2_proxy.*"})
oauth2_proxy_requests_in_flight => 2
Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.
So alert on the condition that made the incident invisible, from families that ARE
scraped:
OAuth2ProxyGateUnobserved
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
unless
count by (namespace) (up{container="oauth2-proxy"})
kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.
Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
check from the app's own login from an XHR that silently got HTML, with the
2026-09-30 jackett trace as the worked example and the Loki queries to use while
there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.
Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
platform/oauth2-proxy
oauth2-proxy — forward-auth at the Gateway for apps without usable OIDC, authenticating against Keycloak.
- Governing ADR: ADR-0014 — IdP: Keycloak
- Spec: docs/15 — Identity & Access
- Serves: R-IDP-01..06 (esp. R-IDP-02: management UIs never unauthenticated)
- Owner: WS-SEC (Gateway wiring shared with WS-NET)
What this stub is
Flux HelmRelease for oauth2-proxy as the forward-auth provider for non-OIDC apps (PiHole, *arr UIs,
qBittorrent, Homepage, Prometheus, Alertmanager, Hubble UI — ADR-0014). Replica count is an overlay
knob (${OAUTH2_PROXY_REPLICAS}, R-ENV-02). Client/cookie secrets referenced by name,
SOPS-encrypted (WS-SEC).
Contract (SSO/auth + Gateway)
Pairs with ../keycloak to realize the "SSO/auth contract": a ServiceDefinition auth: forward-auth service is protected by oauth2-proxy at the Gateway (ext-auth); auth: oidc services go
native to Keycloak. Gateway ext-auth wiring is a WS-NET contract.
Enforcement (ext_authz forward-auth)
The authenticator (oauth2-proxy) is deployed and wired to Keycloak: provider=oidc,
oidc-issuer-url=.../realms/homelab, --reverse-proxy, --upstream=static://202,
--set-xauthrequest, cookie/redirect domain .home.keithjackson.dev, and — added here —
--allowed-groups=platform-admins (R-IDP-06 floor) + --skip-provider-button (302 straight to
Keycloak). The Keycloak oauth2-proxy client already carries the auth.home…/oauth2/callback
redirect URI and a groups claim with full.path=false (bare platform-admins) — no realm change
needed.
Mechanism. Forward-auth is an Envoy ext_authz HTTP filter that calls oauth2-proxy's
/oauth2/auth decision endpoint: 202 for a valid session in platform-admins (request passes to
the upstream, X-Auth-Request-* identity headers forwarded), 401/403 otherwise. With
failure_mode_allow: false the gateway fails closed — a denied or errored check never reaches the
upstream UI (R-IDP-02). The filter + oauth2-proxy cluster live in
forward-auth-cec.yaml.
Why this is NOT wired into the shared Gateway (Cilium 1.16.5 constraint)
The shared Gateway's Envoy program is a machine-owned, constantly-regenerated
CiliumEnvoyConfig (gateway/cilium-gateway-shared, ownerReferences->Gateway, regenerated on every
HTTPRoute change). Cilium 1.16.5 exposes no supported hook to inject an ext_authz filter into
that generated listener, and native Gateway-API ExternalAuth (GEP-1494) ships in no released Cilium
version yet (PoC only). So there is no clean per-route ext_authz on the shared Gateway today.
Options (PM decision):
| # | Approach | Pros | Cons |
|---|---|---|---|
| A | Interim self-authored CEC — a dedicated ext_authz HTTPS listener on its own L2 VIP (forward-auth-cec.yaml), protected hosts moved off the shared Gateway and DNS repointed. |
Works on 1.16.5 today; per-route opt-in; reuses the wildcard TLS via SDS; verified below. | Hand-authored TLS+routes; a 2nd ingress VIP + DNS; per-app maintenance; not Gateway-API-native. |
| B | oauth2-proxy in reverse-proxy mode per app — app HTTPRoute backendRef -> a per-app oauth2-proxy (--upstream=http://app.svc). |
Fully Gateway-API-native (just a backendRef swap); robust; no CEC. | One oauth2-proxy instance/config per app; more pods; not the single-proxy ext_authz model. |
| C | Upgrade Cilium to a release with native ExternalAuth (GEP-1494) and gate routes with an HTTPRoute filter. |
Cleanest, truly per-route, Crafty-friendly. | Not released yet; large jump from 1.16.5; must keep the Ansible-bootstrap/Flux parity contract in lock-step. |
Recommendation: ship the oauth2-proxy hardening now (done); adopt A as the interim enforcement for the sensitive UIs when the Phase-2 L2 VIP pool lands; track C as the long-term target and drop A once released.
Opt-in (per route)
A route is protected only by being listed as a virtualHost on the ext_authz listener — apps not
listed keep hitting the shared Gateway. To protect an app (pihole today; Crafty next): add its
Service + EDS cluster + virtualHost to forward-auth-cec.yaml, remove its HTTPRoute from the shared
Gateway, and repoint its DNS at the ext_authz VIP. The ServiceDefinition already declares
auth.mode: forward-auth + groups: [platform-admins] (rendered onto the HTTPRoute annotations) — the
opt-in signal. Ready to extend to Crafty and the other non-OIDC UIs (PiHole/*arr/qBittorrent/Homepage).
Proof (in-cluster, cilium-envoy v1.30.8, 2026-08-28)
A throwaway standalone cilium-envoy pod ran the exact ext_authz stanza from forward-auth-cec.yaml
against the live oauth2-proxy Service, with a no-filter control route on the same backend:
- Control (
:8081, no ext_authz) -> 200CONTROL-OK. - Protected (
:8080, ext_authz), unauthenticated -> 403; body is oauth2-proxy's sign-in page, not the upstream (UPSTREAM-OKnever served). oauth2-proxy's own access log recorded the/oauth2/authcall from the envoy pod.failure_mode_allow: falseconfirmed fail-closed.
This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on this exact Envoy. The
allow path (valid platform-admins session -> 202 -> upstream) needs a real browser round-trip.
Browser test (for the human, once forward-auth-cec.yaml is deployed)
- Incognito ->
https://pihole.home.keithjackson.dev-> expect a 302 tohttps://keycloak.home.keithjackson.dev/realms/homelab/...(not the PiHole UI). - Log in as a platform-admins user (MFA/OTP enforced) -> land back on the PiHole UI.
- Log in as a non-platform-admins user -> expect 403 (authenticated but not authorized).
- Confirm the session cookie is
.home.keithjackson.dev-scoped (single sign-on across the other protected UIs).
Observability: what this component can and cannot see (measured 2026-09-30)
servicemonitor.yaml scrapes this namespace's deployment and nothing else. On prod that is
two targets, job="oauth2-proxy", both up=1. The complete set of oauth2-proxy-specific series
in the TSDB is:
count by (__name__) ({__name__=~"oauth2_proxy.*"})
oauth2_proxy_requests_in_flight => 2
The per-app gate sidecars are not scraped, and there is nothing to scrape. Six apps run an
oauth2-proxy container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
each with --http-address=0.0.0.0:4180 and no --metrics-address, so oauth2-proxy's metrics
listener never starts. They declare only a http:4180 container port, no metrics port exists on
their Services, and kubectl get servicemonitors,podmonitors -A selects none of them. Any alert
written against oauth2_proxy_requests_total{namespace="jackett"} would be structurally inert.
This matters because the gate's characteristic failure is invisible by construction: an
unauthenticated XHR gets a 302 to Keycloak rather than a 401, the browser follows it
against a live SSO session with no prompt, and the call resolves 200 carrying an HTML page
where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
into a 5-minute session, while jackett-down sat at 1 replica available the whole time.
auth-visibility-alerts.yaml therefore alerts on the gap, not on the incident:
OAuth2ProxyGateUnobserved compares kube_pod_container_info{container="oauth2-proxy"} against
up{container="oauth2-proxy"} — two already-scraped families — and names every namespace running
a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
one gains a scrape. runbooks/oauth2-proxy-auth-failures.md is the by-hand substitute meanwhile,
and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
To close the gap for an app, three things are needed together, on that app's own manifest:
--metrics-address=0.0.0.0:44180 on the sidecar, a metrics containerPort plus the matching
Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
code="302" as a share of oauth2_proxy_requests_total, held across a window longer than the
session lifetime — become writable rather than decorative.