Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.
The reason nothing alerted turns out not to be a missing rule. Measured on prod:
count by (__name__) ({__name__=~"oauth2_proxy.*"})
oauth2_proxy_requests_in_flight => 2
Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.
So alert on the condition that made the incident invisible, from families that ARE
scraped:
OAuth2ProxyGateUnobserved
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
unless
count by (namespace) (up{container="oauth2-proxy"})
kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.
Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
check from the app's own login from an XHR that silently got HTML, with the
2026-09-30 jackett trace as the worked example and the Loki queries to use while
there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.
Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Build the deferred forward-auth ENFORCEMENT for non-OIDC UIs (pihole/*arr/
qbittorrent/homepage; Crafty next), gated by Keycloak via oauth2-proxy.
- helmrelease: add `--allowed-groups=platform-admins` (R-IDP-06 floor:
authenticated AND in group) + `--skip-provider-button` (302 straight to
Keycloak; verified the interstitial 403 sign-in page otherwise).
- forward-auth-cec.yaml: interim self-authored CiliumClusterwideEnvoyConfig
ext_authz listener -> oauth2-proxy /oauth2/auth, failure_mode_allow:false,
per-route opt-in via virtualHost. TEMPLATE — not Flux-wired (PM deploys with
an L2 VIP + DNS repoint). The ext_authz stanza is verified in-cluster.
- README: document the mechanism, why per-route ext_authz on the shared Gateway
is NOT clean on Cilium 1.16.5 (machine-owned cilium-gateway-shared CEC; native
ExternalAuth/GEP-1494 in no released Cilium), the three options + recommendation,
the opt-in steps, the in-cluster proof, and the human browser test.
Keycloak `oauth2-proxy` client already carries the callback redirect URI + a
bare `groups` claim (full.path=false) — no realm change needed.
Proof (cilium-envoy v1.30.8): a standalone envoy running this exact ext_authz
stanza against the live oauth2-proxy denied unauthenticated requests (403,
oauth2-proxy sign-in body; upstream never served), while a no-filter control
route on the same backend returned 200. R-IDP-02 property holds.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 5