jakman85andClaude Opus 5
fc7ec01628
Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.
The reason nothing alerted turns out not to be a missing rule. Measured on prod:
count by (__name__) ({__name__=~"oauth2_proxy.*"})
oauth2_proxy_requests_in_flight => 2
Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.
So alert on the condition that made the incident invisible, from families that ARE
scraped:
OAuth2ProxyGateUnobserved
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
unless
count by (namespace) (up{container="oauth2-proxy"})
kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.
Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
check from the app's own login from an XHR that silently got HTML, with the
2026-09-30 jackett trace as the worked example and the Loki queries to use while
there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.
Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
114 lines
7.3 KiB
YAML
114 lines
7.3 KiB
YAML
---
|
|
# oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
|
|
#
|
|
# ============================================================================================
|
|
# WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
|
|
# ============================================================================================
|
|
# Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
|
|
# log in. The sidecar's own access log had the whole story:
|
|
#
|
|
# [oauthproxy.go:1026] No valid authentication in request. Initiating login.
|
|
# GET "/api/v2.0/indexers/yts/config?_=..." HTTP/1.1 ... 302 400
|
|
# GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
|
|
# ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
|
|
# then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
|
|
#
|
|
# NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
|
|
# Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
|
|
# without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
|
|
# JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
|
|
# is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
|
|
# kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
|
|
# gate had rejected every request ever made.
|
|
#
|
|
# ============================================================================================
|
|
# WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
|
|
# ============================================================================================
|
|
# The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
|
|
# user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
|
|
# reason is this file's real subject:
|
|
#
|
|
# THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
|
|
#
|
|
# $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
|
|
# oauth2_proxy_requests_in_flight => 2 <- the SHARED proxy's two replicas. That is ALL.
|
|
#
|
|
# Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) runs with
|
|
# `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
|
|
# is never started; the container declares only port `http:4180`, there is no `metrics` port on
|
|
# any of their Services, and there is no ServiceMonitor or PodMonitor for any of them anywhere
|
|
# in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
|
|
# telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
|
|
# have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
|
|
# This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
|
|
# why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
|
|
# temptation was refused here rather than ratcheted into a register.
|
|
#
|
|
# The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
|
|
# path for any sidecar-gated app — each app gates itself — and its current pods export no
|
|
# `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
|
|
# the shared instance would not have fired on this incident either. Saying so is the point.
|
|
#
|
|
# So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
|
|
# That condition is measurable today, from two already-scraped metric families, and it is the
|
|
# precondition for ever being able to alert on the incident itself. It self-clears the moment a
|
|
# sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
|
|
# writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
|
|
# by-hand equivalent an operator uses in the meantime.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: oauth2-proxy-auth-visibility
|
|
namespace: oauth2-proxy
|
|
labels:
|
|
release: kube-prometheus-stack # kube-prometheus-stack's ruleSelector
|
|
app.kubernetes.io/name: oauth2-proxy
|
|
spec:
|
|
groups:
|
|
- name: oauth2-proxy.auth-visibility
|
|
rules:
|
|
- alert: OAuth2ProxyGateUnobserved
|
|
# A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
|
|
# target for that container in that namespace.
|
|
#
|
|
# BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
|
|
# * kube_pod_container_info — kube-state-metrics, the declared owner in
|
|
# alert-metrics-have-scrape-targets. It sees the container whether or not the
|
|
# container exports anything, which is exactly the property needed here.
|
|
# * up — UNIVERSAL. `up` carries the `container` label for any
|
|
# ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
|
|
# without knowing what job name it will pick.
|
|
# Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
|
|
# inert for the reason it is reporting.
|
|
#
|
|
# MEASURED ON PROD 2026-09-30 — this expression returns six namespaces right now:
|
|
# {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
|
|
# {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
|
|
# and excludes namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
|
|
# the state the cluster is in; it is not a rule waiting for a hypothetical.
|
|
expr: |
|
|
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
|
|
unless
|
|
count by (namespace) (up{container="oauth2-proxy"})
|
|
# 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
|
|
# discovered yet, and this condition is a standing architectural gap, never an
|
|
# emergency. Long enough that no deploy can produce it.
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
|
|
description: >-
|
|
The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
|
|
Prometheus has NO scrape target for it, so every authentication decision that gate
|
|
makes — every redirect to Keycloak, every 403 on a group check, every session that
|
|
silently fails to hold — is invisible to metrics and to alerting. This is not a
|
|
theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
|
|
XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
|
|
seconds later, and the only record anywhere was a line in the sidecar's stdout.
|
|
No alert could have fired, because there is nothing to alert on. Until a scrape
|
|
exists the runbook's LogQL queries are the substitute; the fix is
|
|
`--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
|
|
ServiceMonitor, after which the auth-churn rule becomes writable.
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"
|