--- # oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04). # # ============================================================================================ # WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED # ============================================================================================ # Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to # log in. The sidecar's own access log had the whole story: # # [oauthproxy.go:1026] No valid authentication in request. Initiating login. # GET "/api/v2.0/indexers/yts/config?_=..." HTTP/1.1 ... 302 400 # GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409 # ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06, # then "No valid authentication in request" again 26 SECONDS LATER, inside the same session. # # NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to # Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it # without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's # JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service # is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches # kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the # gate had rejected every request ever made. # # ============================================================================================ # WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30 # ============================================================================================ # The alert you would want is "a gated app is serving a sustained rate of auth redirects to a # user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the # reason is this file's real subject: # # THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL. # # $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})' # oauth2_proxy_requests_in_flight => 2 <- the SHARED proxy's two replicas. That is ALL. # # PARTLY SUPERSEDED 2026-10-01, and the correction matters more than the original. PR #28 gave # the five FLEET sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics` port, and the # endpoint genuinely serves (verified by payload: http=200, 10208 bytes, all three # `oauth2_proxy_*` families present — not merely by the flag being in `args`). podmonitor.yaml # in this directory now discovers them. But the scrape is still DROPPED by the app namespaces' # default-deny ingress, so the TSDB remains empty for those namespaces; and crafty still has no # metrics listener at all. The rule below is unchanged in purpose and still has work to do. # # Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) ran with # `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener # was never started; the container declared only port `http:4180`, there was no `metrics` port on # any of their Services, and there was no ServiceMonitor or PodMonitor for any of them anywhere # in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero # telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would # have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health. # This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is # why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the # temptation was refused here rather than ratcheted into a register. # # The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data # path for any sidecar-gated app — each app gates itself — and its current pods export no # `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on # the shared instance would not have fired on this incident either. Saying so is the point. # # So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident. # That condition is measurable today, from two already-scraped metric families, and it is the # precondition for ever being able to alert on the incident itself. It self-clears the moment a # sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes # writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the # by-hand equivalent an operator uses in the meantime. apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: oauth2-proxy-auth-visibility namespace: oauth2-proxy labels: release: kube-prometheus-stack # kube-prometheus-stack's ruleSelector app.kubernetes.io/name: oauth2-proxy spec: groups: - name: oauth2-proxy.auth-visibility rules: - alert: OAuth2ProxyGateUnobserved # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape # target for that container in that namespace. # # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately: # * kube_pod_container_info — kube-state-metrics, the declared owner in # alert-metrics-have-scrape-targets. It sees the container whether or not the # container exports anything, which is exactly the property needed here. # * up — UNIVERSAL. `up` carries the `container` label for any # ServiceMonitor/PodMonitor target, so this works for a future per-app scrape # without knowing what job name it will pick. # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be # inert for the reason it is reporting. # # MEASURED ON PROD 2026-09-30 — this expression returned six namespaces: # {namespace="crafty"} {namespace="jackett"} {namespace="pihole"} # {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"} # and excluded namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on # the state the cluster is in; it is not a rule waiting for a hypothetical. # # `== 1` IS LOAD-BEARING, AND IT IS A BUG FIX (2026-10-01). This clause was originally # a bare `up{container="oauth2-proxy"}`, which was WRONG in the one way that matters # here: a target that is DISCOVERED BUT FAILING still publishes `up`, with value 0. So # the series exists, `count by (namespace)` returns 1, `unless` removes the namespace, # and the alert GOES QUIET while nothing whatsoever is being scraped. That is this # branch's own disease — a green signal standing in for an absent one — reproduced # inside the alert written to cure it. It was not hypothetical: podmonitor.yaml lands # five targets whose scrape is DROPPED by NetworkPolicy (measured; see that file), so # the bare form would have silently cleared on the very next commit. `== 1` means # "observed" requires a scrape that actually succeeds. # # A failing target is therefore reported by BOTH this rule and kube-prometheus-stack's # generic TargetDown. That duplication is deliberate: TargetDown says "a target is # down", this says "an authentication gate is unobserved", and during an incident those # are different sentences. The overlap is the cheap half of the trade. expr: | count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"} == 1) # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been # discovered yet, and this condition is a standing architectural gap, never an # emergency. Long enough that no deploy can produce it. for: 1h labels: severity: warning annotations: summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics" description: >- The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and Prometheus has NO scrape target for it, so every authentication decision that gate makes — every redirect to Keycloak, every 403 on a group check, every session that silently fails to hold — is invisible to metrics and to alerting. This is not a theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own XHR calls with 302s to Keycloak, re-established a session, and lost it again 26 seconds later, and the only record anywhere was a line in the sidecar's stdout. No alert could have fired, because there is nothing to alert on. Until a scrape exists the runbook's LogQL queries are the substitute; the fix is `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a ServiceMonitor, after which the auth-churn rule becomes writable. runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"