home-network-k3s-impl/platform/oauth2-proxy/auth-visibility-alerts.yaml

---
# oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
#
# ============================================================================================
# WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
# ============================================================================================
# Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
# log in. The sidecar's own access log had the whole story:
#
#   [oauthproxy.go:1026] No valid authentication in request. Initiating login.
#   GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
#   GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
#   ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
#   then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
#
# NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
# Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
# without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
# JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
# is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
# kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
# gate had rejected every request ever made.
#
# ============================================================================================
# WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
# ============================================================================================
# The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
# user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
# reason is this file's real subject:
#
#   THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
#
#   $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
#     oauth2_proxy_requests_in_flight => 2          <- the SHARED proxy's two replicas. That is ALL.
#
#   PARTLY SUPERSEDED 2026-10-01, and the correction matters more than the original. PR #28 gave
#   the five FLEET sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics` port, and the
#   endpoint genuinely serves (verified by payload: http=200, 10208 bytes, all three
#   `oauth2_proxy_*` families present — not merely by the flag being in `args`). podmonitor.yaml
#   in this directory now discovers them. But the scrape is still DROPPED by the app namespaces'
#   default-deny ingress, so the TSDB remains empty for those namespaces; and crafty still has no
#   metrics listener at all. The rule below is unchanged in purpose and still has work to do.
#
#   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) ran with
#   `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
#   was never started; the container declared only port `http:4180`, there was no `metrics` port on
#   any of their Services, and there was no ServiceMonitor or PodMonitor for any of them anywhere
#   in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
#   telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
#   have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
#   This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
#   why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
#   temptation was refused here rather than ratcheted into a register.
#
#   The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
#   path for any sidecar-gated app — each app gates itself — and its current pods export no
#   `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
#   the shared instance would not have fired on this incident either. Saying so is the point.
#
# So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
# That condition is measurable today, from two already-scraped metric families, and it is the
# precondition for ever being able to alert on the incident itself. It self-clears the moment a
# sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
# writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
# by-hand equivalent an operator uses in the meantime.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: oauth2-proxy-auth-visibility
  namespace: oauth2-proxy
  labels:
    release: kube-prometheus-stack     # kube-prometheus-stack's ruleSelector
    app.kubernetes.io/name: oauth2-proxy
spec:
  groups:
    - name: oauth2-proxy.auth-visibility
      rules:
        - alert: OAuth2ProxyGateUnobserved
          # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
          # target for that container in that namespace.
          #
          # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
          #   * kube_pod_container_info  — kube-state-metrics, the declared owner in
          #     alert-metrics-have-scrape-targets. It sees the container whether or not the
          #     container exports anything, which is exactly the property needed here.
          #   * up                       — UNIVERSAL. `up` carries the `container` label for any
          #     ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
          #     without knowing what job name it will pick.
          # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
          # inert for the reason it is reporting.
          #
          # MEASURED ON PROD 2026-09-30 — this expression returned six namespaces:
          #   {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
          #   {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
          # and excluded namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
          # the state the cluster is in; it is not a rule waiting for a hypothetical.
          #
          # `== 1` IS LOAD-BEARING, AND IT IS A BUG FIX (2026-10-01). This clause was originally
          # a bare `up{container="oauth2-proxy"}`, which was WRONG in the one way that matters
          # here: a target that is DISCOVERED BUT FAILING still publishes `up`, with value 0. So
          # the series exists, `count by (namespace)` returns 1, `unless` removes the namespace,
          # and the alert GOES QUIET while nothing whatsoever is being scraped. That is this
          # branch's own disease — a green signal standing in for an absent one — reproduced
          # inside the alert written to cure it. It was not hypothetical: podmonitor.yaml lands
          # five targets whose scrape is DROPPED by NetworkPolicy (measured; see that file), so
          # the bare form would have silently cleared on the very next commit. `== 1` means
          # "observed" requires a scrape that actually succeeds.
          #
          # A failing target is therefore reported by BOTH this rule and kube-prometheus-stack's
          # generic TargetDown. That duplication is deliberate: TargetDown says "a target is
          # down", this says "an authentication gate is unobserved", and during an incident those
          # are different sentences. The overlap is the cheap half of the trade.
          expr: |
            count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
              unless
            count by (namespace) (up{container="oauth2-proxy"} == 1)
          # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
          # discovered yet, and this condition is a standing architectural gap, never an
          # emergency. Long enough that no deploy can produce it.
          for: 1h
          labels:
            severity: warning
          annotations:
            summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
            description: >-
              The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
              Prometheus has NO scrape target for it, so every authentication decision that gate
              makes — every redirect to Keycloak, every 403 on a group check, every session that
              silently fails to hold — is invisible to metrics and to alerting. This is not a
              theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
              XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
              seconds later, and the only record anywhere was a line in the sidecar's stdout.
              No alert could have fired, because there is nothing to alert on. Until a scrape
              exists the runbook's LogQL queries are the substitute; the fix is
              `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
              ServiceMonitor, after which the auth-churn rule becomes writable.
            runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"