feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics

Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak rather than a 401; the browser follows it silently against a live SSO session and the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript wanted JSON. The user is never re-prompted, reloading never helps, and every uptime check stays green. `jackett-down` watches replicas_available, which was 1 throughout. The reason nothing alerted turns out not to be a missing rule. Measured on prod: count by (__name__) ({__name__=~"oauth2_proxy.*"}) oauth2_proxy_requests_in_flight => 2 Two series, both from the SHARED forward-auth Deployment. The six per-app gate sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at all: no --metrics-address, so the metrics listener never starts; no `metrics` container port; no ServiceMonitor or PodMonitor selects them. The alert the incident calls for — sustained auth redirects to an authenticated user — is therefore NOT WRITABLE, and writing it anyway would have shipped a rule that is present, evaluating, permanently silent and indistinguishable from health. So alert on the condition that made the incident invisible, from families that ARE scraped: OAuth2ProxyGateUnobserved count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"}) kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the container whether or not it exports anything; `up` is UNIVERSAL and carries a `container` label for any target, so the rule needs no advance knowledge of what job name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it cannot be inert for the reason it is reporting. Run verbatim against live prod it returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace as each gate gains a real scrape. Also: - promtool companion test pinning all four states: fires unscraped, quiet when scraped, honours for:1h, and STOPS once a scrape appears under a different job name. - runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group check from the app's own login from an XHR that silently got HTML, with the 2026-09-30 jackett trace as the worked example and the Loki queries to use while there are no metrics. Registered in the configMapGenerator and README. - runbooks/README.md gains an inert-alerts row for the unscraped sidecars. Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have fired on this incident — the shared instance is not in any sidecar-gated app's data path and its current pods have proxied nothing. It would have looked like coverage and been none. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-30 10:55:19 -05:00
1 parent 0fbf544d5e
commit fc7ec01628
@@ -88,3 +88,39 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
 3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
 4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
   protected UIs).
+
+## Observability: what this component can and cannot see (measured 2026-09-30)
+
+`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
+two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
+in the TSDB is:
+
+    count by (__name__) ({__name__=~"oauth2_proxy.*"})
+      oauth2_proxy_requests_in_flight => 2
+
+**The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
+`oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
+each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
+listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
+their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
+written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
+
+This matters because the gate's characteristic failure is invisible by construction: an
+unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
+against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
+where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
+spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
+into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
+
+`auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
+`OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
+`up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
+a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
+one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
+and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
+
+**To close the gap for an app**, three things are needed together, on that app's own manifest:
+`--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
+Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
+`code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
+session lifetime — become writable rather than decorative.
@@ -0,0 +1,117 @@
+---
+# promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
+#
+# WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
+# failure modes of that shape are both silent: a label that does not survive `count by` makes the
+# right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
+# and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
+# visible by reading the expression, and neither is visible on a live cluster where the answer
+# happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
+# stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
+#
+# HOW TO RUN. The one repo-wide harness, by the companion convention:
+#   ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
+# promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
+#
+# THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
+# copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
+# its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
+# ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
+# that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
+# pass while the real expression collapsed differently.
+#
+# ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
+# comparing it verbatim would couple this file to every wording change while proving nothing the
+# ALERTS series does not already prove.
+rule_files:
+  - auth-visibility-alerts.rules.yaml
+evaluation_interval: 1m
+
+tests:
+  # ---------------------------------------------------------------------------------------------
+  # 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
+  #    the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
+  #    jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
+  - interval: 1m
+    name: an unscraped gate namespace fires after 1h; the scraped one never does
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
+        values: '1x130'
+      # The shared instance: kube-state-metrics sees both replicas...
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
+        values: '1x130'
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
+        values: '1x130'
+      # ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
+        values: '1x130'
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
+        values: '1x130'
+    promql_expr_test:
+      # Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
+      # the hold away and turning every rollout into a page.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
+        eval_time: 59m
+        exp_samples: []
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
+        eval_time: 59m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
+            value: 1
+      # Firing for jackett past the hold...
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
+        eval_time: 65m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
+            value: 1
+      # ...and never for the namespace that IS scraped. This is the assertion that catches an
+      # `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
+      # every gate namespace would pass the test above.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
+        eval_time: 65m
+        exp_samples: []
+      # The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
+      # that lost its `by` would collapse to a single unlabelled series.
+      - expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
+        eval_time: 65m
+        exp_samples:
+          - labels: '{namespace="jackett"}'
+            value: 1
+
+  # ---------------------------------------------------------------------------------------------
+  # 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
+  #    `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
+  #    name than the shared proxy's. The alert must stop. This is what makes the rule a gap
+  #    tracker rather than a permanent nag, and it pins that the match is on `container`, not on
+  #    a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
+  - interval: 1m
+    name: the alert clears once the sidecar is actually scraped
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
+        values: '1x130'
+      # Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
+      # exist yet.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
+        values: '_x70 1x60'
+    promql_expr_test:
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
+        eval_time: 65m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
+            value: 1
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
+        eval_time: 80m
+        exp_samples: []
+
+  # ---------------------------------------------------------------------------------------------
+  # 3. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
+  #    rule would page for every app on the platform the day someone widened the matcher.
+  - interval: 1m
+    name: namespaces without a gate sidecar are never reported
+    input_series:
+      - series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
+        values: '1x130'
+    promql_expr_test:
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
+        eval_time: 65m
+        exp_samples: []
@@ -0,0 +1,113 @@
+---
+# oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
+#
+# ============================================================================================
+# WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
+# ============================================================================================
+# Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
+# log in. The sidecar's own access log had the whole story:
+#
+#   [oauthproxy.go:1026] No valid authentication in request. Initiating login.
+#   GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
+#   GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
+#   ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
+#   then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
+#
+# NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
+# Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
+# without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
+# JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
+# is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
+# kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
+# gate had rejected every request ever made.
+#
+# ============================================================================================
+# WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
+# ============================================================================================
+# The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
+# user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
+# reason is this file's real subject:
+#
+#   THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
+#
+#   $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
+#     oauth2_proxy_requests_in_flight => 2          <- the SHARED proxy's two replicas. That is ALL.
+#
+#   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) runs with
+#   `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
+#   is never started; the container declares only port `http:4180`, there is no `metrics` port on
+#   any of their Services, and there is no ServiceMonitor or PodMonitor for any of them anywhere
+#   in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
+#   telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
+#   have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
+#   This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
+#   why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
+#   temptation was refused here rather than ratcheted into a register.
+#
+#   The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
+#   path for any sidecar-gated app — each app gates itself — and its current pods export no
+#   `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
+#   the shared instance would not have fired on this incident either. Saying so is the point.
+#
+# So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
+# That condition is measurable today, from two already-scraped metric families, and it is the
+# precondition for ever being able to alert on the incident itself. It self-clears the moment a
+# sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
+# writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
+# by-hand equivalent an operator uses in the meantime.
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: oauth2-proxy-auth-visibility
+  namespace: oauth2-proxy
+  labels:
+    release: kube-prometheus-stack     # kube-prometheus-stack's ruleSelector
+    app.kubernetes.io/name: oauth2-proxy
+spec:
+  groups:
+    - name: oauth2-proxy.auth-visibility
+      rules:
+        - alert: OAuth2ProxyGateUnobserved
+          # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
+          # target for that container in that namespace.
+          #
+          # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
+          #   * kube_pod_container_info  — kube-state-metrics, the declared owner in
+          #     alert-metrics-have-scrape-targets. It sees the container whether or not the
+          #     container exports anything, which is exactly the property needed here.
+          #   * up                       — UNIVERSAL. `up` carries the `container` label for any
+          #     ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
+          #     without knowing what job name it will pick.
+          # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
+          # inert for the reason it is reporting.
+          #
+          # MEASURED ON PROD 2026-09-30 — this expression returns six namespaces right now:
+          #   {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
+          #   {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
+          # and excludes namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
+          # the state the cluster is in; it is not a rule waiting for a hypothetical.
+          expr: |
+            count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
+              unless
+            count by (namespace) (up{container="oauth2-proxy"})
+          # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
+          # discovered yet, and this condition is a standing architectural gap, never an
+          # emergency. Long enough that no deploy can produce it.
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
+            description: >-
+              The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
+              Prometheus has NO scrape target for it, so every authentication decision that gate
+              makes — every redirect to Keycloak, every 403 on a group check, every session that
+              silently fails to hold — is invisible to metrics and to alerting. This is not a
+              theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
+              XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
+              seconds later, and the only record anywhere was a line in the sidecar's stdout.
+              No alert could have fired, because there is nothing to alert on. Until a scrape
+              exists the runbook's LogQL queries are the substitute; the fix is
+              `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
+              ServiceMonitor, after which the auth-churn rule becomes writable.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"
@@ -14,3 +14,7 @@ resources:
  - oauth2-proxy-credentials.sops.yaml
  - httproute.yaml       # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
  - servicemonitor.yaml  # Prometheus scrape (was referenced in-place by platform-config)
+  # Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
+  # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
+  # tier, which is Ready before this tier's platform-config dependency completes.
+  - auth-visibility-alerts.yaml