Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
fc7ec01628e016c2fc8540d7cbbd178cafdc5067
home-network-k3s-impl/platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml
T
jakman85andClaude Opus 5 fc7ec01628
gate / gate (pull_request) Successful in 1m42s
Details
gate / conformance (pull_request) Has been skipped
Details
feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics 
Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.

The reason nothing alerted turns out not to be a missing rule. Measured on prod:

    count by (__name__) ({__name__=~"oauth2_proxy.*"})
      oauth2_proxy_requests_in_flight => 2

Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.

So alert on the condition that made the incident invisible, from families that ARE
scraped:

  OAuth2ProxyGateUnobserved
    count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
      unless
    count by (namespace) (up{container="oauth2-proxy"})

kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.

Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
  scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
  check from the app's own login from an XHR that silently got HTML, with the
  2026-09-30 jackett trace as the worked example and the Loki queries to use while
  there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.

Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-30 10:55:19 -05:00

118 lines
9.0 KiB
YAML
Raw Blame History

---
# promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
#
# WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
# failure modes of that shape are both silent: a label that does not survive `count by` makes the
# right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
# and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
# visible by reading the expression, and neither is visible on a live cluster where the answer
# happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
# stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
#
# HOW TO RUN. The one repo-wide harness, by the companion convention:
# ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
# promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
#
# THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
# copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
# its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
# ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
# that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
# pass while the real expression collapsed differently.
#
# ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
# comparing it verbatim would couple this file to every wording change while proving nothing the
# ALERTS series does not already prove.
rule_files:
- auth-visibility-alerts.rules.yaml
evaluation_interval: 1m
tests:
# ---------------------------------------------------------------------------------------------
# 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
# the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
# jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
- interval: 1m
name: an unscraped gate namespace fires after 1h; the scraped one never does
input_series:
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
values: '1x130'
# The shared instance: kube-state-metrics sees both replicas...
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
values: '1x130'
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
values: '1x130'
# ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
values: '1x130'
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
values: '1x130'
promql_expr_test:
# Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
# the hold away and turning every rollout into a page.
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
eval_time: 59m
exp_samples: []
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
eval_time: 59m
exp_samples:
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
value: 1
# Firing for jackett past the hold...
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
eval_time: 65m
exp_samples:
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
value: 1
# ...and never for the namespace that IS scraped. This is the assertion that catches an
# `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
# every gate namespace would pass the test above.
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
eval_time: 65m
exp_samples: []
# The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
# that lost its `by` would collapse to a single unlabelled series.
- expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
eval_time: 65m
exp_samples:
- labels: '{namespace="jackett"}'
value: 1
# ---------------------------------------------------------------------------------------------
# 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
# `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
# name than the shared proxy's. The alert must stop. This is what makes the rule a gap
# tracker rather than a permanent nag, and it pins that the match is on `container`, not on
# a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
- interval: 1m
name: the alert clears once the sidecar is actually scraped
input_series:
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
values: '1x130'
# Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
# exist yet.
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
values: '_x70 1x60'
promql_expr_test:
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
eval_time: 65m
exp_samples:
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
value: 1
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
eval_time: 80m
exp_samples: []
# ---------------------------------------------------------------------------------------------
# 3. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
# rule would page for every app on the platform the day someone widened the matcher.
- interval: 1m
name: namespaces without a gate sidecar are never reported
input_series:
- series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
values: '1x130'
promql_expr_test:
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
eval_time: 65m
exp_samples: []
Reference in new issue View Git Blame Copy Permalink