Merge pull request 'feat(obs): make oauth2-proxy auth failures visible — the gates emit nothing' (#29) from feat/oauth2-proxy-auth-failure-visible into main
Reviewed-on: #29 Reviewed-by: Keith Jackson <[email protected]>
This commit was merged in pull request #29.
This commit is contained in:
jakman85 committed
commit
4f16e83eec
10 files changed
+1095
No files matched your search
@@ -88,3 +88,72 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
|
||||
3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
|
||||
4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
|
||||
protected UIs).
|
||||
|
||||
## Observability: what this component can and cannot see (measured 2026-09-30)
|
||||
|
||||
`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
|
||||
two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
|
||||
in the TSDB is:
|
||||
|
||||
count by (__name__) ({__name__=~"oauth2_proxy.*"})
|
||||
oauth2_proxy_requests_in_flight => 2
|
||||
|
||||
**The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
|
||||
`oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
|
||||
each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
|
||||
listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
|
||||
their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
|
||||
written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
|
||||
|
||||
This matters because the gate's characteristic failure is invisible by construction: an
|
||||
unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
|
||||
against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
|
||||
where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
|
||||
spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
|
||||
into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
|
||||
|
||||
`auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
|
||||
`OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
|
||||
`up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
|
||||
a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
|
||||
one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
|
||||
and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
|
||||
|
||||
**To close the gap for an app**, three things are needed together, on that app's own manifest:
|
||||
`--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
|
||||
Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
|
||||
`code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
|
||||
session lifetime — become writable rather than decorative.
|
||||
|
||||
### Update 2026-10-01: the listener exists; the scrape path does not
|
||||
|
||||
PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` container port to the five
|
||||
fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). `podmonitor.yaml` in this directory
|
||||
discovers them by the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — a **PodMonitor**
|
||||
rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the
|
||||
renderer.
|
||||
|
||||
The endpoint is real: port-forwarded straight to the jackett pod, `/metrics` returns **http=200,
|
||||
10208 bytes**, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (with
|
||||
`code` labels) and `oauth2_proxy_response_duration_seconds`. That is verified by payload, not by the
|
||||
flag appearing in `args` — a distinction worth keeping, because the two claims came apart here.
|
||||
|
||||
**The scrape is nevertheless dropped.** From a pod in ns `monitoring`, both `:44180` and `:4180` on
|
||||
the jackett pod time out; the shared proxy's `:44180` answers. The only difference is that ns
|
||||
`oauth2-proxy` has **no NetworkPolicy at all**, while each app namespace has a `default-deny` whose
|
||||
`policyTypes` include `Ingress` with no `ingress` rules (deny-all) plus a single CNP admitting
|
||||
`fromEntities: [ingress]` to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no
|
||||
clause.
|
||||
|
||||
So the targets read `up == 0` and the TSDB is still empty for those namespaces. This is the same bug
|
||||
and the same fingerprint as `platform/longhorn/networkpolicy-metrics.yaml` documents for Longhorn,
|
||||
and the same narrow fix applies: ns `monitoring` → the gate pod label → TCP 44180 only. It cannot
|
||||
ship from this directory, because `oauth2-proxy` and `apps-wave-0` are **siblings** in the Flux graph
|
||||
(both `dependsOn: [platform-config]`), so a NetworkPolicy naming ns `jackett` would apply on today's
|
||||
cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.
|
||||
|
||||
Consequence for `auth-visibility-alerts.yaml`: its `unless` clause requires `up == 1`, not merely a
|
||||
present `up` series. A discovered-but-failing target publishes `up` with value 0, so the bare form
|
||||
would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
|
||||
as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
|
||||
fails it and nothing else.
|
||||
@@ -0,0 +1,177 @@
|
||||
---
|
||||
# promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
|
||||
#
|
||||
# WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
|
||||
# failure modes of that shape are both silent: a label that does not survive `count by` makes the
|
||||
# right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
|
||||
# and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
|
||||
# visible by reading the expression, and neither is visible on a live cluster where the answer
|
||||
# happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
|
||||
# stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
|
||||
#
|
||||
# HOW TO RUN. The one repo-wide harness, by the companion convention:
|
||||
# ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
|
||||
# promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
|
||||
#
|
||||
# THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
|
||||
# copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
|
||||
# its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
|
||||
# ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
|
||||
# that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
|
||||
# pass while the real expression collapsed differently.
|
||||
#
|
||||
# ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
|
||||
# comparing it verbatim would couple this file to every wording change while proving nothing the
|
||||
# ALERTS series does not already prove.
|
||||
rule_files:
|
||||
- auth-visibility-alerts.rules.yaml
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
|
||||
# the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
|
||||
# jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
|
||||
- interval: 1m
|
||||
name: an unscraped gate namespace fires after 1h; the scraped one never does
|
||||
input_series:
|
||||
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
|
||||
values: '1x130'
|
||||
# The shared instance: kube-state-metrics sees both replicas...
|
||||
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
|
||||
values: '1x130'
|
||||
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
|
||||
values: '1x130'
|
||||
# ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
|
||||
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
|
||||
values: '1x130'
|
||||
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
|
||||
values: '1x130'
|
||||
promql_expr_test:
|
||||
# Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
|
||||
# the hold away and turning every rollout into a page.
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
|
||||
eval_time: 59m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
|
||||
eval_time: 59m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
|
||||
value: 1
|
||||
# Firing for jackett past the hold...
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
|
||||
eval_time: 65m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
|
||||
value: 1
|
||||
# ...and never for the namespace that IS scraped. This is the assertion that catches an
|
||||
# `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
|
||||
# every gate namespace would pass the test above.
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
|
||||
eval_time: 65m
|
||||
exp_samples: []
|
||||
# The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
|
||||
# that lost its `by` would collapse to a single unlabelled series.
|
||||
- expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
|
||||
eval_time: 65m
|
||||
exp_samples:
|
||||
- labels: '{namespace="jackett"}'
|
||||
value: 1
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
|
||||
# `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
|
||||
# name than the shared proxy's. The alert must stop. This is what makes the rule a gap
|
||||
# tracker rather than a permanent nag, and it pins that the match is on `container`, not on
|
||||
# a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
|
||||
- interval: 1m
|
||||
name: the alert clears once the sidecar is actually scraped
|
||||
input_series:
|
||||
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
|
||||
values: '1x130'
|
||||
# Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
|
||||
# exist yet.
|
||||
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
|
||||
values: '_x70 1x60'
|
||||
promql_expr_test:
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
|
||||
eval_time: 65m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
|
||||
value: 1
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
|
||||
eval_time: 80m
|
||||
exp_samples: []
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 3. THE REGRESSION GUARD FOR `== 1` — the scenario that exposed a real bug on 2026-10-01.
|
||||
#
|
||||
# A target is DISCOVERED but every scrape FAILS: `up` exists with value 0. This is not
|
||||
# hypothetical — it is the exact state podmonitor.yaml creates today, because the app
|
||||
# namespaces' default-deny drops the scrape (measured; see podmonitor.yaml).
|
||||
#
|
||||
# With the original `unless count by (namespace) (up{container="oauth2-proxy"})`, the `up`
|
||||
# SERIES EXISTS, so `count by (namespace)` returns 1, `unless` removes the namespace, and the
|
||||
# alert GOES QUIET while nothing is being scraped at all — a green signal standing in for an
|
||||
# absent one, which is the precise disease this whole rule was written to cure. Deleting the
|
||||
# `== 1` makes THIS test fail and every other test in this file still pass, which is the
|
||||
# property that makes it worth having.
|
||||
- interval: 1m
|
||||
name: a discovered-but-FAILING target does not count as observed (up==0 must still fire)
|
||||
input_series:
|
||||
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
|
||||
values: '1x130'
|
||||
# The PodMonitor target exists and never succeeds: NetworkPolicy drops the packets, so the
|
||||
# scrape times out and `up` is published as 0 rather than going absent.
|
||||
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.0.189:44180",job="oauth2-proxy-sidecars",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9"}'
|
||||
values: '0x130'
|
||||
promql_expr_test:
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
|
||||
eval_time: 65m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
|
||||
value: 1
|
||||
# And the inner clause itself yields NOTHING while the scrape is failing — which is the
|
||||
# mechanism the alert depends on, asserted directly so a future reader can see why the
|
||||
# `== 1` matters without re-deriving it. (Scenario 4 below covers the other direction:
|
||||
# a succeeding scrape really does clear the alert, so `== 1` has not made the rule
|
||||
# impossible to satisfy.)
|
||||
- expr: 'count by (namespace) (up{container="oauth2-proxy"} == 1)'
|
||||
eval_time: 65m
|
||||
exp_samples: []
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 4. A FLAPPING target: up goes 1 -> 0 and stays there. The alert must come BACK, not latch
|
||||
# cleared. Covers the ordering bug where a once-observed namespace is permanently excluded.
|
||||
- interval: 1m
|
||||
name: a target that starts working and then fails re-fires
|
||||
input_series:
|
||||
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="radarr",pod="radarr-7d55cbffb9-42hmh",service="kube-prometheus-stack-kube-state-metrics",uid="3333cccc-0000-0000-0000-000000000001"}'
|
||||
values: '1x250'
|
||||
# Healthy for the first 60m, then every scrape fails for the rest.
|
||||
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.77:44180",job="oauth2-proxy-sidecars",namespace="radarr",pod="radarr-7d55cbffb9-42hmh"}'
|
||||
values: '1x60 0x180'
|
||||
promql_expr_test:
|
||||
# Quiet while it was working.
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="radarr"}'
|
||||
eval_time: 55m
|
||||
exp_samples: []
|
||||
# Firing again an hour after it broke.
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr"}'
|
||||
eval_time: 125m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr",severity="warning"}'
|
||||
value: 1
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# 5. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
|
||||
# rule would page for every app on the platform the day someone widened the matcher.
|
||||
- interval: 1m
|
||||
name: namespaces without a gate sidecar are never reported
|
||||
input_series:
|
||||
- series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
|
||||
values: '1x130'
|
||||
promql_expr_test:
|
||||
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
|
||||
eval_time: 65m
|
||||
exp_samples: []
|
||||
@@ -0,0 +1,137 @@
|
||||
---
|
||||
# oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
|
||||
#
|
||||
# ============================================================================================
|
||||
# WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
|
||||
# ============================================================================================
|
||||
# Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
|
||||
# log in. The sidecar's own access log had the whole story:
|
||||
#
|
||||
# [oauthproxy.go:1026] No valid authentication in request. Initiating login.
|
||||
# GET "/api/v2.0/indexers/yts/config?_=..." HTTP/1.1 ... 302 400
|
||||
# GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
|
||||
# ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
|
||||
# then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
|
||||
#
|
||||
# NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
|
||||
# Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
|
||||
# without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
|
||||
# JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
|
||||
# is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
|
||||
# kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
|
||||
# gate had rejected every request ever made.
|
||||
#
|
||||
# ============================================================================================
|
||||
# WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
|
||||
# ============================================================================================
|
||||
# The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
|
||||
# user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
|
||||
# reason is this file's real subject:
|
||||
#
|
||||
# THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
|
||||
#
|
||||
# $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
|
||||
# oauth2_proxy_requests_in_flight => 2 <- the SHARED proxy's two replicas. That is ALL.
|
||||
#
|
||||
# PARTLY SUPERSEDED 2026-10-01, and the correction matters more than the original. PR #28 gave
|
||||
# the five FLEET sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics` port, and the
|
||||
# endpoint genuinely serves (verified by payload: http=200, 10208 bytes, all three
|
||||
# `oauth2_proxy_*` families present — not merely by the flag being in `args`). podmonitor.yaml
|
||||
# in this directory now discovers them. But the scrape is still DROPPED by the app namespaces'
|
||||
# default-deny ingress, so the TSDB remains empty for those namespaces; and crafty still has no
|
||||
# metrics listener at all. The rule below is unchanged in purpose and still has work to do.
|
||||
#
|
||||
# Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) ran with
|
||||
# `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
|
||||
# was never started; the container declared only port `http:4180`, there was no `metrics` port on
|
||||
# any of their Services, and there was no ServiceMonitor or PodMonitor for any of them anywhere
|
||||
# in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
|
||||
# telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
|
||||
# have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
|
||||
# This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
|
||||
# why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
|
||||
# temptation was refused here rather than ratcheted into a register.
|
||||
#
|
||||
# The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
|
||||
# path for any sidecar-gated app — each app gates itself — and its current pods export no
|
||||
# `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
|
||||
# the shared instance would not have fired on this incident either. Saying so is the point.
|
||||
#
|
||||
# So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
|
||||
# That condition is measurable today, from two already-scraped metric families, and it is the
|
||||
# precondition for ever being able to alert on the incident itself. It self-clears the moment a
|
||||
# sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
|
||||
# writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
|
||||
# by-hand equivalent an operator uses in the meantime.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: oauth2-proxy-auth-visibility
|
||||
namespace: oauth2-proxy
|
||||
labels:
|
||||
release: kube-prometheus-stack # kube-prometheus-stack's ruleSelector
|
||||
app.kubernetes.io/name: oauth2-proxy
|
||||
spec:
|
||||
groups:
|
||||
- name: oauth2-proxy.auth-visibility
|
||||
rules:
|
||||
- alert: OAuth2ProxyGateUnobserved
|
||||
# A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
|
||||
# target for that container in that namespace.
|
||||
#
|
||||
# BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
|
||||
# * kube_pod_container_info — kube-state-metrics, the declared owner in
|
||||
# alert-metrics-have-scrape-targets. It sees the container whether or not the
|
||||
# container exports anything, which is exactly the property needed here.
|
||||
# * up — UNIVERSAL. `up` carries the `container` label for any
|
||||
# ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
|
||||
# without knowing what job name it will pick.
|
||||
# Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
|
||||
# inert for the reason it is reporting.
|
||||
#
|
||||
# MEASURED ON PROD 2026-09-30 — this expression returned six namespaces:
|
||||
# {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
|
||||
# {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
|
||||
# and excluded namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
|
||||
# the state the cluster is in; it is not a rule waiting for a hypothetical.
|
||||
#
|
||||
# `== 1` IS LOAD-BEARING, AND IT IS A BUG FIX (2026-10-01). This clause was originally
|
||||
# a bare `up{container="oauth2-proxy"}`, which was WRONG in the one way that matters
|
||||
# here: a target that is DISCOVERED BUT FAILING still publishes `up`, with value 0. So
|
||||
# the series exists, `count by (namespace)` returns 1, `unless` removes the namespace,
|
||||
# and the alert GOES QUIET while nothing whatsoever is being scraped. That is this
|
||||
# branch's own disease — a green signal standing in for an absent one — reproduced
|
||||
# inside the alert written to cure it. It was not hypothetical: podmonitor.yaml lands
|
||||
# five targets whose scrape is DROPPED by NetworkPolicy (measured; see that file), so
|
||||
# the bare form would have silently cleared on the very next commit. `== 1` means
|
||||
# "observed" requires a scrape that actually succeeds.
|
||||
#
|
||||
# A failing target is therefore reported by BOTH this rule and kube-prometheus-stack's
|
||||
# generic TargetDown. That duplication is deliberate: TargetDown says "a target is
|
||||
# down", this says "an authentication gate is unobserved", and during an incident those
|
||||
# are different sentences. The overlap is the cheap half of the trade.
|
||||
expr: |
|
||||
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
|
||||
unless
|
||||
count by (namespace) (up{container="oauth2-proxy"} == 1)
|
||||
# 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
|
||||
# discovered yet, and this condition is a standing architectural gap, never an
|
||||
# emergency. Long enough that no deploy can produce it.
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
|
||||
description: >-
|
||||
The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
|
||||
Prometheus has NO scrape target for it, so every authentication decision that gate
|
||||
makes — every redirect to Keycloak, every 403 on a group check, every session that
|
||||
silently fails to hold — is invisible to metrics and to alerting. This is not a
|
||||
theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
|
||||
XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
|
||||
seconds later, and the only record anywhere was a line in the sidecar's stdout.
|
||||
No alert could have fired, because there is nothing to alert on. Until a scrape
|
||||
exists the runbook's LogQL queries are the substitute; the fix is
|
||||
`--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
|
||||
ServiceMonitor, after which the auth-churn rule becomes writable.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"
|
||||
@@ -14,3 +14,12 @@ resources:
|
||||
- oauth2-proxy-credentials.sops.yaml
|
||||
- httproute.yaml # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
|
||||
- servicemonitor.yaml # Prometheus scrape (was referenced in-place by platform-config)
|
||||
# Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
|
||||
# for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
|
||||
# tier, which is Ready before this tier's platform-config dependency completes.
|
||||
- auth-visibility-alerts.yaml
|
||||
# Discovers the PER-APP gate sidecars' :44180 metrics endpoints by pod label (PR #28 added the
|
||||
# listener; no Service exposes it, so this is a PodMonitor, not a ServiceMonitor). The scrape is
|
||||
# still denied by the app namespaces' default-deny ingress — read podmonitor.yaml's header
|
||||
# before concluding the targets should be up.
|
||||
- podmonitor.yaml
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
# PodMonitor for the PER-APP oauth2-proxy gate sidecars (R-OBS-04).
|
||||
#
|
||||
# ============================================================================================
|
||||
# WHY A PodMonitor AND NOT A ServiceMonitor
|
||||
# ============================================================================================
|
||||
# PR #28 gave all five fleet sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics`
|
||||
# container port. It did NOT add a Service port, and measured on prod 2026-10-01 none of the five
|
||||
# Services exposes 44180:
|
||||
#
|
||||
# jackett jackett: [('port-9117', 9117), ('authproxy', 4180)]
|
||||
# radarr radarr: [('port-7878', 7878), ('authproxy', 4180)]
|
||||
# sonarr sonarr: [('port-8989', 8989), ('authproxy', 4180)]
|
||||
# qbittorrent qbittorrent: [('port-8080', 8080), ('authproxy', 4180)]
|
||||
# pihole pihole: [('port-80', 80), ('authproxy', 4180)]
|
||||
#
|
||||
# A ServiceMonitor selects Service PORTS, so it would have nothing to select. A PodMonitor targets
|
||||
# pod ports directly and bypasses Services entirely — so this needs no change to the Services, and
|
||||
# therefore no change to `catalog/render.py` or the `apps/wave-*` renders and goldens.
|
||||
#
|
||||
# SELECTOR: the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL, not a namespace list.
|
||||
# Measured 2026-10-01: that label is present on exactly the five fleet gate pods (jackett, radarr,
|
||||
# sonarr, qbittorrent, pihole) and absent from crafty's, which is a separately-managed deployment
|
||||
# (platform/crafty) that still has no `--metrics-address` at all. So the selector describes the
|
||||
# PROPERTY being scraped rather than enumerating today's apps, and a sixth forward-auth app is
|
||||
# picked up with no edit here. `namespaceSelector.any: true` because those five namespaces are
|
||||
# created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
|
||||
#
|
||||
# ============================================================================================
|
||||
# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
|
||||
# ============================================================================================
|
||||
# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
|
||||
# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
|
||||
#
|
||||
# THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
|
||||
# GET /metrics -> http=200 bytes=10208
|
||||
# # TYPE oauth2_proxy_requests_in_flight gauge
|
||||
# # TYPE oauth2_proxy_requests_total counter
|
||||
# # TYPE oauth2_proxy_response_duration_seconds histogram
|
||||
# oauth2_proxy_requests_total{code="200"} 3
|
||||
# oauth2_proxy_requests_total{code="302"} 2
|
||||
#
|
||||
# THE SCRAPE PATH IS DROPPED — from a pod in ns monitoring (grafana, which sits beside
|
||||
# Prometheus and so occupies the same policy position) to the jackett pod IP:
|
||||
# http://10.0.0.189:44180/metrics -> wget: download timed out
|
||||
# http://10.0.0.189:4180/ping -> wget: download timed out (even the "allowed" port)
|
||||
# http://<shared-proxy>:44180/metrics -> 200, oauth2_proxy_* present (control: ns oauth2-proxy
|
||||
# has NO NetworkPolicy, which is the only reason the
|
||||
# shared instance has ever been scrapable)
|
||||
#
|
||||
# WHY: ns jackett's `default-deny` NetworkPolicy is `podSelector: {}` with
|
||||
# `policyTypes: [Ingress, Egress]` and an `egress` rule ONLY — no `ingress` key, which in
|
||||
# NetworkPolicy semantics is deny-all-ingress. The only ingress allowance is the CNP
|
||||
# `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
|
||||
# and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
|
||||
#
|
||||
# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
|
||||
# measured, documented state — NOT a regression introduced here, and NOT something to silence.
|
||||
# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
|
||||
# requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
|
||||
# (see that file's header — this exact trap is why the `== 1` is there).
|
||||
#
|
||||
# THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
|
||||
# platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
|
||||
# measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
|
||||
# `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
|
||||
# additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
|
||||
#
|
||||
# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
|
||||
# namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
|
||||
# oauth2-proxy dependsOn=['platform-config'] <- this tier
|
||||
# apps-wave-0 dependsOn=['platform-config'] <- SIBLING, no ordering between them
|
||||
# apps-wave-1 dependsOn=['apps-wave-0']
|
||||
# They are siblings, so on a cold bootstrap there is no guarantee ns jackett exists when this tier
|
||||
# applies, and a NetworkPolicy naming a missing namespace fails to apply. It would work on today's
|
||||
# running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
|
||||
# keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
|
||||
# same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
|
||||
# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
|
||||
# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
name: oauth2-proxy-sidecars
|
||||
namespace: oauth2-proxy
|
||||
labels:
|
||||
release: kube-prometheus-stack # kube-prometheus-stack's podMonitorSelector
|
||||
app.kubernetes.io/name: oauth2-proxy
|
||||
homelab.keithjackson.dev/purpose: metrics-scrape
|
||||
spec:
|
||||
namespaceSelector:
|
||||
# The gate sidecars live in the app namespaces, not this one. `any` rather than a fixed list so
|
||||
# a new forward-auth app needs no edit here — the pod label below is what scopes this.
|
||||
any: true
|
||||
selector:
|
||||
matchLabels:
|
||||
# The PROPERTY, not the app list. Present on exactly the five fleet gate pods (measured
|
||||
# 2026-10-01); absent from crafty, which has no metrics listener yet.
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
podMetricsEndpoints:
|
||||
# By PORT NAME (`metrics`, 44180), never by number: the name is what PR #28 declared on the
|
||||
# container, and a numeric port here would silently stop matching if the port were renumbered.
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
interval: 30s
|
||||
# 10s, matching the drop signature above: a DENIED path does not refuse, it hangs until the
|
||||
# scrape times out. Leaving the default would make each failing target hold a scrape slot for
|
||||
# the full interval.
|
||||
scrapeTimeout: 10s
|
||||
Reference in new issue
Block a user