Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics
gate / gate (pull_request) Successful in 1m42s
Details
gate / conformance (pull_request) Has been skipped
Details

Browse Source 
Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.

The reason nothing alerted turns out not to be a missing rule. Measured on prod:

    count by (__name__) ({__name__=~"oauth2_proxy.*"})
      oauth2_proxy_requests_in_flight => 2

Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.

So alert on the condition that made the incident invisible, from families that ARE
scraped:

  OAuth2ProxyGateUnobserved
    count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
      unless
    count by (namespace) (up{container="oauth2-proxy"})

kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.

Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
  scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
  check from the app's own login from an XHR that silently got HTML, with the
  2026-09-30 jackett trace as the worked example and the Loki queries to use while
  there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.

Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-30 10:55:19 -05:00
1 parent 0fbf544d5e
commit fc7ec01628
8 files changed
+640
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md
+175
View File
@@ -0,0 +1,175 @@
---
date: 2026-09-30
branch: feat/oauth2-proxy-auth-failure-visible
title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
---
## What changed
Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
operator's report named the property that matters — *"reloading the page never fixes the issue and
the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
the alert rules. It is an absence of data.
**The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
count by (__name__) ({__name__=~"oauth2_proxy.*"})
oauth2_proxy_requests_in_flight => 2
Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
`--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
cluster selects them.
So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
permanently silent, and indistinguishable from health. It was not written.
What was written instead alerts on the condition that made the incident invisible, from families
that **are** scraped:
- **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
unless
count by (namespace) (up{container="oauth2-proxy"})
`for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
clears per-namespace as each gate gains a real scrape.
- **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
prod, not invented.
- **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
check refused me" from "the app refused me" from "the XHR silently got HTML", with the
2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
`configMapGenerator` and listed in `runbooks/README.md`.
- **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
per-app sidecars, plus the measured reason.
- **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
three-part change that would close the gap.
**Why the obvious alternative was rejected.** An alert on the *shared* proxy's
`oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
and — decisively — **it would not have fired on this incident**: the shared instance is not in the
data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
at all, having proxied nothing since they started (the counter exists in the TSDB only from an
older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
Shipping it would have looked like coverage of the reported failure and been none.
## What I verified
Everything below was measured against live prod through the pinned container, read-only. No
`apply`, no `edit`, no `patch`, no `flux reconcile`.
- **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
→ `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
`1`, `namespace="oauth2-proxy"`.
- **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
`oauth2-proxy` container, with its declared ports: the two shared replicas have
`[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
`kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
- **No existing alert is inert *for this reason*, because no alert references these metrics at
all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
`platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
(scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
why the runbook README's inert-alerts table gains a row phrased like the etcd one.
- **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
observable from the IdP side.
- **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
`count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
`oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
- **The alert expression fires against live data.** Run verbatim on prod:
`count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
(namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
radarr`, value 1 each; `oauth2-proxy` correctly absent.
- **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
`{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
runbook as the thing to compare against.
- **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
- **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
*Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
`runbook_url` points at exactly that name.
- **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
per-stage outcomes in the next section — read them there, not here.
## What I did NOT verify
- **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
only in git on this branch. What is proven is that its expression returns the expected vector
against live prod data and that promtool agrees about its behaviour over synthetic series.
**Merging is what deploys it.**
- **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
any useful sense** — it describes a standing condition that has been true since the sidecars
were deployed, so it would have been firing before, during and after. That is deliberate and it
is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
instead of discovering it during the investigation. **No alert on this platform could have
detected the incident itself, and none can today.**
- **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
**Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
`apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
`apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
`OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
- **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
directory from the one the sidecar writes), and the rendered config contains **no
`alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
caught this incident, and is the single highest-value follow-up here — **would evaluate and
notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
inert rule ConfigMap to look productive.**
- **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
- **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
snapshot; anything older than three days is already gone. This is stated in the runbook because
it bounds what an operator can still investigate.
- **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
requirement touched by this branch is certified. See the exit code below — **rc 3 is
UNDETERMINED, not green.**
### Follow-ups, in order of value
1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
route to an alert that would have caught the actual 2026-09-30 failure, and the data is
already being collected. The rule is one expression:
`sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
> 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
`OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
and it unlocks the ratio rule in Prometheus.
3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
401 instead of 302. That is the fix that would make this failure *loud* rather than merely
observable, and it belongs with whoever owns the sidecar args.
platform/oauth2-proxy/README.md
+36
View File
@@ -88,3 +88,39 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
protected UIs).
## Observability: what this component can and cannot see (measured 2026-09-30)
`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
in the TSDB is:
count by (__name__) ({__name__=~"oauth2_proxy.*"})
oauth2_proxy_requests_in_flight => 2
**The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
`oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
This matters because the gate's characteristic failure is invisible by construction: an
unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
`auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
`OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
`up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
**To close the gap for an app**, three things are needed together, on that app's own manifest:
`--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
`code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
session lifetime — become writable rather than decorative.
platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml
+117
View File
@@ -0,0 +1,117 @@
---
# promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
#
# WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
# failure modes of that shape are both silent: a label that does not survive `count by` makes the
# right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
# and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
# visible by reading the expression, and neither is visible on a live cluster where the answer
# happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
# stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
#
# HOW TO RUN. The one repo-wide harness, by the companion convention:
# ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
# promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
#
# THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
# copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
# its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
# ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
# that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
# pass while the real expression collapsed differently.
#
# ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
# comparing it verbatim would couple this file to every wording change while proving nothing the
# ALERTS series does not already prove.
rule_files:
- auth-visibility-alerts.rules.yaml
evaluation_interval: 1m
tests:
# ---------------------------------------------------------------------------------------------
# 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
# the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
# jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
- interval: 1m
name: an unscraped gate namespace fires after 1h; the scraped one never does
input_series:
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
values: '1x130'
# The shared instance: kube-state-metrics sees both replicas...
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
values: '1x130'
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
values: '1x130'
# ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
values: '1x130'
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
values: '1x130'
promql_expr_test:
# Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
# the hold away and turning every rollout into a page.
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
eval_time: 59m
exp_samples: []
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
eval_time: 59m
exp_samples:
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
value: 1
# Firing for jackett past the hold...
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
eval_time: 65m
exp_samples:
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
value: 1
# ...and never for the namespace that IS scraped. This is the assertion that catches an
# `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
# every gate namespace would pass the test above.
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
eval_time: 65m
exp_samples: []
# The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
# that lost its `by` would collapse to a single unlabelled series.
- expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
eval_time: 65m
exp_samples:
- labels: '{namespace="jackett"}'
value: 1
# ---------------------------------------------------------------------------------------------
# 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
# `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
# name than the shared proxy's. The alert must stop. This is what makes the rule a gap
# tracker rather than a permanent nag, and it pins that the match is on `container`, not on
# a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
- interval: 1m
name: the alert clears once the sidecar is actually scraped
input_series:
- series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
values: '1x130'
# Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
# exist yet.
- series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
values: '_x70 1x60'
promql_expr_test:
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
eval_time: 65m
exp_samples:
- labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
value: 1
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
eval_time: 80m
exp_samples: []
# ---------------------------------------------------------------------------------------------
# 3. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
# rule would page for every app on the platform the day someone widened the matcher.
- interval: 1m
name: namespaces without a gate sidecar are never reported
input_series:
- series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
values: '1x130'
promql_expr_test:
- expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
eval_time: 65m
exp_samples: []
platform/oauth2-proxy/auth-visibility-alerts.yaml
+113
View File
@@ -0,0 +1,113 @@
---
# oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
#
# ============================================================================================
# WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
# ============================================================================================
# Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
# log in. The sidecar's own access log had the whole story:
#
# [oauthproxy.go:1026] No valid authentication in request. Initiating login.
# GET "/api/v2.0/indexers/yts/config?_=..." HTTP/1.1 ... 302 400
# GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
# ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
# then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
#
# NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
# Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
# without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
# JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
# is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
# kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
# gate had rejected every request ever made.
#
# ============================================================================================
# WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
# ============================================================================================
# The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
# user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
# reason is this file's real subject:
#
# THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
#
# $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
# oauth2_proxy_requests_in_flight => 2 <- the SHARED proxy's two replicas. That is ALL.
#
# Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) runs with
# `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
# is never started; the container declares only port `http:4180`, there is no `metrics` port on
# any of their Services, and there is no ServiceMonitor or PodMonitor for any of them anywhere
# in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
# telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
# have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
# This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
# why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
# temptation was refused here rather than ratcheted into a register.
#
# The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
# path for any sidecar-gated app — each app gates itself — and its current pods export no
# `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
# the shared instance would not have fired on this incident either. Saying so is the point.
#
# So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
# That condition is measurable today, from two already-scraped metric families, and it is the
# precondition for ever being able to alert on the incident itself. It self-clears the moment a
# sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
# writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
# by-hand equivalent an operator uses in the meantime.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: oauth2-proxy-auth-visibility
namespace: oauth2-proxy
labels:
release: kube-prometheus-stack # kube-prometheus-stack's ruleSelector
app.kubernetes.io/name: oauth2-proxy
spec:
groups:
- name: oauth2-proxy.auth-visibility
rules:
- alert: OAuth2ProxyGateUnobserved
# A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
# target for that container in that namespace.
#
# BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
# * kube_pod_container_info — kube-state-metrics, the declared owner in
# alert-metrics-have-scrape-targets. It sees the container whether or not the
# container exports anything, which is exactly the property needed here.
# * up — UNIVERSAL. `up` carries the `container` label for any
# ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
# without knowing what job name it will pick.
# Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
# inert for the reason it is reporting.
#
# MEASURED ON PROD 2026-09-30 — this expression returns six namespaces right now:
# {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
# {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
# and excludes namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
# the state the cluster is in; it is not a rule waiting for a hypothetical.
expr: |
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
unless
count by (namespace) (up{container="oauth2-proxy"})
# 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
# discovered yet, and this condition is a standing architectural gap, never an
# emergency. Long enough that no deploy can produce it.
for: 1h
labels:
severity: warning
annotations:
summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
description: >-
The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
Prometheus has NO scrape target for it, so every authentication decision that gate
makes — every redirect to Keycloak, every 403 on a group check, every session that
silently fails to hold — is invisible to metrics and to alerting. This is not a
theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
seconds later, and the only record anywhere was a line in the sidecar's stdout.
No alert could have fired, because there is nothing to alert on. Until a scrape
exists the runbook's LogQL queries are the substitute; the fix is
`--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
ServiceMonitor, after which the auth-churn rule becomes writable.
runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"
platform/oauth2-proxy/kustomization.yaml
+4
View File
@@ -14,3 +14,7 @@ resources:
- oauth2-proxy-credentials.sops.yaml
- httproute.yaml # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
- servicemonitor.yaml # Prometheus scrape (was referenced in-place by platform-config)
# Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
# for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
# tier, which is Ready before this tier's platform-config dependency completes.
- auth-visibility-alerts.yaml
runbooks/README.md
+20
View File
@@ -51,6 +51,18 @@ rule groups still evaluate against nothing and **cannot fire**:
| Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
| Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — measured 2026-09-30 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
The oauth2-proxy row is the newest and the least obvious, because the scrape is not merely missing —
there is **nothing to scrape**. `count by (__name__) ({__name__=~"oauth2_proxy.*"})` on prod returns
exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
forward-auth instance in the `oauth2-proxy` namespace. The six per-app gate sidecars run without
`--metrics-address`, so their metrics listener never starts; they declare no `metrics` container port;
and no ServiceMonitor or PodMonitor selects them. Any rule written against `oauth2_proxy_*` for those
namespaces would be inert on arrival. `OAuth2ProxyGateUnobserved`
(`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on that gap itself, using
`kube_pod_container_info` and `up` — two families that *are* scraped — and clears when a gate gains a
real scrape. See `oauth2-proxy-auth-failures`.
Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
@@ -108,6 +120,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
- **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
- **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
every green signal missed.
- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
`OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
rather than a 401, the browser follows it silently against a live SSO session, and the call
resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
so these logs (72h of them) are the only record this class of failure leaves.
### Operator procedures (no alert links here)
- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
runbooks/kustomization.yaml
+4
View File
@@ -85,6 +85,10 @@ configMapGenerator:
- node-io=node-io.md
- node-memory=node-memory.md
- node-memory-swap=node-memory-swap.md
# Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
# that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
# not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
# Operator procedure, not an alert page (no alert links to it — the
# alert-runbooks-resolvable check treats that as advisory). It is served
# because the plex.tv claim window is short and is not a thing to go looking
runbooks/oauth2-proxy-auth-failures.md
+171
View File
@@ -0,0 +1,171 @@
# oauth2-proxy auth failures — telling the three layers apart
Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
page is the diagnostic procedure, and it works whether or not the alert ever fired.
## What this means here
Every app behind Keycloak on this platform has **three** places a request can be refused, and
from a browser they look identical — because two of the three answer with **HTTP 200**.
| Layer | What refuses | What the browser sees |
|---|---|---|
| 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
| 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
| 3. The app itself | the app's own login / API key | the app's own 401 or login page |
**Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
failed.
The worked example, jackett on 2026-09-30, straight from the sidecar log:
[oauthproxy.go:1026] No valid authentication in request. Initiating login.
GET "/api/v2.0/indexers/yts/config?_=..." HTTP/1.1 ... 302 400
GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
... ten of these at 15:30:53
[AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
GET "/UI/Dashboard" 200 44379
GET "/api/v2.0/indexers?_=..." 200 1501229
... and 26 seconds later, inside that same 5-minute session:
[oauthproxy.go:1026] No valid authentication in request. Initiating login.
A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
signature of the session being **re-established over and over instead of held** — a cookie that
another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
down and not of the app being broken.
**Read this before you trust any silence:** the per-app gate sidecars export **no metrics**. They
run without `--metrics-address`, so oauth2-proxy's metrics listener never starts; they declare no
`metrics` port; and no ServiceMonitor or PodMonitor selects them. Only the shared
`oauth2-proxy` namespace instance is scraped, and it is not in the data path for any of these
apps. So there is nothing to graph and nothing that could have alerted. The logs below are not a
convenience — until that changes they are the **only** record that this class of failure happened.
## First commands
`kubectl` does not exist on the workstation; everything runs in the pinned container, and always
with an explicit context.
**1. Which namespaces have an unobserved gate** (this is what the alert measures):
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
-c prometheus -- /bin/promtool query instant http://localhost:9090 \
"count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"})"'
```
**2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
`kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
| grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
```
Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.
**3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
you can ask about the whole fleet and about a window that has already scrolled out of
`kubectl logs`. Port-forward Loki and query it:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
```
and the denominator, which is what makes the number mean something:
```
... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
```
**How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):
| namespace | `No valid authentication` | `AuthSuccess` | ratio |
|---|---|---|---|
| crafty | 81 | 1 | 81:1 |
| radarr | 67 | 1 | 67:1 |
| jackett | 39 | 4 | ~10:1 |
| pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |
All three of those are the failure, not the baseline.
**4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
shape, and `runbooks/crafty-admin-credential.md` covers it.
**5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
**same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
other's cookie and the loser logs exactly the `No valid authentication` line above. List what
each gate is configured to emit (arguments only — never print a Secret's data):
```
infra/devcontainer/dev.sh run bash -c \
'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
kubectl --context k3s-prod -n $ns get deploy $ns \
-o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
| tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
```
A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
evicting each other, and the symptom is intermittent and depends on which app you visited last.
## Likely causes, most likely first
1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
`decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
`_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
gate logs the same line. Look for numbered cookie names in the log, and for a session that
never survives a single navigation.
3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
`/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
jackett trace above.
4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
session above was a 5-minute session. If `refresh_token:true` is present and sessions still
die early, the refresh is failing, not missing.
5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
for the `groups:[...]` list on the `[AuthSuccess]` line.
6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.
## When to escalate or stop
- **Stop and hand over if the fix is a manifest change.** Everything above is read-only
diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
reverts, and in the meantime you have an unreproducible cluster.
- **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
transcript here exactly this way.
- **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
`KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
- **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
report is older than three days, the evidence is gone and the only remaining move is to
reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
the reason to close it rather than silence it.