jakman85andClaude Opus 5
fc7ec01628
Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.
The reason nothing alerted turns out not to be a missing rule. Measured on prod:
count by (__name__) ({__name__=~"oauth2_proxy.*"})
oauth2_proxy_requests_in_flight => 2
Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.
So alert on the condition that made the incident invisible, from families that ARE
scraped:
OAuth2ProxyGateUnobserved
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
unless
count by (namespace) (up{container="oauth2-proxy"})
kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.
Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
check from the app's own login from an XHR that silently got HTML, with the
2026-09-30 jackett trace as the worked example and the Loki queries to use while
there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.
Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
172 lines
10 KiB
Markdown
172 lines
10 KiB
Markdown
# oauth2-proxy auth failures — telling the three layers apart
|
|
|
|
Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
|
|
does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
|
|
stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
|
|
page is the diagnostic procedure, and it works whether or not the alert ever fired.
|
|
|
|
## What this means here
|
|
|
|
Every app behind Keycloak on this platform has **three** places a request can be refused, and
|
|
from a browser they look identical — because two of the three answer with **HTTP 200**.
|
|
|
|
| Layer | What refuses | What the browser sees |
|
|
|---|---|---|
|
|
| 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
|
|
| 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
|
|
| 3. The app itself | the app's own login / API key | the app's own 401 or login page |
|
|
|
|
**Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
|
|
redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
|
|
JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
|
|
Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
|
|
dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
|
|
nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
|
|
failed.
|
|
|
|
The worked example, jackett on 2026-09-30, straight from the sidecar log:
|
|
|
|
[oauthproxy.go:1026] No valid authentication in request. Initiating login.
|
|
GET "/api/v2.0/indexers/yts/config?_=..." HTTP/1.1 ... 302 400
|
|
GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
|
|
... ten of these at 15:30:53
|
|
[AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
|
|
expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
|
|
GET "/UI/Dashboard" 200 44379
|
|
GET "/api/v2.0/indexers?_=..." 200 1501229
|
|
... and 26 seconds later, inside that same 5-minute session:
|
|
[oauthproxy.go:1026] No valid authentication in request. Initiating login.
|
|
|
|
A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
|
|
signature of the session being **re-established over and over instead of held** — a cookie that
|
|
another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
|
|
down and not of the app being broken.
|
|
|
|
**Read this before you trust any silence:** the per-app gate sidecars export **no metrics**. They
|
|
run without `--metrics-address`, so oauth2-proxy's metrics listener never starts; they declare no
|
|
`metrics` port; and no ServiceMonitor or PodMonitor selects them. Only the shared
|
|
`oauth2-proxy` namespace instance is scraped, and it is not in the data path for any of these
|
|
apps. So there is nothing to graph and nothing that could have alerted. The logs below are not a
|
|
convenience — until that changes they are the **only** record that this class of failure happened.
|
|
|
|
## First commands
|
|
|
|
`kubectl` does not exist on the workstation; everything runs in the pinned container, and always
|
|
with an explicit context.
|
|
|
|
**1. Which namespaces have an unobserved gate** (this is what the alert measures):
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
|
|
-c prometheus -- /bin/promtool query instant http://localhost:9090 \
|
|
"count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"})"'
|
|
```
|
|
|
|
**2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
|
|
and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
|
|
`kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
|
|
| grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
|
|
```
|
|
|
|
Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.
|
|
|
|
**3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
|
|
you can ask about the whole fleet and about a window that has already scrolled out of
|
|
`kubectl logs`. Port-forward Loki and query it:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
|
|
curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
|
|
http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
|
|
```
|
|
|
|
and the denominator, which is what makes the number mean something:
|
|
|
|
```
|
|
... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
|
|
```
|
|
|
|
**How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
|
|
lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
|
|
ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):
|
|
|
|
| namespace | `No valid authentication` | `AuthSuccess` | ratio |
|
|
|---|---|---|---|
|
|
| crafty | 81 | 1 | 81:1 |
|
|
| radarr | 67 | 1 | 67:1 |
|
|
| jackett | 39 | 4 | ~10:1 |
|
|
| pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |
|
|
|
|
All three of those are the failure, not the baseline.
|
|
|
|
**4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
|
|
and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
|
|
page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
|
|
shape, and `runbooks/crafty-admin-credential.md` covers it.
|
|
|
|
**5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
|
|
**same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
|
|
other's cookie and the loser logs exactly the `No valid authentication` line above. List what
|
|
each gate is configured to emit (arguments only — never print a Secret's data):
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
|
|
kubectl --context k3s-prod -n $ns get deploy $ns \
|
|
-o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
|
|
| tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
|
|
```
|
|
|
|
A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
|
|
share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
|
|
evicting each other, and the symptom is intermittent and depends on which app you visited last.
|
|
|
|
## Likely causes, most likely first
|
|
|
|
1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
|
|
cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
|
|
dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
|
|
Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
|
|
`decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
|
|
2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
|
|
access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
|
|
`_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
|
|
gate logs the same line. Look for numbered cookie names in the log, and for a session that
|
|
never survives a single navigation.
|
|
3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
|
|
differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
|
|
`/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
|
|
jackett trace above.
|
|
4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
|
|
session above was a 5-minute session. If `refresh_token:true` is present and sessions still
|
|
die early, the refresh is failing, not missing.
|
|
5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
|
|
the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
|
|
for the `groups:[...]` list on the `[AuthSuccess]` line.
|
|
6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Stop and hand over if the fix is a manifest change.** Everything above is read-only
|
|
diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
|
|
through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
|
|
reverts, and in the meantime you have an unreproducible cluster.
|
|
- **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
|
|
in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
|
|
Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
|
|
transcript here exactly this way.
|
|
- **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
|
|
`KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
|
|
2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
|
|
- **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
|
|
Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
|
|
report is older than three days, the evidence is gone and the only remaining move is to
|
|
reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
|
|
the reason to close it rather than silence it.
|