home-network-k3s-impl/runbooks/oauth2-proxy-auth-failures.md

# oauth2-proxy auth failures — telling the three layers apart

Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
page is the diagnostic procedure, and it works whether or not the alert ever fired.

## What this means here

Every app behind Keycloak on this platform has **three** places a request can be refused, and
from a browser they look identical — because two of the three answer with **HTTP 200**.

| Layer | What refuses | What the browser sees |
|---|---|---|
| 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
| 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
| 3. The app itself | the app's own login / API key | the app's own 401 or login page |

**Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
failed.

The worked example, jackett on 2026-09-30, straight from the sidecar log:

    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
    GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
    GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
    ... ten of these at 15:30:53
    [AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
                  expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
    GET "/UI/Dashboard"        200 44379
    GET "/api/v2.0/indexers?_=..." 200 1501229
    ... and 26 seconds later, inside that same 5-minute session:
    [oauthproxy.go:1026] No valid authentication in request. Initiating login.

A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
signature of the session being **re-established over and over instead of held** — a cookie that
another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
down and not of the app being broken.

**Read this before you trust any silence (updated 2026-10-01).** The five fleet gate sidecars now
*do* run a metrics listener on `:44180`, and `platform/oauth2-proxy/podmonitor.yaml` discovers
them — but **the scrape is dropped by the app namespaces' `default-deny` NetworkPolicy**, so those
targets read `up == 0` and the TSDB holds nothing for them. Crafty has no listener at all. Only
the shared `oauth2-proxy` namespace instance is genuinely scraped, and it is not in the data path
for any of these apps. So there is still nothing to graph and nothing that could have alerted.
**The logs below remain the only record that this class of failure happened.** Check which state
you are in before you trust a quiet dashboard — "First commands" step 1 tells you in one query.

## First commands

`kubectl` does not exist on the workstation; everything runs in the pinned container, and always
with an explicit context.

**1. Which namespaces have an unobserved gate** (this is what the alert measures):

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"} == 1)"'
```

**1b. If a namespace is listed, is it "no target" or "target failing"?** The two need different
fixes and the alert deliberately does not distinguish them:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
     "up{container=\"oauth2-proxy\"}"'
```

No series for the namespace → nothing discovers it (no PodMonitor match; check the pod carries
`homelab.keithjackson.dev/auth-mode=forward-auth` and a named `metrics` port). A series with
value **0** → discovered but every scrape fails; get the reason from the target's `lastError`:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s localhost:19090/api/v1/targets | python3 -c "
import json,sys
for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
    if t[\"labels\"].get(\"container\")==\"oauth2-proxy\":
        print(t[\"labels\"].get(\"namespace\"), t[\"health\"], t.get(\"lastError\",\"\")[:90])"'
```

`context deadline exceeded` is a **drop**, not a refusal — that is NetworkPolicy, not a dead
endpoint. Confirm the endpoint itself is fine by port-forwarding straight to the pod, which
bypasses policy entirely:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n jackett port-forward \
     $(kubectl --context k3s-prod -n jackett get pod -l app.kubernetes.io/name=jackett -o name | head -1) \
     14418:44180 & sleep 5; curl -s -o /dev/null -w "http=%{http_code} bytes=%{size_download}\n" \
     http://127.0.0.1:14418/metrics'
```

`http=200` with a few KB means the listener is healthy and the path is what is broken. As of
2026-10-01 that is the state of all five fleet apps, and the fix is an ingress allowance from ns
`monitoring` to port 44180 — see `platform/longhorn/networkpolicy-metrics.yaml` for the idiom.

**2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
`kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
```

Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.

**3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
you can ask about the whole fleet and about a window that has already scrolled out of
`kubectl logs`. Port-forward Loki and query it:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
```

and the denominator, which is what makes the number mean something:

```
   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
```

**How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):

| namespace | `No valid authentication` | `AuthSuccess` | ratio |
|---|---|---|---|
| crafty | 81 | 1 | 81:1 |
| radarr | 67 | 1 | 67:1 |
| jackett | 39 | 4 | ~10:1 |
| pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |

All three of those are the failure, not the baseline.

**The cleanest worked case of all — radarr, measured 2026-10-01.** After the cookie fix landed,
jackett dropped to **0** `No valid authentication` lines in 13,612 (from 39), and `removing
session` / `cookie signature not valid` went to **0 on both apps** (radarr had 8 and 8 before).
The eviction fingerprint is gone. But radarr still logged **206** rejections — and sampling 209 of
them, **every single one was the same request**:

    GET "/signalr/messages/negotiate" ... 302    <- browser UA, over and over

That is one left-open browser tab. Its cookie is gone, its SignalR socket keeps trying to
reconnect, and every attempt is silently 302'd to Keycloak. Nobody is at the keyboard, so nobody
is ever re-prompted; the tab just sits there looking like a working page that has stopped
updating. **This is the entire failure class in one line**: a non-navigation request answered with
a redirect, forever, with no error surfaced anywhere.

So when you read a rejection count, **always group by request path before concluding anything**:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n radarr logs deploy/radarr -c oauth2-proxy --since=24h \
     | grep -v "/ping" | grep "No valid authentication" -A 1 \
     | grep -oE "\"[^\"]+\" HTTP" | sort | uniq -c | sort -rn | head'
```

One path dominating means an orphaned client, not a broken gate — annoying but harmless, and it
clears when the tab is closed. Many different paths from one session means the session itself is
not surviving, which is the real defect. A per-app `--api-route` is the structural fix for the
first shape: it makes those paths answer **401** instead of 302, so the client fails loudly
instead of looping. Note that a count of **0 `AuthSuccess`** alongside a high rejection count
means nobody has signed in since the last restart — so `--cookie-refresh` has never run, and no
session-lifecycle conclusion can be drawn from that window at all.

**4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
shape, and `runbooks/crafty-admin-credential.md` covers it.

**5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
**same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
other's cookie and the loser logs exactly the `No valid authentication` line above. List what
each gate is configured to emit (arguments only — never print a Secret's data):

```
infra/devcontainer/dev.sh run bash -c \
  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
     kubectl --context k3s-prod -n $ns get deploy $ns \
       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
```

A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
evicting each other, and the symptom is intermittent and depends on which app you visited last.

## Likely causes, most likely first

1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
   cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
   dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
   Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
   `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
   access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
   `_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
   gate logs the same line. Look for numbered cookie names in the log, and for a session that
   never survives a single navigation.
3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
   differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
   `/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
   jackett trace above.
4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
   session above was a 5-minute session. If `refresh_token:true` is present and sessions still
   die early, the refresh is failing, not missing.
5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
   the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
   for the `groups:[...]` list on the `[AuthSuccess]` line.
6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.

## When to escalate or stop

- **Stop and hand over if the fix is a manifest change.** Everything above is read-only
  diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
  through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
  reverts, and in the meantime you have an unreproducible cluster.
- **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
  in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
  Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
  transcript here exactly this way.
- **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
  `KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
  2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
- **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
  Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
  report is older than three days, the evidence is gone and the only remaining move is to
  reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
  the reason to close it rather than silence it.