# oauth2-proxy auth failures — telling the three layers apart

Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
page is the diagnostic procedure, and it works whether or not the alert ever fired.

## What this means here

Every app behind Keycloak on this platform has **three** places a request can be refused, and
from a browser they look identical — because two of the three answer with **HTTP 200**.

| Layer | What refuses | What the browser sees |
|---|---|---|
| 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
| 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
| 3. The app itself | the app's own login / API key | the app's own 401 or login page |

**Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
failed.

The worked example, jackett on 2026-09-30, straight from the sidecar log:

    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
    GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
    GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
    ... ten of these at 15:30:53
    [AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
                  expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
    GET "/UI/Dashboard"        200 44379
    GET "/api/v2.0/indexers?_=..." 200 1501229
    ... and 26 seconds later, inside that same 5-minute session:
    [oauthproxy.go:1026] No valid authentication in request. Initiating login.

A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
signature of the session being **re-established over and over instead of held** — a cookie that
another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
down and not of the app being broken.

**Read this before you trust any silence:** the per-app gate sidecars export **no metrics**. They
run without `--metrics-address`, so oauth2-proxy's metrics listener never starts; they declare no
`metrics` port; and no ServiceMonitor or PodMonitor selects them. Only the shared
`oauth2-proxy` namespace instance is scraped, and it is not in the data path for any of these
apps. So there is nothing to graph and nothing that could have alerted. The logs below are not a
convenience — until that changes they are the **only** record that this class of failure happened.

## First commands

`kubectl` does not exist on the workstation; everything runs in the pinned container, and always
with an explicit context.

**1. Which namespaces have an unobserved gate** (this is what the alert measures):

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"})"'
```

**2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
`kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
```

Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.

**3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
you can ask about the whole fleet and about a window that has already scrolled out of
`kubectl logs`. Port-forward Loki and query it:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
```

and the denominator, which is what makes the number mean something:

```
   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
```

**How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):

| namespace | `No valid authentication` | `AuthSuccess` | ratio |
|---|---|---|---|
| crafty | 81 | 1 | 81:1 |
| radarr | 67 | 1 | 67:1 |
| jackett | 39 | 4 | ~10:1 |
| pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |

All three of those are the failure, not the baseline.

**4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
shape, and `runbooks/crafty-admin-credential.md` covers it.

**5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
**same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
other's cookie and the loser logs exactly the `No valid authentication` line above. List what
each gate is configured to emit (arguments only — never print a Secret's data):

```
infra/devcontainer/dev.sh run bash -c \
  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
     kubectl --context k3s-prod -n $ns get deploy $ns \
       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
```

A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
evicting each other, and the symptom is intermittent and depends on which app you visited last.

## Likely causes, most likely first

1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
   cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
   dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
   Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
   `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
   access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
   `_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
   gate logs the same line. Look for numbered cookie names in the log, and for a session that
   never survives a single navigation.
3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
   differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
   `/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
   jackett trace above.
4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
   session above was a 5-minute session. If `refresh_token:true` is present and sessions still
   die early, the refresh is failing, not missing.
5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
   the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
   for the `groups:[...]` list on the `[AuthSuccess]` line.
6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.

## When to escalate or stop

- **Stop and hand over if the fix is a manifest change.** Everything above is read-only
  diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
  through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
  reverts, and in the meantime you have an unreproducible cluster.
- **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
  in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
  Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
  transcript here exactly this way.
- **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
  `KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
  2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
- **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
  Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
  report is older than three days, the evidence is gone and the only remaining move is to
  reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
  the reason to close it rather than silence it.