home-network-k3s-impl/runbooks/oauth2-proxy-auth-failures.md

oauth2-proxy auth failures — telling the three layers apart

Linked from OAuth2ProxyGateUnobserved, and the page to open whenever a gated app "is up but does not work": a UI that loads and then shows nothing, a page that never finishes, a list that stays empty, a save that silently does not save. The alert is about the visibility gap; this page is the diagnostic procedure, and it works whether or not the alert ever fired.

What this means here

Every app behind Keycloak on this platform has three places a request can be refused, and from a browser they look identical — because two of the three answer with HTTP 200.

Layer	What refuses	What the browser sees
1. The gate (oauth2-proxy sidecar)	no valid session cookie	302 to Keycloak, then usually 200
2. The gate's group check	session valid, wrong group	403, or a redirect loop
3. The app itself	the app's own login / API key	the app's own 401 or login page

Layer 1 is the one that hides. oauth2-proxy answers an unauthenticated request with a redirect, not a 401 — correct for a browser navigation, disastrous for an XHR. The app's JavaScript asks for JSON, the gate answers 302, the browser follows it silently, the live Keycloak SSO session satisfies it with no prompt, and the XHR resolves 200 with an HTML dashboard in the body. fetch() sees a success. The user sees an empty page. Reloading does nothing, and nobody is ever asked to log in, because from the browser's point of view nothing failed.

The worked example, jackett on 2026-09-30, straight from the sidecar log:

[oauthproxy.go:1026] No valid authentication in request. Initiating login.
GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
... ten of these at 15:30:53
[AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
              expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
GET "/UI/Dashboard"        200 44379
GET "/api/v2.0/indexers?_=..." 200 1501229
... and 26 seconds later, inside that same 5-minute session:
[oauthproxy.go:1026] No valid authentication in request. Initiating login.

A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the signature of the session being re-established over and over instead of held — a cookie that another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being down and not of the app being broken.

Read this before you trust any silence: the per-app gate sidecars export no metrics. They run without --metrics-address, so oauth2-proxy's metrics listener never starts; they declare no metrics port; and no ServiceMonitor or PodMonitor selects them. Only the shared oauth2-proxy namespace instance is scraped, and it is not in the data path for any of these apps. So there is nothing to graph and nothing that could have alerted. The logs below are not a convenience — until that changes they are the only record that this class of failure happened.

First commands

kubectl does not exist on the workstation; everything runs in the pinned container, and always with an explicit context.

1. Which namespaces have an unobserved gate (this is what the alert measures):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"})"'

2. The sidecar log — but you must grep it. The gate's stdout is ~11,500 lines a day per app and almost all of it is the kubelet health probe (GET "/ping" ... 200). A plain kubectl logs --tail=100 shows you nothing but /ping and looks perfectly healthy. Filter:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'

Substitute the namespace: crafty, jackett, pihole, qbittorrent, radarr, sonarr.

3. The same question across every gate at once, from Loki. Alloy collects all of these, so you can ask about the whole fleet and about a window that has already scrolled out of kubectl logs. Port-forward Loki and query it:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '

and the denominator, which is what makes the number mean something:

   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...

How to read the ratio. A healthy sign-in is roughly one or two No valid authentication lines per AuthSuccess — the first request of a session is unauthenticated by definition. A ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):

namespace	No valid authentication	AuthSuccess	ratio
crafty	81	1	81:1
radarr	67	1	67:1
jackett	39	4	~10:1
pihole, qbittorrent, sonarr	0	0	(unused in the window)

All three of those are the failure, not the baseline.

4. Did the gate admit the user, or did the app refuse them? Look for the [AuthSuccess] line and what follows it. If [AuthSuccess] appears and the very next thing is the app's own login page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty shape, and runbooks/crafty-admin-credential.md covers it.

5. Cookie names, when sessions do not survive. Two oauth2-proxy instances emitting the same cookie name on the same domain hold different cookie-secrets, so each overwrites the other's cookie and the loser logs exactly the No valid authentication line above. List what each gate is configured to emit (arguments only — never print a Secret's data):

infra/devcontainer/dev.sh run bash -c \
  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
     kubectl --context k3s-prod -n $ns get deploy $ns \
       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'

A gate with no --cookie-name is using oauth2-proxy's default _oauth2_proxy. If two of them share that default and a zone-wide --cookie-domain=.home.keithjackson.dev, they are evicting each other, and the symptom is intermittent and depends on which app you visited last.

Likely causes, most likely first

1. Cookie collision between two gates on one domain. Default cookie names plus a zone-wide cookie domain. Signature: high No valid authentication-to-AuthSuccess ratio, sessions dying seconds after being created, symptom depends on browsing order, and nothing wrong at Keycloak. Fix is a unique --cookie-name and a host-scoped --cookie-domain per gate — see decisions/2026-09-29-crafty-oauth2-proxy-signin.md, which found and fixed exactly this.
2. The cookie is too large to come back. With --pass-access-token the session carries the access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into _oauth2_proxy_0, _oauth2_proxy_1… If a chunk is dropped the session is unreadable and the gate logs the same line. Look for numbered cookie names in the log, and for a session that never survives a single navigation.
3. The XHR path is not the navigation path. The app's API is under a prefix the gate handles differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature: /UI/... returns 200 and /api/... returns 302 in the same second, which is precisely the jackett trace above.
4. Session lifetime shorter than the app's polling interval, with no refresh. The jackett session above was a 5-minute session. If refresh_token:true is present and sessions still die early, the refresh is failing, not missing.
5. Group check, not authentication. --allowed-group is a floor: a valid Keycloak session in the wrong group is refused after a successful login, which reads as a loop. Look for 403 and for the groups:[...] list on the [AuthSuccess] line.
6. The app, not the gate. [AuthSuccess] followed by the app's own login page or 401.

When to escalate or stop

• Stop and hand over if the fix is a manifest change. Everything above is read-only diagnosis. Changing a --cookie-name, adding --metrics-address, or editing an alert goes through a branch and a PR; kubectl edit on a gate is drift that the next Flux reconcile reverts, and in the meantime you have an unreproducible cluster.
• Stop if you find yourself about to print a Secret. The cookie-secret and client-secret live in oauth2-proxy-credentials and the per-app equivalents. Compare key names and whether two Secrets differ; never dump .data or .stringData. A password has been leaked into a transcript here exactly this way.
• Escalate to the platform owner if Keycloak itself is the suspect — check KeycloakDown and KeycloakDatabaseDown first (runbooks/keycloak-down.md, runbooks/keycloak-db.md). In the 2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
• The honest limit of this page. With no metrics from the sidecars there is no history beyond Loki's 72h retention (platform/monitoring/loki.yaml, retention_period: 72h). If the report is older than three days, the evidence is gone and the only remaining move is to reproduce it. That is the cost of the gap OAuth2ProxyGateUnobserved is tracking, and it is the reason to close it rather than silence it.