home-network-k3s-impl/runbooks/oauth2-proxy-auth-failures.md

oauth2-proxy auth failures — telling the three layers apart

Linked from OAuth2ProxyGateUnobserved, and the page to open whenever a gated app "is up but does not work": a UI that loads and then shows nothing, a page that never finishes, a list that stays empty, a save that silently does not save. The alert is about the visibility gap; this page is the diagnostic procedure, and it works whether or not the alert ever fired.

What this means here

Every app behind Keycloak on this platform has three places a request can be refused, and from a browser they look identical — because two of the three answer with HTTP 200.

Layer	What refuses	What the browser sees
1. The gate (oauth2-proxy sidecar)	no valid session cookie	302 to Keycloak, then usually 200
2. The gate's group check	session valid, wrong group	403, or a redirect loop
3. The app itself	the app's own login / API key	the app's own 401 or login page

Layer 1 is the one that hides. oauth2-proxy answers an unauthenticated request with a redirect, not a 401 — correct for a browser navigation, disastrous for an XHR. The app's JavaScript asks for JSON, the gate answers 302, the browser follows it silently, the live Keycloak SSO session satisfies it with no prompt, and the XHR resolves 200 with an HTML dashboard in the body. fetch() sees a success. The user sees an empty page. Reloading does nothing, and nobody is ever asked to log in, because from the browser's point of view nothing failed.

The worked example, jackett on 2026-09-30, straight from the sidecar log:

[oauthproxy.go:1026] No valid authentication in request. Initiating login.
GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
... ten of these at 15:30:53
[AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
              expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
GET "/UI/Dashboard"        200 44379
GET "/api/v2.0/indexers?_=..." 200 1501229
... and 26 seconds later, inside that same 5-minute session:
[oauthproxy.go:1026] No valid authentication in request. Initiating login.

A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the signature of the session being re-established over and over instead of held — a cookie that another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being down and not of the app being broken.

Read this before you trust any silence (updated 2026-10-01). The five fleet gate sidecars now do run a metrics listener on :44180, and platform/oauth2-proxy/podmonitor.yaml discovers them — but the scrape is dropped by the app namespaces' default-deny NetworkPolicy, so those targets read up == 0 and the TSDB holds nothing for them. Crafty has no listener at all. Only the shared oauth2-proxy namespace instance is genuinely scraped, and it is not in the data path for any of these apps. So there is still nothing to graph and nothing that could have alerted. The logs below remain the only record that this class of failure happened. Check which state you are in before you trust a quiet dashboard — "First commands" step 1 tells you in one query.

First commands

kubectl does not exist on the workstation; everything runs in the pinned container, and always with an explicit context.

1. Which namespaces have an unobserved gate (this is what the alert measures):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"} == 1)"'

1b. If a namespace is listed, is it "no target" or "target failing"? The two need different fixes and the alert deliberately does not distinguish them:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
     "up{container=\"oauth2-proxy\"}"'

No series for the namespace → nothing discovers it (no PodMonitor match; check the pod carries homelab.keithjackson.dev/auth-mode=forward-auth and a named metrics port). A series with value 0 → discovered but every scrape fails; get the reason from the target's lastError:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s localhost:19090/api/v1/targets | python3 -c "
import json,sys
for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
    if t[\"labels\"].get(\"container\")==\"oauth2-proxy\":
        print(t[\"labels\"].get(\"namespace\"), t[\"health\"], t.get(\"lastError\",\"\")[:90])"'

context deadline exceeded is a drop, not a refusal — that is NetworkPolicy, not a dead endpoint. Confirm the endpoint itself is fine by port-forwarding straight to the pod, which bypasses policy entirely:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n jackett port-forward \
     $(kubectl --context k3s-prod -n jackett get pod -l app.kubernetes.io/name=jackett -o name | head -1) \
     14418:44180 & sleep 5; curl -s -o /dev/null -w "http=%{http_code} bytes=%{size_download}\n" \
     http://127.0.0.1:14418/metrics'

http=200 with a few KB means the listener is healthy and the path is what is broken. As of 2026-10-01 that is the state of all five fleet apps, and the fix is an ingress allowance from ns monitoring to port 44180 — see platform/longhorn/networkpolicy-metrics.yaml for the idiom.

2. The sidecar log — but you must grep it. The gate's stdout is ~11,500 lines a day per app and almost all of it is the kubelet health probe (GET "/ping" ... 200). A plain kubectl logs --tail=100 shows you nothing but /ping and looks perfectly healthy. Filter:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'

Substitute the namespace: crafty, jackett, pihole, qbittorrent, radarr, sonarr.

3. The same question across every gate at once, from Loki. Alloy collects all of these, so you can ask about the whole fleet and about a window that has already scrolled out of kubectl logs. Port-forward Loki and query it:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '

and the denominator, which is what makes the number mean something:

   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...

How to read the ratio. A healthy sign-in is roughly one or two No valid authentication lines per AuthSuccess — the first request of a session is unauthenticated by definition. A ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):

namespace	No valid authentication	AuthSuccess	ratio
crafty	81	1	81:1
radarr	67	1	67:1
jackett	39	4	~10:1
pihole, qbittorrent, sonarr	0	0	(unused in the window)

All three of those are the failure, not the baseline.

The cleanest worked case of all — radarr, measured 2026-10-01. After the cookie fix landed, jackett dropped to 0 No valid authentication lines in 13,612 (from 39), and removing session / cookie signature not valid went to 0 on both apps (radarr had 8 and 8 before). The eviction fingerprint is gone. But radarr still logged 206 rejections — and sampling 209 of them, every single one was the same request:

GET "/signalr/messages/negotiate" ... 302    <- browser UA, over and over

That is one left-open browser tab. Its cookie is gone, its SignalR socket keeps trying to reconnect, and every attempt is silently 302'd to Keycloak. Nobody is at the keyboard, so nobody is ever re-prompted; the tab just sits there looking like a working page that has stopped updating. This is the entire failure class in one line: a non-navigation request answered with a redirect, forever, with no error surfaced anywhere.

So when you read a rejection count, always group by request path before concluding anything:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n radarr logs deploy/radarr -c oauth2-proxy --since=24h \
     | grep -v "/ping" | grep "No valid authentication" -A 1 \
     | grep -oE "\"[^\"]+\" HTTP" | sort | uniq -c | sort -rn | head'

One path dominating means an orphaned client, not a broken gate — annoying but harmless, and it clears when the tab is closed. Many different paths from one session means the session itself is not surviving, which is the real defect. A per-app --api-route is the structural fix for the first shape: it makes those paths answer 401 instead of 302, so the client fails loudly instead of looping. Note that a count of 0 AuthSuccess alongside a high rejection count means nobody has signed in since the last restart — so --cookie-refresh has never run, and no session-lifecycle conclusion can be drawn from that window at all.

4. Did the gate admit the user, or did the app refuse them? Look for the [AuthSuccess] line and what follows it. If [AuthSuccess] appears and the very next thing is the app's own login page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty shape, and runbooks/crafty-admin-credential.md covers it.

5. Cookie names, when sessions do not survive. Two oauth2-proxy instances emitting the same cookie name on the same domain hold different cookie-secrets, so each overwrites the other's cookie and the loser logs exactly the No valid authentication line above. List what each gate is configured to emit (arguments only — never print a Secret's data):

infra/devcontainer/dev.sh run bash -c \
  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
     kubectl --context k3s-prod -n $ns get deploy $ns \
       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'

A gate with no --cookie-name is using oauth2-proxy's default _oauth2_proxy. If two of them share that default and a zone-wide --cookie-domain=.home.keithjackson.dev, they are evicting each other, and the symptom is intermittent and depends on which app you visited last.

Likely causes, most likely first

1. Cookie collision between two gates on one domain. Default cookie names plus a zone-wide cookie domain. Signature: high No valid authentication-to-AuthSuccess ratio, sessions dying seconds after being created, symptom depends on browsing order, and nothing wrong at Keycloak. Fix is a unique --cookie-name and a host-scoped --cookie-domain per gate — see decisions/2026-09-29-crafty-oauth2-proxy-signin.md, which found and fixed exactly this.
2. The cookie is too large to come back. With --pass-access-token the session carries the access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into _oauth2_proxy_0, _oauth2_proxy_1… If a chunk is dropped the session is unreadable and the gate logs the same line. Look for numbered cookie names in the log, and for a session that never survives a single navigation.
3. The XHR path is not the navigation path. The app's API is under a prefix the gate handles differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature: /UI/... returns 200 and /api/... returns 302 in the same second, which is precisely the jackett trace above.
4. Session lifetime shorter than the app's polling interval, with no refresh. The jackett session above was a 5-minute session. If refresh_token:true is present and sessions still die early, the refresh is failing, not missing.
5. Group check, not authentication. --allowed-group is a floor: a valid Keycloak session in the wrong group is refused after a successful login, which reads as a loop. Look for 403 and for the groups:[...] list on the [AuthSuccess] line.
6. The app, not the gate. [AuthSuccess] followed by the app's own login page or 401.

When to escalate or stop

• Stop and hand over if the fix is a manifest change. Everything above is read-only diagnosis. Changing a --cookie-name, adding --metrics-address, or editing an alert goes through a branch and a PR; kubectl edit on a gate is drift that the next Flux reconcile reverts, and in the meantime you have an unreproducible cluster.
• Stop if you find yourself about to print a Secret. The cookie-secret and client-secret live in oauth2-proxy-credentials and the per-app equivalents. Compare key names and whether two Secrets differ; never dump .data or .stringData. A password has been leaked into a transcript here exactly this way.
• Escalate to the platform owner if Keycloak itself is the suspect — check KeycloakDown and KeycloakDatabaseDown first (runbooks/keycloak-down.md, runbooks/keycloak-db.md). In the 2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
• The honest limit of this page. With no metrics from the sidecars there is no history beyond Loki's 72h retention (platform/monitoring/loki.yaml, retention_period: 72h). If the report is older than three days, the evidence is gone and the only remaining move is to reproduce it. That is the cost of the gap OAuth2ProxyGateUnobserved is tracking, and it is the reason to close it rather than silence it.