fix(auth): five oauth2-proxy sidecars shared one cookie, died at 5 minutes, and served no metrics #28
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Home-Network/home-network-k3s-impl#28
Reference in new issue
Block a user
Follow-up #3 of
decisions/2026-09-29-crafty-oauth2-proxy-signin.md: "Any further per-app reverse-proxy sidecar needs a unique--cookie-nametoo. The collision is a property of the pattern, not of Crafty."It is a property of the pattern — and the pattern lives in one function, not in five manifests.
The collision, measured on prod (read-only, 2026-09-30)
*-oauth2-proxySecrets holding acookie-secret--cookie-name(so emitting_oauth2_proxy/_oauth2_proxy_csrf)--cookie-domain=.home.keithjackson.devSecrets were compared without printing any value: hashed inside the container, only the first 8 hex chars of each
sha256emitted — enough to say same-or-different, recovering nothing. Key names were listed; no.data/.stringDatawas printed (CLAUDE.md §6).Seven independent instances, seven different keys, one cookie. None can decrypt another's cookie — only overwrite it, and then
ClearSessionexpires it on its own--cookie-domain, i.e. the whole zone.Why it was intermittent, and why reloading never helped
The loser does not merely fail to read the cookie — it calls
ClearSession, which expires the cookie zone-wide. And the *arr UIs poll: radarr's rejections are/api/v3/command,/api/v3/queue/detailsand/signalr/negotiate, 155 of them in 12h. Every poll carrying an undecryptable cookie clears it again, so a left-open Radarr tab deleted Jackett's cookie every few seconds, from another namespace, with no user action at all. Signing in again could not help for longer than the gap between two polls. This is now a comment in the renderer — it is the part of the bug nobody would reconstruct from the flags.Counter-evidence, resolved
I was handed a result that looked bad for this fix: over a 24h Loki window, crafty — the app already carrying the cookie fix — had the worst rejection ratio of the set (
crafty 81:1,radarr 67:1,jackett 39:4). The window straddles a 13h52m-old pod, so I bounded it, reading each sidecar's own log rather than Loki:kubectl logson a pod with 0 restarts cannot reach a pod older than itself, so a window inside the pod's age is inside the fixed spec by construction.reject=No valid authentication in request·success=AuthSuccess·cleared=removing session(ClearSession) ·badsig=cookie signature not validAt n=3 the content is the evidence, so I read crafty's lines rather than counting them — all four auth-relevant lines out of 6743:
Four findings, kept apart:
crafty 81:1is a window artefact and must not be cited against this fix. The fixed pod emitted 3 rejections in 13h52m; the 81 came from the pre-fix process inside the same 24h window.removing session, 0cookie signature not valid, across its whole life. Unfixed radarr: 8 and 8 in 12h. Unfixed jackett: 1 and 1. That is directional and it is the strongest evidence available that the fix works.AuthSuccess, so this is absence of the failure mode, not a session watched working end-to-end. The three rejections are consistent with benign first-hits (one client; two 5s apart, i.e. a page plus a/wsreconnect). I'll say the fix works; I won't say it's been watched working.302-on-XHR (deliberately not).rejectranks chattiness, not breakage — a busy *arr inflates it with nobody inconvenienced.The fix is in the generator, not the manifests
apps/wave-*/{jackett,pihole,qbittorrent,radarr,sonarr}.yamlare generated goldens. One function emits all five:catalog/render.py::_oauth2_sidecar_container(). Hand-editing a golden fails twice over —ci/render-golden.shFAILS on the drift, and the next re-render silently reverts it.A per-service table was rejected for the reason twelve ServiceMonitors once acquired one identical wrong port name in a single edit: it leaves the next app onboarded defaulting straight back into the collision.
--whitelist-domaingoverns post-login redirect targets, not cookie scope; narrowing it breaks the sign-in bounce.Every flag verified against the PINNED v7.8.1 binary
Not against docs, not against memory, and not against the shared proxy — which runs v7.15.3 and therefore could not have vouched for v7.8.1.
podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help:--cookie-name string … (default "_oauth2_proxy")— confirms, from the implementation, the constant this whole check is built on. The central claim — that an omitted flag is a claim on a name, not an absence — now rests on the binary's own help text rather than on my reading of its source.--cookie-refresh duration refresh the cookie after this duration; 0 to disable— confirms the default really is disabled.--metrics-address string— present in v7.8.1, so the new flag cannot crashloop the pods.--api-route strings— singular. The plural--api-routesdoes not exist and the proxy refuses to start with it; that error was propagating through briefs.--session-cookie-minimal— exists, and is the flag mutually exclusive with--cookie-refresh.44180is the platform's convention, measured not chosen: the shared proxy already runs the same flag and port, with ametrics:44180Service port, a ServiceMonitor endpoint namedmetrics, and 2 ready endpoint addresses.Deliberately NOT done
--api-route=^/api/renderer-wide — not merely out of scope, it would be wrong: it was measured on jackett, and PiHole does not serve/api/, so a renderer constant applies one app's path shape to five. Doing it properly means per-service API path prefixes, and theauthblock isadditionalProperties: false— a schema change. Separate PR.--session-cookie-minimalas a cookie-size fix — mutually exclusive with--cookie-refresh("cookie_refresh > 0 requires oauth tokens in sessions"). Dropping--pass-access-tokenis a least-privilege argument, never a size one.:4180, and each namespace's default-deny baseline isingress: [], so Prometheus can neither resolve nor reach:44180until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor land. Nobody should read this commit as "the sidecars have metrics now."--cookie-refreshor metrics. Its subject is cookie identity; those share neither its inputs nor its failure mode, and bundling them would make the check's name a lie and couple unrelated ratchets. The cost of that discipline is named honestly below.The gate check
conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/, declaringconformance.home.k3s/scope: whole-reposoci/l0.sh's--include-scopeunion runs it on every gate invocation. No bespokeci/l0.shstage —whole-repo-invariants-wiredfails the gate if anyone adds one.The modelling decision that is the whole check: an omitted
--cookie-nameis not an absence, it is a claim on the built-in_oauth2_proxy. Reading it as an absence is exactly why this survived five reviews. Then: (1) no two instances share an effective name; (2) no name is another's name +_csrf, since oauth2-proxy derives the CSRF cookie that way and two such session names look distinct while their CSRF cookies collide; (3) the default has exactly one owner, registered with its reason in a shrink-only register; (4) every non-registered instance is host-scoped; (5)catalog/render.pyderives rather than lists; (6) the shipped renderer is proven end-to-end on two synthetic ServiceDefinitions run as a subprocess.On the shared proxy — "distinct" or "explicitly declared"?
Neither, quite.
platform/oauth2-proxy/must keep_oauth2_proxy: it is the single session every forward-auth-gated surface without its own sidecar rides on, so renaming it signs all of them out at once. Requiring every instance to declare a name would need an exemption, and an exemption is a hole — delete the shared proxy and whichever app next omitted the flag would silently inherit the default with the rule still satisfied.So the assertion is plain distinctness over effective names, with the default treated as a name like any other, held by exactly one registered owner. The shared proxy needs no exemption; it is simply the registered holder of that name. A stale register entry fails (the exemption would outlive its object); an unregistered holder fails (no silent heirs). Both were provoked.
On host-scoped
--cookie-domain— overreach, or not? Asserted.Against: distinct names alone close the eviction completely — two differently-named cookies do not overwrite each other however wide their domains are. For, and this wins: a zone-wide cookie is transmitted by the browser to every host in the zone, so each app's session token was being sent to every sibling, including hosts behind no proxy. It is also what made
ClearSessiona cross-namespace weapon (see above), and it makes a future name collision contained instead of platform-wide. The escape hatch is the same register — by name, with a reason — not a loosened rule.Tested in both directions
git archive HEAD+ this test dropped in)_oauth2_proxycollision, the zone-wide domains, the renderer emitting no--cookie-name, both synthetic fixtures renderingNone_oauth2_proxy_sonarr_csrfas a session name--cookie-domainon pihole--cookie-namedeletedA false positive was found and fixed by this testing:
platform/oauth2-proxy/forward-auth-cec.yamlcarriesserviceName: oauth2-proxy/oauth2-proxy:80— an Envoynamespace/service:portreference, not an image. The scan is now anchored onimage:, and the near-miss is documented in the check.Gate
Local (toolbox):
rc 3 (UNDETERMINED). Green:spec-immutability,decisions-log,lint,l0,render-golden,l1,approval-policy. Skipped:gitea-settings(no token locally),conformance(declared,CLUSTER_AVAILABLEunset). rc 3 is UNDETERMINED, not green. The new L0 check PASSES on this branch — all five apps fixed in one commit, so the sequencing problem is gone.Touched requirements
R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-PLAT-02 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04are UNVERIFIED at L2/L3.Tooling finding worth propagating:
spec-immutabilitywas skipping because of a path bug in the worktree ENGINE shim, not the environment.dev.shderives the spec path from its ownREPO_ROOT, which in a worktree is.claude/worktrees/agent-*/, so../home-network-k3sdoes not resolve and it adds no spec mount — and the skip then reads as "the one tolerated skip". Mounting the spec repo at/home-network-k3s(the pathci/spec-immutability.shactually defaults to) makes R-TEST-06 genuinely run and pass. CLAUDE.md §3's two-mount shim should say so.Why CI is red, and why that is correct
On the previous head,
gate / gate (pull_request)failed for exactly one reason, read from the job log rather than the tick:Every substantive stage was green in CI —
gitea-settings: pass(CI has the token my local run does not),decisions-log,lint,l0,render-golden,l1. So CI is a stricter and more complete run than mine on every axis except the spec repo, and it agrees.This PR touches
conformance/**, whichinfra/gitea/approval-policy.jsonlists as sensitive — "the tests and the harness; changing them changes what green MEANS". A change to the gate cannot be gated by the gate, so it is fail-closed pending an approving review from someone who is not its author. The red tick is the policy; it clears when a human reviews.Note also that
gate / conformance (pull_request)reportssuccesswith description "Has been skipped" — the exact trap CLAUDE.md §2b names. It certifies nothing.The jackett overlap, and the general lesson
With the renderer fixed and jackett's golden left at its
maincontent,./ci/render-golden.shexits 1:So the three options were: hand-edit the goldens and leave the generator broken (fails
render-golden, reverts on the next render); add a per-service carve-out so jackett alone keeps defaulting (ships the exact "list, don't derive" anti-pattern this check forbids); or fix the generator and regenerate all five. The third is the only correct one.Both branches independently reached the same two values (
_oauth2_proxy_jackett,jackett.${BASE_DOMAIN}) — a good sign for the diagnosis. The concurrent branch is now redundant, but its evidence is not, and it is carried into this PR and the renderer's comments: the interleaved radarr/jackett log showingremoving sessiondeleting a clean session 26s after login, and theexpires − created = 300.000smeasurement that produced--cookie-refresh. Credit where due; the branch should be closed rather than rebased, since its hand-edit of a rendered file will failrender-goldenon its own terms.The general lesson, worth more than this bug: an ownership split across
apps/wave-*/*.yamlcannot hold, because those files are outputs. Two agents assigned two rendered apps are assigned the same function and will collide inrender-goldenno matter how carefully they stay in their own files. Ownership of a rendered app is ownership ofcatalog/render.py, and briefs should say so.Not verified
AuthSuccess, so no session has been observed working end-to-end on a fixed instance. Re-measure after real sign-in traffic.--cookie-refresh=2m30shas never been exercised. The value is derived from a measured 300.000s lifetime and asserted to sit inside(0, 300s), but no session has been held past the five-minute cliff to watch a refresh succeed. If Keycloak's refresh-token or SSO idle timeouts are shorter than expected, the symptom moves rather than disappears. This is the fix most likely to need a second pass.metricsport has never been fetched. Listener start is asserted by flag presence only, and it is unreachable by design until the scrape wiring lands.--cookie-refreshand--metrics-addressare UNGATED. Nothing inci/or.gitea/runscatalog/render.py --selftest— I grepped. So the six assertions I added there are dev-time only: an edit that drops either flag and re-renders would pass the whole merge gate. The repo already has the precedent and the one-line shape of the fix —conformance/tests/l0-static/auditability-proofs-producible/check.sh:291drives other tools'--selftestfor exactly this reason, its header saying it does so "because nothing [else] drives them". Highest-value follow-up here; not done because it widens an already review-blocked PR into a third subject.--help. Merging is what deploys — it rolls five Deployments, invalidating every existing session on them once. radarr/sonarr/qbittorrent usestrategy: Recreateon RWO volumes, so there is a brief gap per app. This now changes three flags plus a port on all five, so a bad--cookie-refreshwould surface as five apps misbehaving, not one.kubectl logs --since=returns only what the kubelet retains, andrejectconflates unauthenticated first hits with XHR polls. Thebadsig/clearedpair is the only collision-specific signal, and nothing gates it.vars.CLUSTER_AVAILABLEis still unset in CI, so even a green tick proves the L0/L1 offline subset and nothing about runtime.Full handoff:
decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md🤖 Generated with Claude Code
Every oauth2-proxy on this platform is an independent instance with its OWN cookie-secret. Measured on prod 2026-09-30: seven `*-oauth2-proxy` Secrets, seven DISTINCT cookie-secret values (compared in-cluster, never printed). So no instance can DECRYPT another's cookie -- it can only OVERWRITE it. All of them were emitting oauth2-proxy's DEFAULT names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to `.${BASE_DOMAIN}`, so the browser kept ONE cookie pair for SIX live instances (jackett, pihole, qbittorrent, radarr, sonarr, and the shared forward-auth proxy). Whichever authenticated last clobbered the rest; the loser logged `No valid authentication in request`, bounced to Keycloak, and -- the SSO session still being live -- was redirected straight back and clobbered it again. The symptom is "I signed into Keycloak but I'm still not signed in", which sends you hunting in Keycloak, where nothing is wrong. Crafty was fixed on its own side on 2026-09-29 (#22); this is that fix generalized to the pattern. THE DEFECT WAS NOT IN THE MANIFESTS. apps/wave-*/{jackett,pihole,qbittorrent,radarr, sonarr}.yaml are GENERATED goldens; one function emits all five. So the fix is in catalog/render.py::_oauth2_sidecar_container(), and both flags are now DERIVED from the ServiceDefinition rather than defaulted: --cookie-name=_oauth2_proxy_{s['name']} unique by construction --cookie-domain={host} the app's own host, not the zone A per-service table was rejected for the reason twelve ServiceMonitors once acquired one identical wrong port name in a single edit: it would leave the NEXT app onboarded defaulting straight back into the collision. --whitelist-domain stays zone-wide; it governs post-login REDIRECT targets, not cookie scope. Host-scoping is defence in depth and is asserted deliberately: a zone-wide cookie is transmitted by the browser to every host in the zone, so each app's session token was being sent to every sibling -- and it is what would make a FUTURE name collision platform-wide instead of contained. New gate check (whole-repo scope, so it runs on EVERY gate invocation, not only when a touched path happens to attribute an identity requirement): conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/ It resolves each instance's EFFECTIVE cookie name -- declared, or oauth2-proxy's built-in default when omitted, because an omitted flag is not an absence but a CLAIM on a name, and reading it as an absence is exactly why this survived five reviews. Then: no two collide; none is another's name + `_csrf` (oauth2-proxy derives the CSRF cookie that way); the default has exactly ONE owner, registered with its reason in a shrink-only register -- the SHARED proxy, which MUST keep `_oauth2_proxy` because renaming it signs every other gated surface out at once; every other instance is host-scoped; render.py derives rather than lists; and the SHIPPED renderer is proven end-to-end on two synthetic ServiceDefinitions, so the check stays falsifiable once the tree is clean. Tested in both directions: rc 0 on this tree (39 assertions, 7 instances), rc 1 with 26 failures against the pre-fix tree, plus one targeted mutation per assertion class (duplicate name, CSRF derivation, zone-wide domain, unregistered default heir, stale register entry) -- each fired, and nothing else did. Also regenerates the three L1 rendering goldens, which are expected-output fixtures of the same renderer. UNVERIFIED: no browser round-trip. The collision has still never been REPRODUCED in a browser -- doing so needs two gated apps visited in one session -- so this is reasoned from measured configuration and from the mechanism, not from a failure anyone watched. Nothing was applied; merging rolls five Deployments and invalidates every existing session on them once. See decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>Counter-evidence was raised against this branch: over a 24h Loki window crafty -- the app already carrying the cookie fix -- showed the WORST auth-rejection ratio of the set (crafty 81:1, radarr 67:1, jackett 39:4). Taken at face value that says the fix does not work. It does not say that, and it does not say the opposite either. Re-measured bounded to the fixed pod, read from each sidecar's OWN log rather than Loki (`kubectl logs` on a pod with 0 restarts cannot reach a pod older than itself, so a window inside the pod's age is inside the fixed spec by construction): namespace age(h) rst reject success ratio cleared badsig crafty 13.9 0 3 0 n/a 0 0 <- FIXED, whole life radarr 422.5 0 155 1 155.0:1 8 8 jackett 422.6 0 39 4 9.8:1 1 1 pihole/qbittorrent/sonarr/shared: 0 reject, 0 success Three findings, kept apart: 1. `crafty 81:1` is a WINDOW ARTEFACT. The fixed pod has emitted 3 rejections in its entire 13.9h life; the 81 came from a 24h window straddling it, i.e. from the pre-fix pod. It is not evidence against the fix and must not be cited as such. 2. The fix is still NOT SHOWN TO WORK. Crafty logged ZERO AuthSuccess in that window -- no traffic, so no ratio. Absence of failure against absence of traffic proves nothing. 3. The collision mechanism IS independently confirmed, and it is a MINORITY of the rejections. `cookie signature not valid` + `removing session` is its fingerprint -- a cookie IS present, written with a key this instance does not have, so it cannot decrypt it and then ClearSession expires it zone-wide. radarr 8 of 155 (5%), jackett 1 of 39, fixed crafty 0. ~95% of radarr's rejections carry no cookie at all and point at the session-lifetime and 302-on-XHR mechanisms this branch does not touch. So the handoff now says, at the top and in both evidence sections, that "the instances collide" and "the collisions cause the reported symptom" are different claims and only the first is supported. The four fixes stand -- distinct cookie names are correct regardless -- but nothing here should be read as a cure. Also records three deliberate NON-fixes with their reasoning (--cookie-refresh, which all four of my apps need as much as jackett does and which belongs in the renderer rather than in a golden; --api-routes for 302-on-XHR, which is path-shaped and needs a schema change; and sidecar metrics, which I judged overreach for a check whose subject is cookie identity), and two gate facts: * spec-immutability was skipping because of a PATH BUG IN THE WORKTREE SHIM, not the environment -- dev.sh derives the spec path from its own REPO_ROOT, which in a worktree does not resolve. Mounting the spec repo at /home-network-k3s with SPEC_REPO set makes R-TEST-06 actually run and PASS. Local gate is now rc 3 for one reason (gitea-settings, no token) instead of two. * CI's gate reports FAILURE on75a4029for exactly one reason, and it is not this change: `approval-policy: a sensitive change lacks a non-author approval`. Every substantive stage is green in CI, including gitea-settings. conformance/** is a sensitive path, so the PR is fail-closed pending a human review -- CLAUDE.md §1 working as designed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>Widens the cookie-collision fix to the whole sidecar contract, because all three defects live in the same function and land on the same five goldens, so they are one atomic change or they are five conflicting ones. Added to _oauth2_sidecar_container(), on top of the cookie-name/cookie-domain fix: --cookie-refresh=2m30s The session died FIVE MINUTES after sign-in. Keycloak's homelab realm issues 300-second access tokens (MEASURED on a live session: expires - created = 300.000s exactly, with refresh_token:true on the same line), and --cookie-refresh defaults to 0 -- which the pinned binary's own help spells "0 to disable". So the proxy held a refresh token it was never told to spend. Refresh at HALF the lifetime: two chances to renew before expiry. This, not the collision, is what dominates the rejection VOLUME -- ~95% of radarr's 155 rejections in 12h carry no cookie at all. --metrics-address=0.0.0.0:44180 + a `metrics` container port Without the flag the /metrics listener never starts, so no per-app sidecar was serving metrics at all (`ports` was ['http'] on all six). 44180 is the platform's existing convention, measured not chosen: the shared proxy already runs the same flag and port with 2 ready endpoints. PREREQUISITE ONLY -- NOT scraped. A container port is not a Service port, and each namespace's default-deny baseline is `ingress: []`, so this is unreachable until a Service port (or PodMonitor), a monitoring allow and a ServiceMonitor land. Do not read this as "the sidecars have metrics now". EVERY FLAG VERIFIED AGAINST THE PINNED v7.8.1 BINARY (`--help`), not against docs or the shared proxy -- which runs v7.15.3 and could not have vouched for v7.8.1: * --metrics-address exists in 7.8.1, so this cannot crashloop the pods * --cookie-refresh ... "0 to disable" confirms the default * --cookie-name ... (default "_oauth2_proxy") -- this confirms from the IMPLEMENTATION the constant the new L0 check is built on * --api-route is SINGULAR; the plural does not exist and the proxy refuses to start * --session-cookie-minimal exists and is mutually exclusive with --cookie-refresh, so dropping --pass-access-token is a least-privilege argument, never a size fix NOT done, deliberately: --api-route=^/api/ renderer-wide. It was measured on jackett and PiHole does not serve /api/, so a renderer constant would apply one app's path shape to five. It needs a per-app hook the `auth` schema forbids (additionalProperties: false) -- a schema change, separate PR. Also records the mechanism that made this INTERMITTENT, as a renderer comment, because nobody would reconstruct it from the flags: the loser does not just fail to read the cookie, it calls ClearSession, which expires the cookie on its own --cookie-domain -- the whole zone. The *arr UIs poll (/api/v3/command, /api/v3/queue/details, /signalr/negotiate), so every poll carrying an undecryptable cookie clears it again: A LEFT-OPEN RADARR TAB DELETED JACKETT'S COOKIE EVERY FEW SECONDS, from another namespace, with no user action. That is why signing in again never helped for longer than the gap between two polls. Crafty's counter-evidence resolved: the 81:1 ratio was a 24h window straddling a 13h52m-old pod. Bounded to the fixed process, crafty's ENTIRE life is 3 rejections in 6743 log lines, with ZERO `removing session` and ZERO `cookie signature not valid` against unfixed radarr's 8 and 8. The eviction fingerprint is gone where the fix is deployed. It logged 0 AuthSuccess too, so that is absence-of-the-failure-mode rather than a session watched working -- stated as such in the handoff. Renderer selftest gains six assertions, including a --cookie-refresh check that PARSES the duration and requires 0 < refresh < 300s (so 2m30s/150s/2.5m all stay green while a silent widening past the token lifetime does not). NOTE, and it is in the handoff as the top follow-up: NOTHING IN ci/ RUNS `render.py --selftest`, so those assertions are dev-time only. conformance/tests/l0-static/auditability-proofs-producible/check.sh:291 already drives other tools' selftests for exactly this reason and is the one-line fix. Gate (toolbox, worktree shim with the spec repo mounted at /home-network-k3s): rc 3 UNDETERMINED. Green: spec-immutability, decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped: gitea-settings (no local token), conformance (declared). The new L0 check PASSES on this branch -- all five apps fixed in one commit. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>fix(auth): five oauth2-proxy sidecars shared one cookie and evicted each otherto fix(auth): five oauth2-proxy sidecars shared one cookie, died at 5 minutes, and served no metrics