fix(auth): five oauth2-proxy sidecars shared one cookie and evicted each other
Every oauth2-proxy on this platform is an independent instance with its OWN
cookie-secret. Measured on prod 2026-09-30: seven `*-oauth2-proxy` Secrets, seven
DISTINCT cookie-secret values (compared in-cluster, never printed). So no instance
can DECRYPT another's cookie -- it can only OVERWRITE it.
All of them were emitting oauth2-proxy's DEFAULT names (`_oauth2_proxy`,
`_oauth2_proxy_csrf`) scoped to `.${BASE_DOMAIN}`, so the browser kept ONE cookie
pair for SIX live instances (jackett, pihole, qbittorrent, radarr, sonarr, and the
shared forward-auth proxy). Whichever authenticated last clobbered the rest; the
loser logged `No valid authentication in request`, bounced to Keycloak, and -- the
SSO session still being live -- was redirected straight back and clobbered it again.
The symptom is "I signed into Keycloak but I'm still not signed in", which sends you
hunting in Keycloak, where nothing is wrong. Crafty was fixed on its own side on
2026-09-29 (#22); this is that fix generalized to the pattern.
THE DEFECT WAS NOT IN THE MANIFESTS. apps/wave-*/{jackett,pihole,qbittorrent,radarr,
sonarr}.yaml are GENERATED goldens; one function emits all five. So the fix is in
catalog/render.py::_oauth2_sidecar_container(), and both flags are now DERIVED from
the ServiceDefinition rather than defaulted:
--cookie-name=_oauth2_proxy_{s['name']} unique by construction
--cookie-domain={host} the app's own host, not the zone
A per-service table was rejected for the reason twelve ServiceMonitors once acquired
one identical wrong port name in a single edit: it would leave the NEXT app onboarded
defaulting straight back into the collision. --whitelist-domain stays zone-wide; it
governs post-login REDIRECT targets, not cookie scope.
Host-scoping is defence in depth and is asserted deliberately: a zone-wide cookie is
transmitted by the browser to every host in the zone, so each app's session token was
being sent to every sibling -- and it is what would make a FUTURE name collision
platform-wide instead of contained.
New gate check (whole-repo scope, so it runs on EVERY gate invocation, not only when
a touched path happens to attribute an identity requirement):
conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/
It resolves each instance's EFFECTIVE cookie name -- declared, or oauth2-proxy's
built-in default when omitted, because an omitted flag is not an absence but a CLAIM
on a name, and reading it as an absence is exactly why this survived five reviews.
Then: no two collide; none is another's name + `_csrf` (oauth2-proxy derives the CSRF
cookie that way); the default has exactly ONE owner, registered with its reason in a
shrink-only register -- the SHARED proxy, which MUST keep `_oauth2_proxy` because
renaming it signs every other gated surface out at once; every other instance is
host-scoped; render.py derives rather than lists; and the SHIPPED renderer is proven
end-to-end on two synthetic ServiceDefinitions, so the check stays falsifiable once
the tree is clean.
Tested in both directions: rc 0 on this tree (39 assertions, 7 instances), rc 1 with
26 failures against the pre-fix tree, plus one targeted mutation per assertion class
(duplicate name, CSRF derivation, zone-wide domain, unregistered default heir, stale
register entry) -- each fired, and nothing else did.
Also regenerates the three L1 rendering goldens, which are expected-output fixtures
of the same renderer.
UNVERIFIED: no browser round-trip. The collision has still never been REPRODUCED in a
browser -- doing so needs two gated apps visited in one session -- so this is reasoned
from measured configuration and from the mechanism, not from a failure anyone watched.
Nothing was applied; merging rolls five Deployments and invalidates every existing
session on them once. See decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
0fbf544d5e
commit
9a4489b733
12 files changed
+930
-10
No files matched your search
+85
-2
@@ -971,7 +971,11 @@ def _oauth2_sidecar_container(s: dict) -> dict:
|
||||
auth.group (R-IDP-06: a valid session is not enough — the user must be in an authorized group,
|
||||
else /oauth2 returns 403). client-id/secret + cookie-secret come by name from the SOPS-encrypted
|
||||
`<app>-oauth2-proxy` Secret (R-IDP-04). Single provider => --skip-provider-button 302s straight
|
||||
to Keycloak. The sidecar->app hop is pod-local (127.0.0.1) and not subject to network policy."""
|
||||
to Keycloak. The sidecar->app hop is pod-local (127.0.0.1) and not subject to network policy.
|
||||
|
||||
--cookie-name and --cookie-domain are DERIVED per app (see the block comment on those two
|
||||
args): every instance holds a different cookie-secret, so sharing one cookie name across the
|
||||
zone makes the instances evict each other's sessions."""
|
||||
route = s["ingressRoutes"][0]
|
||||
host = route["host"]
|
||||
upstream_port = route["port"]
|
||||
@@ -991,7 +995,41 @@ def _oauth2_sidecar_container(s: dict) -> dict:
|
||||
args += [
|
||||
"--skip-provider-button=true",
|
||||
f"--redirect-url=https://{host}/oauth2/callback",
|
||||
f"--cookie-domain=.{ZONE}",
|
||||
# ── COOKIE ISOLATION (fleet fix 2026-09-30; crafty's 2026-09-29 fix, generalized) ──────
|
||||
# EVERY oauth2-proxy on this platform — this sidecar, its four siblings, and the SHARED
|
||||
# forward-auth proxy (platform/oauth2-proxy) — is an INDEPENDENT instance holding its OWN
|
||||
# `cookie-secret`. Measured on prod 2026-09-30: seven `*-oauth2-proxy` Secrets, seven
|
||||
# DISTINCT cookie-secret values, so no instance can decrypt any other's cookie.
|
||||
#
|
||||
# Left at oauth2-proxy's DEFAULTS (`_oauth2_proxy` / `_oauth2_proxy_csrf`) and scoped to
|
||||
# `.${BASE_DOMAIN}`, the browser therefore keeps ONE cookie pair for ALL of them:
|
||||
# whichever proxy authenticates last OVERWRITES every other's session. The loser logs
|
||||
# "No valid authentication in request" and bounces to Keycloak, which — the SSO session
|
||||
# still being live — redirects straight back and overwrites the cookie again. The
|
||||
# user-visible symptom is "I signed into Keycloak but I'm still not signed in", and it
|
||||
# sends you hunting in Keycloak, where nothing is wrong. Silent, intermittent, and
|
||||
# misattributed: the exact shape this repo keeps getting burned by.
|
||||
#
|
||||
# DERIVED, NOT LISTED. Both flags come from the ServiceDefinition (`s['name']` and the
|
||||
# route host), so a NEW forward-auth app is isolated the moment it is onboarded. A
|
||||
# per-service table would leave the next app defaulting into the collision, which is how
|
||||
# twelve ServiceMonitors acquired one identical wrong port name in a single edit.
|
||||
# conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct asserts both properties
|
||||
# over the WHOLE tree and asserts THIS function derives rather than lists.
|
||||
#
|
||||
# --cookie-name a unique name per instance, so two sessions cannot alias at all.
|
||||
# This is the fix; the one below is defence in depth.
|
||||
# --cookie-domain HOST-scoped. A per-app reverse proxy has no reason to broadcast its
|
||||
# session cookie to every other host in the zone — a zone-wide cookie
|
||||
# is transmitted to every sibling app on `.${BASE_DOMAIN}`, so any
|
||||
# FUTURE name collision would again be platform-wide instead of
|
||||
# contained. The SHARED forward-auth proxy keeps the wide scope,
|
||||
# because cross-host forward-auth is precisely its job.
|
||||
#
|
||||
# --whitelist-domain deliberately stays ZONE-WIDE: it governs permitted post-login
|
||||
# REDIRECT targets, not cookie scope. Narrowing it here would break the sign-in bounce.
|
||||
f"--cookie-name=_oauth2_proxy_{s['name']}",
|
||||
f"--cookie-domain={host}",
|
||||
f"--whitelist-domain=.{ZONE}",
|
||||
"--cookie-secure=true",
|
||||
"--pass-access-token=true",
|
||||
@@ -2475,6 +2513,51 @@ def _selftest() -> int:
|
||||
not any(a.startswith("--ssl-upstream-insecure-skip-verify") for a in side["args"]),
|
||||
"http upstream must NOT skip TLS verify (that flag is only for crafty's self-signed https)",
|
||||
)
|
||||
# COOKIE ISOLATION. Every oauth2-proxy instance holds its own cookie-secret, so two instances
|
||||
# sharing a cookie NAME on a shared DOMAIN evict each other's sessions silently. Both flags
|
||||
# must be DERIVED from this service, not defaulted and not listed.
|
||||
check(
|
||||
"--cookie-name=_oauth2_proxy_fa" in side["args"],
|
||||
"sidecar must declare a cookie name DERIVED from the service name — the oauth2-proxy "
|
||||
"default (_oauth2_proxy) is shared with every other instance on the zone",
|
||||
)
|
||||
check(
|
||||
"--cookie-domain=fa.home.keithjackson.dev" in side["args"],
|
||||
"sidecar cookie-domain must be HOST-scoped (the app's own host), not zone-wide",
|
||||
)
|
||||
check(
|
||||
f"--whitelist-domain=.{ZONE}" in side["args"],
|
||||
"whitelist-domain must stay ZONE-wide — it governs post-login redirect targets, not "
|
||||
"cookie scope; narrowing it breaks the sign-in bounce",
|
||||
)
|
||||
# Two forward-auth apps must not be able to collide. Renders a SECOND one and compares, so a
|
||||
# future refactor to a constant (or to a per-service table with a copy-paste slip) fails here
|
||||
# rather than silently in a browser six weeks later.
|
||||
fa2_side = next(
|
||||
c
|
||||
for c in next(
|
||||
d[1]
|
||||
for d in render(
|
||||
{
|
||||
"name": "fa2",
|
||||
"source": {"image": "a:1"},
|
||||
"ingressRoutes": [{"host": "fa2.home.keithjackson.dev", "port": 9118}],
|
||||
"auth": {"mode": "forward-auth", "groups": ["media-users"]},
|
||||
}
|
||||
)
|
||||
if d[1]["kind"] == "Deployment"
|
||||
)["spec"]["template"]["spec"]["containers"]
|
||||
if c["name"] == "oauth2-proxy"
|
||||
)
|
||||
|
||||
def _cookie_name(container):
|
||||
return next(a for a in container["args"] if a.startswith("--cookie-name="))
|
||||
|
||||
check(
|
||||
_cookie_name(side) != _cookie_name(fa2_side),
|
||||
"two forward-auth apps rendered IDENTICAL cookie names — they would overwrite each "
|
||||
"other's session on every sign-in (the 2026-09-29 crafty defect, fleet-wide)",
|
||||
)
|
||||
check(
|
||||
side["env"][0]["valueFrom"]["secretKeyRef"]["name"] == "fa-oauth2-proxy",
|
||||
"sidecar creds must come by name from the per-app <app>-oauth2-proxy Secret (R-IDP-04)",
|
||||
|
||||
Reference in new issue
Block a user