home-network-k3s-impl/decisions/2026-09-29-crafty-oauth2-proxy-signin.md

date, branch, title

date	branch	title
2026-09-29	fix/crafty-oauth2-proxy-signin	Crafty "won't sign in after Keycloak" — the gate was fine, Crafty's own login was the wall

What changed

Reported as "crafty utilizing the keycloak oauth2-proxy does not sign in to the service after signing into keycloak; there is no way to set the service up without credentials." The sidecar's own access log settles which layer refused, and it is not the one the report names:

GET /                            -> 302 to Keycloak
[AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy did its job
GET /oauth2/callback             -> 302
GET /panel/dashboard             -> 302
GET /login?next=/panel/dashboard -> 200           <- CRAFTY's own login page
POST /api/v2/auth/login/         -> 401           <- CRAFTY refused

The Keycloak gate was never broken. Crafty 4 has no OIDC and cannot consume the sidecar's identity headers, so it presents its own local login to an already-authenticated user. That second factor is intended. What was not intended — and what made the service genuinely impossible to set up — is that nothing documented where that password comes from. Crafty generates it once, on first boot only (users table empty → DatabaseBuilder.default_settings()) into /crafty/app/config/default-creds.txt, mode 0600, on the PVC's config subPath. Never in git; a fresh PVC generates a new one and the old is gone with the volume. platform/crafty/README.md asserted the double login was "expected, not a bug" and then left the reader with no way forward — the assertion was true and useless in the same sentence.

So this branch:

• Adds runbooks/crafty-admin-credential.md — tell the two auth layers apart from the sidecar log, retrieve and rotate the bootstrap credential, and reset a lost one via argon2. Registered in the runbooks configMapGenerator/kustomization.yaml and listed in runbooks/README.md, so the URL actually resolves rather than the file merely existing.
• Documents the credential's provenance and the two auth layers in platform/crafty/README.md.
• Fixes a second, real defect found while proving the first. Both oauth2-proxy instances emitted oauth2-proxy's default cookie names (_oauth2_proxy, _oauth2_proxy_csrf) scoped to .home.keithjackson.dev, while holding different cookie-secrets. One cookie pair, two proxies: whichever authenticated last clobbered the other's session; the loser logged No valid authentication in request, bounced to Keycloak, and — SSO session still live — was redirected straight back, clobbering it again. That mutual eviction presents exactly as the reported symptom, intermittently, and would have been blamed on Keycloak indefinitely. Fixed on the crafty side only (renaming the shared proxy's cookie would sign out every other gated app): a unique --cookie-name=_oauth2_proxy_crafty plus a host-scoped --cookie-domain. --whitelist-domain stays zone-wide — it governs post-login redirect targets, not cookie scope.

Considered and rejected: seeding default.json from a SOPS Secret. Crafty does ship the hook — find_default_password() reads app/config/default.json (username, password, min 8 chars) and delete_default_json controls its removal — so an initContainer could make a rebuilt Crafty's admin password deterministic and git-managed. Rejected: the image entrypoint pulls its default config only when app/config is empty (if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r ./app/config_original/* ./app/config/). Writing default.json there pre-boot makes the directory non-empty and suppresses the entire config bootstrap — no config.json, no credits.json. Doing it safely means replicating the entrypoint's copy inside the initContainer, i.e. co-owning image-internal bootstrap logic that a 4.11 bump can change silently, to replace a one-command retrieval. Not worth it. If it is ever revisited, the ls -A guard is the landmine to design around.

What I verified

• Which layer refused, from the sidecar's own access log: [AuthSuccess] ... groups:[platform-admins] followed by Crafty's /login page and POST /api/v2/auth/login/ → 401. oauth2-proxy admitted the user; Crafty rejected the local credential.
• The existing bootstrap credential is still usable, so no DB surgery is needed to get in: users.last_update equals users.created for admin (09/05/2026, 08:10:04) — the password has never been changed — and admin's last_login is 09/14 from 127.0.0.1, the API login that created the "homelab" server. default-creds.txt is therefore live.
• The two cookie-secrets differ — compared the two live Secrets (key presence/values compared in-cluster, not printed), so neither proxy can decrypt the other's cookie; and both instances were emitting the default cookie names on the shared .home.keithjackson.dev scope.
• The 3-strikes cooldown cannot fire behind this sidecar. Crafty records failed attempts under get_remote_ip() (honours X-Forwarded-For — the live tracker key is the chain "192.168.2.132, 10.0.0.123", with 1 attempt: the reported 401) but checks the cooldown under self.request.remote_ip, always 127.0.0.1 here. The halves key differently, so get_recent_attempts() always reads empty and max_login_attempts: 3 never fires. Good (repeated guesses cannot lock the operator out) and bad (that throttle is not a control we have; the platform-admins gate is the only thing limiting who reaches the form). I had written the opposite into the README from the config value alone before measuring it — the config key existing is not the same claim as the code path running.
• set_passwd is unreachable in k8s: it calls getpass.getpass() and needs a TTY; the container runs main.py -d -i. Recovery must go through the DB.
• argon2 is installed only in /crafty/.venv/bin/python — the system python3 in that container cannot import it, so the obvious reset one-liner fails.
• dev.sh does not forward stdin. A kubectl exec -i through it hangs forever (provoked, not assumed). So the runbook's reset generates its password in-pod into a 0600 file — Crafty's own default-creds.txt idiom — rather than passing a secret through argv, stdin or shell history.
• Gate: toolbox ./ci/gate.sh — see the next section for the exit code. Green stages: spec-immutability, lint, l0, render-golden, l1.

What I did NOT verify

• No browser round-trip. The sign-in fix is UNVERIFIED end-to-end. The cookie change is reasoned from measured facts (default cookie names on both, shared domain, differing secrets) and renders correctly, but nobody has signed in through it. The human test is in platform/crafty/README.md.
• Nothing was applied. No kubectl apply/edit/patch, no flux reconcile. The running pod is still the pre-fix spec (crafty-64f7b6d78-pkpgp, 2/2, 15d, 0 restarts — unchanged by this session's read-only probes). Merging is what deploys, and it will restart the pod — strategy: Recreate on an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick the moment.
• Gate rc 3, not 0. Green: spec-immutability, lint, l0, render-golden, l1. Skipped: gitea-settings (no SETTINGS_API_TOKEN) and L2/L3 conformance (CLUSTER_AVAILABLE unset) — touched requirements R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01 are not certified. Nothing failed; nothing runtime was proven either. rc 3 is UNDETERMINED, not green.
• The admin password was not rotated, and the bootstrap credential is still the standing admin credential. The runbook says to rotate on first sign-in; that is an operator action and is still open.
• The cookie collision's symptom was never reproduced in a browser. This is the part of the change with no observed failure to point at — the sidecar trace above shows a working session, because that browser had not visited another gated app first. It is a latent-bug fix.

Follow-ups

1. Sign in and rotate — runbooks/crafty-admin-credential.md, then confirm last_update moves off created. Until then the panel's admin password is a value generated on 2026-09-05 that has sat unchanged and undocumented.
2. Decide whether the inert brute-force cooldown matters. Fixing it is upstream (Crafty keys the two halves differently); the alternative is to accept the platform-admins gate as the only throttle and say so.
3. Any further per-app reverse-proxy sidecar needs a unique --cookie-name too. The collision is a property of the pattern, not of Crafty. Worth an L0 check asserting that every oauth2-proxy instance in the tree declares a distinct cookie name — the failure is silent and intermittent, which is exactly the shape this repo keeps getting burned by.