From dd194bbc9668dee63a5ff509a38fd21b58243760 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 29 Sep 2026 22:44:28 -0500 Subject: [PATCH] =?UTF-8?q?docs(decisions):=20Crafty=20sign-in=20handoff?= =?UTF-8?q?=20=E2=80=94=20the=20gate=20was=20fine,=20the=20local=20login?= =?UTF-8?q?=20was=20the=20wall?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records what refused (Crafty's own 401, not oauth2-proxy), that the existing bootstrap credential is still live (users.last_update == users.created), the separate cookie-name collision fixed alongside, the traps measured rather than assumed, the default.json seeding approach considered and rejected with the entrypoint `ls -A` landmine that kills it, and — plainly — that no browser round-trip was performed and the gate was rc 3, not green. Written as its OWN FILE under decisions/ rather than appended to the tail of DECISIONS-LOG.md. The single-file append is what made this PR (#22) and three others unmergeable on 2026-09-29: every branch collided on the same last hunk, with DECISIONS-LOG.md the only conflicting path in all four. The one-file-per- entry layout and its ci/decisions-log.sh guard land in #24; this entry is already written in that format, and validates against that checker (rc 0, "1 entry well-formed"). This branch now touches DECISIONS-LOG.md not at all. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-09-29-crafty-oauth2-proxy-signin.md | 123 ++++++++++++++++++ 1 file changed, 123 insertions(+) create mode 100644 decisions/2026-09-29-crafty-oauth2-proxy-signin.md diff --git a/decisions/2026-09-29-crafty-oauth2-proxy-signin.md b/decisions/2026-09-29-crafty-oauth2-proxy-signin.md new file mode 100644 index 0000000..2b2cff8 --- /dev/null +++ b/decisions/2026-09-29-crafty-oauth2-proxy-signin.md @@ -0,0 +1,123 @@ +--- +date: 2026-09-29 +branch: fix/crafty-oauth2-proxy-signin +title: Crafty "won't sign in after Keycloak" — the gate was fine, Crafty's own login was the wall +--- + +## What changed + +Reported as *"crafty utilizing the keycloak oauth2-proxy does not sign in to the service after +signing into keycloak; there is no way to set the service up without credentials."* The sidecar's +own access log settles which layer refused, and it is not the one the report names: + + GET / -> 302 to Keycloak + [AuthSuccess] ... groups:[platform-admins] <- oauth2-proxy did its job + GET /oauth2/callback -> 302 + GET /panel/dashboard -> 302 + GET /login?next=/panel/dashboard -> 200 <- CRAFTY's own login page + POST /api/v2/auth/login/ -> 401 <- CRAFTY refused + +**The Keycloak gate was never broken.** Crafty 4 has no OIDC and cannot consume the sidecar's +identity headers, so it presents its own local login to an already-authenticated user. That second +factor is intended. What was not intended — and what made the service genuinely impossible to set +up — is that **nothing documented where that password comes from**. Crafty generates it **once, on +first boot only** (users table empty → `DatabaseBuilder.default_settings()`) into +`/crafty/app/config/default-creds.txt`, mode `0600`, on the PVC's `config` subPath. Never in git; a +fresh PVC generates a new one and the old is gone with the volume. `platform/crafty/README.md` +asserted the double login was "expected, not a bug" and then left the reader with no way forward — +the assertion was true and useless in the same sentence. + +So this branch: + +- Adds `runbooks/crafty-admin-credential.md` — tell the two auth layers apart from the sidecar log, + retrieve and rotate the bootstrap credential, and reset a lost one via argon2. Registered in the + runbooks `configMapGenerator`/`kustomization.yaml` and listed in `runbooks/README.md`, so the URL + actually resolves rather than the file merely existing. +- Documents the credential's provenance and the two auth layers in `platform/crafty/README.md`. +- **Fixes a second, real defect found while proving the first.** Both oauth2-proxy instances emitted + oauth2-proxy's **default** cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to + `.home.keithjackson.dev`, while holding **different** cookie-secrets. One cookie pair, two + proxies: whichever authenticated last clobbered the other's session; the loser logged `No valid + authentication in request`, bounced to Keycloak, and — SSO session still live — was redirected + straight back, clobbering it again. **That mutual eviction presents exactly as the reported + symptom**, intermittently, and would have been blamed on Keycloak indefinitely. Fixed on the + crafty side only (renaming the shared proxy's cookie would sign out every other gated app): a + unique `--cookie-name=_oauth2_proxy_crafty` plus a **host-scoped** `--cookie-domain`. + `--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope. + +**Considered and rejected: seeding `default.json` from a SOPS Secret.** Crafty does ship the hook — +`find_default_password()` reads `app/config/default.json` (`username`, `password`, min 8 chars) and +`delete_default_json` controls its removal — so an initContainer could make a rebuilt Crafty's admin +password deterministic and git-managed. Rejected: the image entrypoint pulls its default config +**only when `app/config` is empty** (`if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r +./app/config_original/* ./app/config/`). Writing `default.json` there pre-boot makes the directory +non-empty and **suppresses the entire config bootstrap** — no `config.json`, no `credits.json`. +Doing it safely means replicating the entrypoint's copy inside the initContainer, i.e. co-owning +image-internal bootstrap logic that a 4.11 bump can change silently, to replace a one-command +retrieval. Not worth it. If it is ever revisited, the `ls -A` guard is the landmine to design around. + +## What I verified + +- **Which layer refused**, from the sidecar's own access log: `[AuthSuccess] ... + groups:[platform-admins]` followed by Crafty's `/login` page and `POST /api/v2/auth/login/ → 401`. + oauth2-proxy admitted the user; Crafty rejected the local credential. +- **The existing bootstrap credential is still usable**, so no DB surgery is needed to get in: + `users.last_update` **equals** `users.created` for `admin` (`09/05/2026, 08:10:04`) — the password + has never been changed — and admin's `last_login` is 09/14 from `127.0.0.1`, the API login that + created the "homelab" server. `default-creds.txt` is therefore live. +- **The two cookie-secrets differ** — compared the two live Secrets (key presence/values compared + in-cluster, not printed), so neither proxy can decrypt the other's cookie; and both instances were + emitting the default cookie names on the shared `.home.keithjackson.dev` scope. +- **The 3-strikes cooldown cannot fire behind this sidecar.** Crafty *records* failed attempts under + `get_remote_ip()` (honours `X-Forwarded-For` — the live tracker key is the chain + `"192.168.2.132, 10.0.0.123"`, with 1 attempt: the reported 401) but *checks* the cooldown under + `self.request.remote_ip`, always `127.0.0.1` here. The halves key differently, so + `get_recent_attempts()` always reads empty and `max_login_attempts: 3` never fires. Good + (**repeated guesses cannot lock the operator out**) and bad (that throttle is not a control we + have; the `platform-admins` gate is the only thing limiting who reaches the form). I had written + the opposite into the README from the config value alone before measuring it — the config key + existing is not the same claim as the code path running. +- **`set_passwd` is unreachable in k8s**: it calls `getpass.getpass()` and needs a TTY; the container + runs `main.py -d -i`. Recovery must go through the DB. +- **argon2 is installed only in `/crafty/.venv/bin/python`** — the system `python3` in that container + cannot import it, so the obvious reset one-liner fails. +- **`dev.sh` does not forward stdin.** A `kubectl exec -i` through it hangs forever (provoked, not + assumed). So the runbook's reset generates its password **in-pod** into a `0600` file — Crafty's own + `default-creds.txt` idiom — rather than passing a secret through argv, stdin or shell history. +- **Gate:** toolbox `./ci/gate.sh` — see the next section for the exit code. Green stages: + `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`. + +## What I did NOT verify + +- **No browser round-trip.** The sign-in fix is **UNVERIFIED end-to-end**. The cookie change is + reasoned from measured facts (default cookie names on both, shared domain, differing secrets) and + renders correctly, but nobody has signed in through it. The human test is in + `platform/crafty/README.md`. +- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is + still the pre-fix spec (`crafty-64f7b6d78-pkpgp`, 2/2, 15d, 0 restarts — unchanged by this + session's read-only probes). **Merging is what deploys, and it will restart the pod** — + `strategy: Recreate` on an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick + the moment. +- **Gate rc 3, not 0.** Green: `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`. **Skipped:** + `gitea-settings` (no `SETTINGS_API_TOKEN`) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset) — + touched requirements `R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01` are **not certified**. + Nothing failed; nothing runtime was proven either. **rc 3 is UNDETERMINED, not green.** +- **The admin password was not rotated**, and the bootstrap credential is still the standing admin + credential. The runbook says to rotate on first sign-in; that is an operator action and is still + open. +- **The cookie collision's symptom was never reproduced in a browser.** This is the part of the + change with no observed failure to point at — the sidecar trace above shows a *working* session, + because that browser had not visited another gated app first. It is a latent-bug fix. + +### Follow-ups + +1. **Sign in and rotate** — `runbooks/crafty-admin-credential.md`, then confirm `last_update` moves + off `created`. Until then the panel's admin password is a value generated on 2026-09-05 that has + sat unchanged and undocumented. +2. **Decide whether the inert brute-force cooldown matters.** Fixing it is upstream (Crafty keys the + two halves differently); the alternative is to accept the `platform-admins` gate as the only + throttle and say so. +3. **Any further per-app reverse-proxy sidecar needs a unique `--cookie-name` too.** The collision is + a property of the pattern, not of Crafty. Worth an L0 check asserting that every oauth2-proxy + instance in the tree declares a distinct cookie name — the failure is silent and intermittent, + which is exactly the shape this repo keeps getting burned by.