diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 588a5f3..4b03cf2 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -5109,3 +5109,120 @@ next L2/L3 piece on top of unmerged branches would create a rebase tangle for no cannot land any faster. The next piece is specified and ready to start once these are merged: `docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its boundary landed in #13. + +--- + +## 2026-09-29 — Crafty "won't sign in after Keycloak": the gate was fine, the local login was the wall + +**Branch:** `fix/crafty-oauth2-proxy-signin` → PR to `main`. **Gate:** toolbox `./ci/gate.sh` rc **3** +(UNDETERMINED — see "What was NOT verified"). + +### The report, and what actually refused + +Reported as "crafty utilizing the keycloak oauth2-proxy does not sign in to the service after signing +into keycloak; there is no way to set the service up without credentials." + +The sidecar's own access log settles which layer refused, and it is not the one the report names: + + GET / -> 302 to Keycloak + [AuthSuccess] ... groups:[platform-admins] <- oauth2-proxy did its job + GET /oauth2/callback -> 302 + GET /panel/dashboard -> 302 + GET /login?next=/panel/dashboard -> 200 <- CRAFTY's own login page + POST /api/v2/auth/login/ -> 401 <- CRAFTY refused + +**The Keycloak gate was never broken.** Crafty 4 has no OIDC and cannot consume the sidecar's identity +headers, so it presents its own local login to an already-authenticated user. That second factor is +intended. What was not intended — and what made the service genuinely impossible to set up — is that +**nothing documented where that password comes from.** Crafty generates it **once, on first boot only** +(users table empty → `DatabaseBuilder.default_settings()`) into `/crafty/app/config/default-creds.txt`, +mode `0600`, on the PVC's `config` subPath. Never in git; a fresh PVC generates a new one and the old is +gone with the volume. `platform/crafty/README.md` asserted the double login was "expected, not a bug" +and then left the reader with no way forward — the assertion was true and useless in the same sentence. + +**The existing credential is still usable**, so no DB surgery is needed to get in: `users.last_update` +**equals** `users.created` for `admin` (`09/05/2026, 08:10:04`) — the password has never been changed — +and admin's `last_login` is 09/14 from `127.0.0.1`, the API login that created the "homelab" server. +`default-creds.txt` is therefore live. Retrieval is one command, now in the runbook. + +### A second, real defect found while proving the first + +Both oauth2-proxy instances emitted oauth2-proxy's **default** cookie names (`_oauth2_proxy`, +`_oauth2_proxy_csrf`) scoped to `.home.keithjackson.dev`, while holding **different** cookie-secrets +(compared the two live Secrets — they differ, so neither can decrypt the other's cookie). One cookie +pair, two proxies: whichever authenticated last clobbered the other's session; the loser logged +`No valid authentication in request`, bounced to Keycloak, and — SSO session still live — was +redirected straight back, clobbering it again. **That mutual eviction presents exactly as the reported +symptom**, intermittently, and would have been blamed on Keycloak indefinitely. + +Fixed on the crafty side only — renaming the shared proxy's cookie would sign out every other gated app: +a unique `--cookie-name` plus a **host-scoped** `--cookie-domain` (a per-app reverse proxy has no reason +to broadcast its session zone-wide). `--whitelist-domain` stays zone-wide: it governs post-login +*redirect* targets, not cookie scope. **This one is a latent-bug fix, so it is the part of this change +with no observed failure to point at** — the trace above shows a working session, because that browser +had not visited another gated app first. + +### Traps measured rather than assumed + +- **The 3-strikes cooldown cannot fire behind this sidecar.** Crafty *records* failed attempts under + `get_remote_ip()` (honours `X-Forwarded-For` — the live tracker key is the chain + `"192.168.2.132, 10.0.0.123"`, with 1 attempt: the reported 401) but *checks* the cooldown under + `self.request.remote_ip`, always `127.0.0.1` here. The halves key differently, so + `get_recent_attempts()` always reads empty and `max_login_attempts: 3` never fires. Good (**repeated + guesses cannot lock the operator out**) and bad (that throttle is not a control we have; the + `platform-admins` gate is the only thing limiting who reaches the form). I had written the opposite + into the README from the config value alone before measuring it — the config key existing is not the + same claim as the code path running. +- **`set_passwd` is unreachable in k8s**: it calls `getpass.getpass()` and needs a TTY; the container + runs `main.py -d -i`. Recovery must go through the DB. +- **argon2 is installed only in `/crafty/.venv/bin/python`** — the system `python3` in that container + cannot import it, so the obvious reset one-liner fails. +- **`dev.sh` does not forward stdin.** A `kubectl exec -i` through it hangs forever (provoked, not + assumed). So the reset generates its password **in-pod** into a `0600` file — Crafty's own + `default-creds.txt` idiom — rather than passing a secret through argv, stdin or shell history. + +### Considered and rejected: seeding `default.json` from a SOPS Secret + +Crafty *does* ship the hook — `find_default_password()` reads `app/config/default.json` (`username`, +`password`, min 8 chars) and `delete_default_json` controls its removal — so an initContainer could make +a rebuilt Crafty's admin password deterministic and git-managed, which is the tidy answer to "no way to +set it up without credentials." + +**Rejected.** The image entrypoint pulls its default config **only when `app/config` is empty**: + + if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r ./app/config_original/* ./app/config/ + +Writing `default.json` there pre-boot makes the directory non-empty and **suppresses the entire config +bootstrap** — no `config.json`, no `credits.json`. Doing it safely means replicating the entrypoint's +copy inside the initContainer, i.e. co-owning image-internal bootstrap logic that a 4.11 bump can change +silently, to replace a one-command retrieval. Not worth it. If it is ever revisited, the `ls -A` guard is +the landmine to design around. + +### What was NOT verified + +- **No browser round-trip.** The sign-in fix is **UNVERIFIED end-to-end**. The cookie change is reasoned + from measured facts (default cookie names on both, shared domain, differing secrets) and renders + correctly, but nobody has signed in through it. The human test is in `platform/crafty/README.md`. +- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is still + the pre-fix spec (`crafty-64f7b6d78-pkpgp`, 2/2, 15d, 0 restarts — unchanged by this session's + read-only probes). **Merging is what deploys, and it will restart the pod** — `strategy: Recreate` on + an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick the moment. +- **Gate rc 3, not 0.** Green: `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`. **Skipped:** + `gitea-settings` (no `SETTINGS_API_TOKEN`) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset) — + touched requirements `R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01` are **not certified**. Nothing + failed; nothing runtime was proven either. +- **The admin password was not rotated**, and the bootstrap credential is still the standing admin + credential. The runbook says to rotate on first sign-in; that is an operator action and is still open. + +### Follow-ups + +1. **Sign in and rotate** — `runbooks/crafty-admin-credential.md`, then confirm `last_update` moves off + `created`. Until then the panel's admin password is a value generated on 2026-09-05 that has sat + unchanged and undocumented. +2. **Decide whether the inert brute-force cooldown matters.** Fixing it is upstream (Crafty keys the two + halves differently); the alternative is to accept the `platform-admins` gate as the only throttle and + say so. +3. **The other per-app reverse-proxy sidecars, if any are added, need a unique `--cookie-name` too.** The + collision is a property of the pattern, not of Crafty. Worth an L0 check asserting that every + oauth2-proxy instance in the tree declares a distinct cookie name — the failure is silent and + intermittent, which is exactly the shape this repo keeps getting burned by.