fix(crafty): the Keycloak gate works -- Crafty's own login was the wall

Reported: "crafty does not sign in after signing into keycloak, and there is
no way to set the service up without credentials."

MEASURED, not inferred. The sidecar log settles which layer refused:

  GET /                        -> 302 to Keycloak
  [AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy is FINE
  GET /oauth2/callback         -> 302
  GET /login?next=/panel/dashboard -> 200           <- Crafty's OWN login page
  POST /api/v2/auth/login/     -> 401               <- Crafty refused

So the Keycloak gate was never broken. Crafty 4 has no OIDC and cannot consume
the sidecar's identity headers, so it shows its own local login to an already
authenticated user. That second factor is intended. What was NOT intended is
that nothing documented where its password comes from, which is exactly why
setup was impossible: Crafty generates it ONCE on first boot (users table
empty -> DatabaseBuilder.default_settings()) into /crafty/app/config/
default-creds.txt, mode 0600, on the PVC's `config` subPath. Never in git;
a fresh PVC generates a new one. The README asserted the double login was
"expected, not a bug" and then left the reader with no way forward.

Verified in-cluster that the existing credential is still usable, so the
service can be set up without touching the DB: users.last_update equals
users.created for `admin` (09/05/2026, 08:10:04), i.e. the password has never
been changed, and admin's last_login is 09/14 from 127.0.0.1 -- the API login
that created the "homelab" server. default-creds.txt is therefore live.

ALSO FIXED -- a real, separate defect found while proving the above, which
produces the reported symptom intermittently and would have been blamed on
Keycloak forever: this sidecar and the SHARED forward-auth oauth2-proxy both
emitted oauth2-proxy's DEFAULT cookie names (_oauth2_proxy,
_oauth2_proxy_csrf) scoped to .home.keithjackson.dev, while holding DIFFERENT
cookie-secrets (compared the two live Secrets: they differ, so neither can
decrypt the other's cookie). One cookie pair, two proxies: whichever
authenticated last clobbered the other's session; the loser logged "No valid
authentication in request", bounced to Keycloak, and -- SSO session still live
-- was redirected straight back, clobbering it again. Fixed on this side only
(renaming the shared proxy's cookie would sign out every other gated app):
a unique --cookie-name plus a host-scoped --cookie-domain, since a per-app
reverse proxy has no reason to broadcast its session zone-wide.
--whitelist-domain stays zone-wide: it governs redirect targets, not cookies.

Two traps measured rather than assumed, both now documented:
  - The 3-strikes cooldown CANNOT fire behind this sidecar. Crafty records
    attempts under get_remote_ip() (honours X-Forwarded-For -- the live
    tracker key is the chain "192.168.2.132, 10.0.0.123") but checks the
    cooldown under request.remote_ip, which is always 127.0.0.1 here. The
    halves key differently, so get_recent_attempts() always reads empty.
    Good (no lockout risk) and bad (that throttle is not a control we have).
  - Crafty's `set_passwd` console command is unreachable in k8s: it calls
    getpass.getpass() and needs a TTY; the container runs `main.py -d -i`.
    Recovery has to go through the DB, and argon2 is installed ONLY in
    /crafty/.venv/bin/python -- the system python3 lacks it.

Deliberately NOT done: an initContainer seeding default.json from a SOPS
Secret to make a rebuilt Crafty's password deterministic. Crafty does support
that hook (find_default_password reads app/config/default.json, min 8 chars),
but the image entrypoint pulls its default config ONLY when app/config is
EMPTY -- so writing a file there pre-boot suppresses the whole config
bootstrap. Buying reproducibility means co-owning image-internal bootstrap
logic that a 4.11 bump could silently change, to replace a one-command
retrieval. Recorded in DECISIONS-LOG as considered and rejected.

New runbook runbooks/crafty-admin-credential.md: tell the two auth layers
apart from the sidecar log, retrieve and rotate the bootstrap credential,
and reset a lost one via argon2 -- generating the password in-pod into a 0600
file rather than passing it in, because dev.sh does not forward stdin (a
`kubectl exec -i` through it hangs; verified). Registered in the configMapGenerator
so the URL actually resolves, not just the file exists.

Gate: toolbox ./ci/gate.sh -- lint/l0/render-golden/l1 green, spec-immutability
green. rc 3 overall: gitea-settings SKIPPED (no SETTINGS_API_TOKEN) and L2/L3
NOT RUN (CLUSTER_AVAILABLE unset). No runtime browser round-trip performed --
the sign-in fix is UNVERIFIED end-to-end and needs the human test in the README.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

platform/crafty/README.md

@@ -49,9 +49,49 @@ Crafty's UI is gated by a **dedicated oauth2-proxy sidecar in the Crafty pod** (
 `crafty.home.keithjackson.dev`). The `crafty` Service targets the sidecar, never Crafty's `:8443`, so
 the gate is **in the data path** (fail-closed: no valid `platform-admins` session ⇒ no upstream).
 
-> **Note:** Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the
-> user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs
-> in with Crafty's local admin credentials (set on first boot). This is expected, not a bug.
+#### Cookie isolation (why the session used to vanish)
+
+This sidecar and the **shared** forward-auth oauth2-proxy are independent instances with **different
+`cookie-secret` values** (measured in-cluster — neither can decrypt the other's cookie). Both used to
+emit oauth2-proxy's *default* cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
+`.home.keithjackson.dev`, so the browser held **one** cookie pair for **both**: whichever proxy
+authenticated last clobbered the other's session, and the loser logged
+`No valid authentication in request`, bounced to Keycloak, and — because the Keycloak SSO session was
+still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents
+exactly as *"I signed in to Keycloak but I'm still not signed in."*
+
+Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app):
+`--cookie-name=_oauth2_proxy_crafty` and a **host-scoped** `--cookie-domain=crafty.home.keithjackson.dev`.
+`--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
+
+#### Crafty's own local login — this is where setup actually blocks
+
+Crafty has **no OIDC**, so after oauth2-proxy admits the user they still meet Crafty's own
+username/password page. That second factor is intended. What was **not** intended — and what made the
+service impossible to set up — is that **nothing documented where that password comes from**.
+
+Crafty generates it **once, on first boot only**, when the users table is empty
+(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`), and writes it to
+**`/crafty/app/config/default-creds.txt`** (mode `0600`, on the `config` subPath of the PVC). It is
+**never** in git and **cannot** be recovered from git: a rebuild onto a fresh PVC generates a *new*
+random password. Submitting anything else yields `POST /api/v2/auth/login/` → **401**, which is exactly
+what a browser sees with no hint as to why.
+
+**→ Retrieval, rotation, and lost-password recovery: [`../../runbooks/crafty-admin-credential.md`](../../runbooks/crafty-admin-credential.md).**
+
+Two traps that runbook covers:
+
+- Crafty's `set_passwd` console command is **unreachable** in Kubernetes — it calls
+  `getpass.getpass()` and so needs a TTY, while the container runs `main.py -d -i` (daemon).
+  Recovery therefore has to go through the DB with Crafty's own argon2 hasher.
+- Crafty's own 3-strikes brute-force cooldown is **inert behind this sidecar**, verified live. Crafty
+  *records* failed attempts under `get_remote_ip()` (which honours `X-Forwarded-For` — the live
+  tracker key is the chain `"192.168.2.132, 10.0.0.123"`) but *checks* for a cooldown under
+  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The two halves key
+  differently, so `get_recent_attempts()` always reads an empty list and `max_login_attempts: 3`
+  never fires. Good news for the operator (**repeated guesses cannot lock you out**), but it means
+  the only thing actually limiting who can reach the login form is the oauth2-proxy
+  `platform-admins` gate — worth knowing before anyone treats that cooldown as a control.
 
 ### Keycloak client
 

platform/crafty/deployment.yaml

@@ -150,7 +150,27 @@ spec:
             # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
             - --skip-provider-button=true
             - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
-            - --cookie-domain=.${BASE_DOMAIN}
+            # ── COOKIE ISOLATION (bug fix 2026-09-29) ────────────────────────────────────────────
+            # This sidecar and the SHARED forward-auth oauth2-proxy (platform/oauth2-proxy) are
+            # INDEPENDENT instances holding DIFFERENT cookie-secrets (verified in-cluster: the two
+            # `cookie-secret` values differ, so neither can decrypt the other's cookie). Both used
+            # to emit oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and `_oauth2_proxy_csrf`)
+            # scoped to `.${BASE_DOMAIN}`, so the browser kept ONE pair of cookies for BOTH proxies:
+            # whichever authenticated last overwrote the other's session, and the loser then logged
+            # "No valid authentication in request" and bounced to Keycloak -- which, thanks to the
+            # live Keycloak SSO session, silently redirected back and overwrote the cookie again.
+            # That mutual eviction presents EXACTLY as the reported symptom: "I signed in to
+            # Keycloak but I am still not signed in to the app."
+            # Two independent fixes, both applied here (only this side changes -- renaming the
+            # SHARED proxy's cookie would sign every other gated app out):
+            #   1. --cookie-name: a unique name, so the two sessions cannot alias at all.
+            #   2. --cookie-domain: HOST-scoped. A per-app reverse proxy has no reason to broadcast
+            #      its session to every other host in the zone; the shared proxy keeps the wide
+            #      scope because cross-host forward-auth is precisely its job.
+            # --whitelist-domain deliberately stays zone-wide: it governs permitted post-login
+            # REDIRECT targets, not cookie scope.
+            - --cookie-name=_oauth2_proxy_crafty
+            - --cookie-domain=crafty.${BASE_DOMAIN}
             - --whitelist-domain=.${BASE_DOMAIN}
             - --cookie-secure=true
             - --pass-access-token=true

runbooks/README.md

@@ -115,6 +115,12 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
   plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
   token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
   Includes why a changed Secret does not restart anything, and why an expired token is harmless.
+- **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
+  Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
+  factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap
+  password exists **only** on the PVC (`default-creds.txt`, generated once on first boot, never in git).
+  Includes the argon2 reset for a lost password, why the 3-strikes lockout cannot fire behind the
+  sidecar, and why deleting `crafty.sqlite` to force a new password destroys every server definition.
 
 ### Virtualisation
 - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.

runbooks/crafty-admin-credential.md

@@ -0,0 +1,179 @@
+# crafty-admin-credential — Crafty's own login rejects you after Keycloak sign-in
+
+**Not alert-driven.** This is an operator procedure, linked from
+[`platform/crafty/README.md`](../platform/crafty/README.md). It covers getting into the Crafty panel the
+first time, rotating that credential, and recovering when it is lost.
+
+> **Everything below prints or sets a LIVE credential.** Do not paste the output into a PR, an issue, a
+> chat, or an AI agent transcript. Read it, use it, rotate it. (Repo rule §6: secrets never enter a
+> transcript.)
+
+## What this means here
+
+`crafty.home.keithjackson.dev` sits behind **two** independent authentications, and only the first is
+Keycloak's:
+
+1. the **oauth2-proxy sidecar** in the Crafty pod — Keycloak `homelab` realm, group `platform-admins`;
+2. **Crafty's own local username/password login** — Crafty 4 has no OIDC and cannot consume the
+   sidecar's identity headers, so it shows its own login page to an already-authenticated user.
+
+So "I signed into Keycloak and I'm still not signed in" is the **expected** shape of this service, and
+the real question is only ever *which* of the two rejected you. They fail differently:
+
+| Where you are | What you see | Which layer |
+|---|---|---|
+| Bounced to Keycloak repeatedly, never reach Crafty | sidecar log: `No valid authentication in request` | oauth2-proxy |
+| Authenticated but `403` | not in `platform-admins` | oauth2-proxy |
+| **Crafty's own login page, credentials refused** | `POST /api/v2/auth/login/` → **401** | **Crafty local** |
+
+The third row is the one that blocks setup, because **nothing in the UI tells you where that password
+comes from**. Crafty generates it **once, on first boot only** — when the users table is empty
+(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`) — and writes it to
+`/crafty/app/config/default-creds.txt` (mode `0600`) on the `config` subPath of the `crafty-data` PVC.
+It is **never in git** and **cannot be recovered from git**: a rebuild onto a fresh PVC generates a new
+random password, and the old one is gone with the old volume.
+
+Two things that are *not* the problem, both verified live:
+
+- **Repeated wrong guesses cannot lock you out.** Crafty *records* failed attempts under
+  `get_remote_ip()` (honours `X-Forwarded-For`) but *checks* the 3-strikes cooldown under
+  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The halves key
+  differently, so `max_login_attempts: 3` never fires. (It also means that throttle is not a control
+  you have — the `platform-admins` gate is what limits who reaches the form at all.)
+- **Crafty's `set_passwd` console command is unreachable here.** It calls `getpass.getpass()` and needs
+  a TTY; the container runs `main.py -d -i` as a daemon. Recovery goes through the DB instead.
+
+## First commands
+
+`kubectl` does not exist on the workstation — the toolchain lives only in the pinned dev container, and
+`dev.sh` picks a current-context **per invocation**, so always name the context.
+
+**1. Is the pod healthy, and is the password still the generated one?**
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+kubectl --context k3s-prod -n crafty get pods -l app.kubernetes.io/name=crafty
+kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
+import sqlite3
+c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
+for r in c.execute(\"SELECT username,created,last_update,last_login FROM users WHERE user_id=1\"):
+    print(\"created    :\",r[1]); print(\"last_update:\",r[2]); print(\"last_login :\",r[3])
+"'
+```
+
+Read `last_update` against `created`: **equal ⇒ the password has never been changed**, so
+`default-creds.txt` is still live. Different ⇒ someone rotated it and that file is stale — skip to
+*Reset*.
+
+**2. Read the bootstrap credential** (prints a live secret — see the warning at the top):
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/default-creds.txt'
+```
+
+The username is `admin`. Sign in at `https://crafty.home.keithjackson.dev` (Keycloak first, then this).
+
+**3. Rotate it immediately**, in the UI: user menu → your profile → change password. That is the only
+supported path that keeps Crafty's own bookkeeping consistent. Then confirm it took — re-run command 1
+and check `last_update` has moved away from `created`.
+
+## Likely causes, most likely first
+
+1. **You never had the password.** By far the most common, and the reason this runbook exists. The
+   credential was only ever on the PVC. → commands 1–3 above.
+2. **The password was rotated and not recorded.** `last_update` ≠ `created` while nobody knows the
+   value. `default-creds.txt` is now misleading, not authoritative. → *Reset* below.
+3. **The PVC was rebuilt.** A fresh volume means a fresh random password *and* an empty panel — no
+   servers, no worlds (`platform/crafty/README.md`, "the servers themselves are NOT in git"). Read
+   `default-creds.txt` from the new pod; do not expect the old credential to work.
+4. **You are actually stuck at the oauth2-proxy layer, not Crafty's.** Check which layer is talking
+   before touching any password:
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n crafty logs deploy/crafty -c oauth2-proxy --tail=200 | grep -v kube-probe'
+   ```
+   `[AuthSuccess] ... groups:[platform-admins]` followed by a `200` on `/login` means oauth2-proxy did
+   its job and the 401 is Crafty's. Repeated `No valid authentication in request` with no `AuthSuccess`
+   means the session cookie is not surviving — see *cookie isolation* in
+   `platform/crafty/README.md`; this sidecar must keep its own `--cookie-name` distinct from the shared
+   proxy's, or the two evict each other.
+5. **`platform-admins` membership lapsed.** A `403` rather than a login page. Fix in Keycloak, not here.
+
+## Reset when the password is unknown
+
+Crafty stores an **argon2** hash in `users.password`. This writes a fresh random password, invalidates
+existing sessions, and drops the new value into a `0600` file **inside the pod** — deliberately
+mirroring Crafty's own `default-creds.txt` idiom so the secret never crosses argv, stdin or shell
+history. (`dev.sh` does **not** forward stdin — a `kubectl exec -i` here hangs forever, which is why the
+password is generated in-pod rather than passed in.)
+
+Two details that will otherwise cost you the afternoon: argon2 is installed **only** in Crafty's venv,
+so it must be `/crafty/.venv/bin/python`, not the system `python3`; and this is a **mutating operator
+action**, not an agent action.
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+ 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- /crafty/.venv/bin/python -c "
+import os, secrets, sqlite3, string, datetime
+from argon2 import PasswordHasher
+pw  = \"\".join(secrets.choice(string.ascii_letters + string.digits) for _ in range(24))
+now = datetime.datetime.now()
+c   = sqlite3.connect(\"/crafty/app/config/db/crafty.sqlite\", timeout=30)
+cur = c.execute(
+    \"UPDATE users SET password=?, last_update=?, valid_tokens_from=? WHERE username=?\",
+    (PasswordHasher().hash(pw), str(now),
+     str(datetime.datetime.now(datetime.timezone.utc)), \"admin\"),
+)
+c.commit()
+print(\"rows updated:\", cur.rowcount)
+out = \"/crafty/app/config/reset-creds.txt\"
+open(out, \"w\").write(pw + chr(10))
+os.chmod(out, 0o600)
+print(\"new password written to\", out)
+"'
+```
+
+Expect `rows updated: 1`. Then read it, sign in, rotate in the UI, and delete the file:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/reset-creds.txt'
+# after signing in and rotating:
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- rm -f /crafty/app/config/reset-creds.txt'
+```
+
+`rows updated: 0` means the username is not `admin` — list the accounts (names only, never the
+`password` column) and retry against the right one:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+ 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
+import sqlite3
+c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
+print([r[0] for r in c.execute(\"SELECT username FROM users\")])
+"'
+```
+
+`database is locked` means Crafty held the write lock; the `timeout=30` usually rides it out, otherwise
+retry. The panel picks the new hash up on the next login attempt — no restart needed, and a restart
+costs a Minecraft-server stop (`strategy: Recreate` on an RWO volume), so do not restart reflexively.
+
+## When to escalate or stop
+
+- **Stop** as soon as you can sign in, and **rotate immediately** — do not leave the generated
+  bootstrap password in place as the standing admin credential, and do not leave `reset-creds.txt` on
+  the PVC.
+- **Do not `kubectl edit`/`patch`/`apply`** the Deployment to work around a login problem. Flux
+  reconciles `main`; hand-applied changes are drift. Manifest changes go through a branch and a PR.
+- **Do not delete `crafty.sqlite`** to force a "fresh install" and a new generated password. It holds
+  every server definition — jar, ports, heap, `auto_start` — and losing it means recreating each server
+  by hand, re-accepting the Mojang EULA, and re-`PATCH`ing `auto_start`. The worlds under
+  `/crafty/servers/<uuid>/` are not in git and not recreatable. The reset above is always cheaper.
+- **Escalate to the Keycloak side** (`runbooks/keycloak-down.md`,
+  `runbooks/keycloak-client-secrets.md`) if there is no `AuthSuccess` in the sidecar log at all — that
+  is an IdP or client-secret problem and no Crafty password will help.
+- **If `default-creds.txt` is absent on a pod whose DB says the password was never changed**, stop and
+  hand off: that combination should not occur, and it suggests the `config` subPath is not the volume
+  you think it is. Record what you measured rather than guessing.

runbooks/kustomization.yaml

@@ -60,6 +60,9 @@ configMapGenerator:
       - cnpg-exporter=cnpg-exporter.md
       - cnpg-primary-spread=cnpg-primary-spread.md
       - cnpg-wal-growth=cnpg-wal-growth.md
+      # Operator procedure, not alert-linked (Crafty's local login is a second factor behind
+      # the Keycloak gate and its bootstrap password lives only on the PVC).
+      - crafty-admin-credential=crafty-admin-credential.md
       - denied-flows=denied-flows.md
       - flux-reconcile=flux-reconcile.md
       - flux-suspended=flux-suspended.md