platform/crafty/README.md

@@ -49,9 +49,49 @@ Crafty's UI is gated by a **dedicated oauth2-proxy sidecar in the Crafty pod** (
 `crafty.home.keithjackson.dev`). The `crafty` Service targets the sidecar, never Crafty's `:8443`, so
 the gate is **in the data path** (fail-closed: no valid `platform-admins` session ⇒ no upstream).
 
-> **Note:** Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the
-> user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs
-> in with Crafty's local admin credentials (set on first boot). This is expected, not a bug.
+#### Cookie isolation (why the session used to vanish)
+
+This sidecar and the **shared** forward-auth oauth2-proxy are independent instances with **different
+`cookie-secret` values** (measured in-cluster — neither can decrypt the other's cookie). Both used to
+emit oauth2-proxy's *default* cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
+`.home.keithjackson.dev`, so the browser held **one** cookie pair for **both**: whichever proxy
+authenticated last clobbered the other's session, and the loser logged
+`No valid authentication in request`, bounced to Keycloak, and — because the Keycloak SSO session was
+still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents
+exactly as *"I signed in to Keycloak but I'm still not signed in."*
+
+Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app):
+`--cookie-name=_oauth2_proxy_crafty` and a **host-scoped** `--cookie-domain=crafty.home.keithjackson.dev`.
+`--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
+
+#### Crafty's own local login — this is where setup actually blocks
+
+Crafty has **no OIDC**, so after oauth2-proxy admits the user they still meet Crafty's own
+username/password page. That second factor is intended. What was **not** intended — and what made the
+service impossible to set up — is that **nothing documented where that password comes from**.
+
+Crafty generates it **once, on first boot only**, when the users table is empty
+(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`), and writes it to
+**`/crafty/app/config/default-creds.txt`** (mode `0600`, on the `config` subPath of the PVC). It is
+**never** in git and **cannot** be recovered from git: a rebuild onto a fresh PVC generates a *new*
+random password. Submitting anything else yields `POST /api/v2/auth/login/` → **401**, which is exactly
+what a browser sees with no hint as to why.
+
+**→ Retrieval, rotation, and lost-password recovery: [`../../runbooks/crafty-admin-credential.md`](../../runbooks/crafty-admin-credential.md).**
+
+Two traps that runbook covers:
+
+- Crafty's `set_passwd` console command is **unreachable** in Kubernetes — it calls
+  `getpass.getpass()` and so needs a TTY, while the container runs `main.py -d -i` (daemon).
+  Recovery therefore has to go through the DB with Crafty's own argon2 hasher.
+- Crafty's own 3-strikes brute-force cooldown is **inert behind this sidecar**, verified live. Crafty
+  *records* failed attempts under `get_remote_ip()` (which honours `X-Forwarded-For` — the live
+  tracker key is the chain `"192.168.2.132, 10.0.0.123"`) but *checks* for a cooldown under
+  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The two halves key
+  differently, so `get_recent_attempts()` always reads an empty list and `max_login_attempts: 3`
+  never fires. Good news for the operator (**repeated guesses cannot lock you out**), but it means
+  the only thing actually limiting who can reach the login form is the oauth2-proxy
+  `platform-admins` gate — worth knowing before anyone treats that cooldown as a control.
 
 ### Keycloak client
 

platform/crafty/deployment.yaml

@@ -150,7 +150,27 @@ spec:
             # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
             - --skip-provider-button=true
             - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
-            - --cookie-domain=.${BASE_DOMAIN}
+            # ── COOKIE ISOLATION (bug fix 2026-09-29) ────────────────────────────────────────────
+            # This sidecar and the SHARED forward-auth oauth2-proxy (platform/oauth2-proxy) are
+            # INDEPENDENT instances holding DIFFERENT cookie-secrets (verified in-cluster: the two
+            # `cookie-secret` values differ, so neither can decrypt the other's cookie). Both used
+            # to emit oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and `_oauth2_proxy_csrf`)
+            # scoped to `.${BASE_DOMAIN}`, so the browser kept ONE pair of cookies for BOTH proxies:
+            # whichever authenticated last overwrote the other's session, and the loser then logged
+            # "No valid authentication in request" and bounced to Keycloak -- which, thanks to the
+            # live Keycloak SSO session, silently redirected back and overwrote the cookie again.
+            # That mutual eviction presents EXACTLY as the reported symptom: "I signed in to
+            # Keycloak but I am still not signed in to the app."
+            # Two independent fixes, both applied here (only this side changes -- renaming the
+            # SHARED proxy's cookie would sign every other gated app out):
+            #   1. --cookie-name: a unique name, so the two sessions cannot alias at all.
+            #   2. --cookie-domain: HOST-scoped. A per-app reverse proxy has no reason to broadcast
+            #      its session to every other host in the zone; the shared proxy keeps the wide
+            #      scope because cross-host forward-auth is precisely its job.
+            # --whitelist-domain deliberately stays zone-wide: it governs permitted post-login
+            # REDIRECT targets, not cookie scope.
+            - --cookie-name=_oauth2_proxy_crafty
+            - --cookie-domain=crafty.${BASE_DOMAIN}
             - --whitelist-domain=.${BASE_DOMAIN}
             - --cookie-secure=true
             - --pass-access-token=true

runbooks/README.md

@@ -115,6 +115,12 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
   plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
   token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
   Includes why a changed Secret does not restart anything, and why an expired token is harmless.
+- **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
+  Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
+  factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap
+  password exists **only** on the PVC (`default-creds.txt`, generated once on first boot, never in git).
+  Includes the argon2 reset for a lost password, why the 3-strikes lockout cannot fire behind the
+  sidecar, and why deleting `crafty.sqlite` to force a new password destroys every server definition.
 
 ### Virtualisation
 - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.

runbooks/crafty-admin-credential.md

@@ -0,0 +1,179 @@
+# crafty-admin-credential — Crafty's own login rejects you after Keycloak sign-in
+
+**Not alert-driven.** This is an operator procedure, linked from
+[`platform/crafty/README.md`](../platform/crafty/README.md). It covers getting into the Crafty panel the
+first time, rotating that credential, and recovering when it is lost.
+
+> **Everything below prints or sets a LIVE credential.** Do not paste the output into a PR, an issue, a
+> chat, or an AI agent transcript. Read it, use it, rotate it. (Repo rule §6: secrets never enter a
+> transcript.)
+
+## What this means here
+
+`crafty.home.keithjackson.dev` sits behind **two** independent authentications, and only the first is
+Keycloak's:
+
+1. the **oauth2-proxy sidecar** in the Crafty pod — Keycloak `homelab` realm, group `platform-admins`;
+2. **Crafty's own local username/password login** — Crafty 4 has no OIDC and cannot consume the
+   sidecar's identity headers, so it shows its own login page to an already-authenticated user.
+
+So "I signed into Keycloak and I'm still not signed in" is the **expected** shape of this service, and
+the real question is only ever *which* of the two rejected you. They fail differently:
+
+| Where you are | What you see | Which layer |
+|---|---|---|
+| Bounced to Keycloak repeatedly, never reach Crafty | sidecar log: `No valid authentication in request` | oauth2-proxy |
+| Authenticated but `403` | not in `platform-admins` | oauth2-proxy |
+| **Crafty's own login page, credentials refused** | `POST /api/v2/auth/login/` → **401** | **Crafty local** |
+
+The third row is the one that blocks setup, because **nothing in the UI tells you where that password
+comes from**. Crafty generates it **once, on first boot only** — when the users table is empty
+(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`) — and writes it to
+`/crafty/app/config/default-creds.txt` (mode `0600`) on the `config` subPath of the `crafty-data` PVC.
+It is **never in git** and **cannot be recovered from git**: a rebuild onto a fresh PVC generates a new
+random password, and the old one is gone with the old volume.
+
+Two things that are *not* the problem, both verified live:
+
+- **Repeated wrong guesses cannot lock you out.** Crafty *records* failed attempts under
+  `get_remote_ip()` (honours `X-Forwarded-For`) but *checks* the 3-strikes cooldown under
+  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The halves key
+  differently, so `max_login_attempts: 3` never fires. (It also means that throttle is not a control
+  you have — the `platform-admins` gate is what limits who reaches the form at all.)
+- **Crafty's `set_passwd` console command is unreachable here.** It calls `getpass.getpass()` and needs
+  a TTY; the container runs `main.py -d -i` as a daemon. Recovery goes through the DB instead.
+
+## First commands
+
+`kubectl` does not exist on the workstation — the toolchain lives only in the pinned dev container, and
+`dev.sh` picks a current-context **per invocation**, so always name the context.
+
+**1. Is the pod healthy, and is the password still the generated one?**
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+kubectl --context k3s-prod -n crafty get pods -l app.kubernetes.io/name=crafty
+kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
+import sqlite3
+c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
+for r in c.execute(\"SELECT username,created,last_update,last_login FROM users WHERE user_id=1\"):
+    print(\"created    :\",r[1]); print(\"last_update:\",r[2]); print(\"last_login :\",r[3])
+"'
+```
+
+Read `last_update` against `created`: **equal ⇒ the password has never been changed**, so
+`default-creds.txt` is still live. Different ⇒ someone rotated it and that file is stale — skip to
+*Reset*.
+
+**2. Read the bootstrap credential** (prints a live secret — see the warning at the top):
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/default-creds.txt'
+```
+
+The username is `admin`. Sign in at `https://crafty.home.keithjackson.dev` (Keycloak first, then this).
+
+**3. Rotate it immediately**, in the UI: user menu → your profile → change password. That is the only
+supported path that keeps Crafty's own bookkeeping consistent. Then confirm it took — re-run command 1
+and check `last_update` has moved away from `created`.
+
+## Likely causes, most likely first
+
+1. **You never had the password.** By far the most common, and the reason this runbook exists. The
+   credential was only ever on the PVC. → commands 1–3 above.
+2. **The password was rotated and not recorded.** `last_update` ≠ `created` while nobody knows the
+   value. `default-creds.txt` is now misleading, not authoritative. → *Reset* below.
+3. **The PVC was rebuilt.** A fresh volume means a fresh random password *and* an empty panel — no
+   servers, no worlds (`platform/crafty/README.md`, "the servers themselves are NOT in git"). Read
+   `default-creds.txt` from the new pod; do not expect the old credential to work.
+4. **You are actually stuck at the oauth2-proxy layer, not Crafty's.** Check which layer is talking
+   before touching any password:
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n crafty logs deploy/crafty -c oauth2-proxy --tail=200 | grep -v kube-probe'
+   ```
+   `[AuthSuccess] ... groups:[platform-admins]` followed by a `200` on `/login` means oauth2-proxy did
+   its job and the 401 is Crafty's. Repeated `No valid authentication in request` with no `AuthSuccess`
+   means the session cookie is not surviving — see *cookie isolation* in
+   `platform/crafty/README.md`; this sidecar must keep its own `--cookie-name` distinct from the shared
+   proxy's, or the two evict each other.
+5. **`platform-admins` membership lapsed.** A `403` rather than a login page. Fix in Keycloak, not here.
+
+## Reset when the password is unknown
+
+Crafty stores an **argon2** hash in `users.password`. This writes a fresh random password, invalidates
+existing sessions, and drops the new value into a `0600` file **inside the pod** — deliberately
+mirroring Crafty's own `default-creds.txt` idiom so the secret never crosses argv, stdin or shell
+history. (`dev.sh` does **not** forward stdin — a `kubectl exec -i` here hangs forever, which is why the
+password is generated in-pod rather than passed in.)
+
+Two details that will otherwise cost you the afternoon: argon2 is installed **only** in Crafty's venv,
+so it must be `/crafty/.venv/bin/python`, not the system `python3`; and this is a **mutating operator
+action**, not an agent action.
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+ 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- /crafty/.venv/bin/python -c "
+import os, secrets, sqlite3, string, datetime
+from argon2 import PasswordHasher
+pw  = \"\".join(secrets.choice(string.ascii_letters + string.digits) for _ in range(24))
+now = datetime.datetime.now()
+c   = sqlite3.connect(\"/crafty/app/config/db/crafty.sqlite\", timeout=30)
+cur = c.execute(
+    \"UPDATE users SET password=?, last_update=?, valid_tokens_from=? WHERE username=?\",
+    (PasswordHasher().hash(pw), str(now),
+     str(datetime.datetime.now(datetime.timezone.utc)), \"admin\"),
+)
+c.commit()
+print(\"rows updated:\", cur.rowcount)
+out = \"/crafty/app/config/reset-creds.txt\"
+open(out, \"w\").write(pw + chr(10))
+os.chmod(out, 0o600)
+print(\"new password written to\", out)
+"'
+```
+
+Expect `rows updated: 1`. Then read it, sign in, rotate in the UI, and delete the file:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/reset-creds.txt'
+# after signing in and rotating:
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- rm -f /crafty/app/config/reset-creds.txt'
+```
+
+`rows updated: 0` means the username is not `admin` — list the accounts (names only, never the
+`password` column) and retry against the right one:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+ 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
+import sqlite3
+c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
+print([r[0] for r in c.execute(\"SELECT username FROM users\")])
+"'
+```
+
+`database is locked` means Crafty held the write lock; the `timeout=30` usually rides it out, otherwise
+retry. The panel picks the new hash up on the next login attempt — no restart needed, and a restart
+costs a Minecraft-server stop (`strategy: Recreate` on an RWO volume), so do not restart reflexively.
+
+## When to escalate or stop
+
+- **Stop** as soon as you can sign in, and **rotate immediately** — do not leave the generated
+  bootstrap password in place as the standing admin credential, and do not leave `reset-creds.txt` on
+  the PVC.
+- **Do not `kubectl edit`/`patch`/`apply`** the Deployment to work around a login problem. Flux
+  reconciles `main`; hand-applied changes are drift. Manifest changes go through a branch and a PR.
+- **Do not delete `crafty.sqlite`** to force a "fresh install" and a new generated password. It holds
+  every server definition — jar, ports, heap, `auto_start` — and losing it means recreating each server
+  by hand, re-accepting the Mojang EULA, and re-`PATCH`ing `auto_start`. The worlds under
+  `/crafty/servers/<uuid>/` are not in git and not recreatable. The reset above is always cheaper.
+- **Escalate to the Keycloak side** (`runbooks/keycloak-down.md`,
+  `runbooks/keycloak-client-secrets.md`) if there is no `AuthSuccess` in the sidecar log at all — that
+  is an IdP or client-secret problem and no Crafty password will help.
+- **If `default-creds.txt` is absent on a pod whose DB says the password was never changed**, stop and
+  hand off: that combination should not occur, and it suggests the `config` subPath is not the volume
+  you think it is. Record what you measured rather than guessing.

runbooks/kustomization.yaml

@@ -60,6 +60,9 @@ configMapGenerator:
       - cnpg-exporter=cnpg-exporter.md
       - cnpg-primary-spread=cnpg-primary-spread.md
       - cnpg-wal-growth=cnpg-wal-growth.md
+      # Operator procedure, not alert-linked (Crafty's local login is a second factor behind
+      # the Keycloak gate and its bootstrap password lives only on the PVC).
+      - crafty-admin-credential=crafty-admin-credential.md
       - denied-flows=denied-flows.md
       - flux-reconcile=flux-reconcile.md
       - flux-suspended=flux-suspended.md