diff --git a/platform/crafty/README.md b/platform/crafty/README.md index 37b07f7..c110fed 100644 --- a/platform/crafty/README.md +++ b/platform/crafty/README.md @@ -49,9 +49,49 @@ Crafty's UI is gated by a **dedicated oauth2-proxy sidecar in the Crafty pod** ( `crafty.home.keithjackson.dev`). The `crafty` Service targets the sidecar, never Crafty's `:8443`, so the gate is **in the data path** (fail-closed: no valid `platform-admins` session ⇒ no upstream). -> **Note:** Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the -> user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs -> in with Crafty's local admin credentials (set on first boot). This is expected, not a bug. +#### Cookie isolation (why the session used to vanish) + +This sidecar and the **shared** forward-auth oauth2-proxy are independent instances with **different +`cookie-secret` values** (measured in-cluster — neither can decrypt the other's cookie). Both used to +emit oauth2-proxy's *default* cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to +`.home.keithjackson.dev`, so the browser held **one** cookie pair for **both**: whichever proxy +authenticated last clobbered the other's session, and the loser logged +`No valid authentication in request`, bounced to Keycloak, and — because the Keycloak SSO session was +still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents +exactly as *"I signed in to Keycloak but I'm still not signed in."* + +Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app): +`--cookie-name=_oauth2_proxy_crafty` and a **host-scoped** `--cookie-domain=crafty.home.keithjackson.dev`. +`--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope. + +#### Crafty's own local login — this is where setup actually blocks + +Crafty has **no OIDC**, so after oauth2-proxy admits the user they still meet Crafty's own +username/password page. That second factor is intended. What was **not** intended — and what made the +service impossible to set up — is that **nothing documented where that password comes from**. + +Crafty generates it **once, on first boot only**, when the users table is empty +(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`), and writes it to +**`/crafty/app/config/default-creds.txt`** (mode `0600`, on the `config` subPath of the PVC). It is +**never** in git and **cannot** be recovered from git: a rebuild onto a fresh PVC generates a *new* +random password. Submitting anything else yields `POST /api/v2/auth/login/` → **401**, which is exactly +what a browser sees with no hint as to why. + +**→ Retrieval, rotation, and lost-password recovery: [`../../runbooks/crafty-admin-credential.md`](../../runbooks/crafty-admin-credential.md).** + +Two traps that runbook covers: + +- Crafty's `set_passwd` console command is **unreachable** in Kubernetes — it calls + `getpass.getpass()` and so needs a TTY, while the container runs `main.py -d -i` (daemon). + Recovery therefore has to go through the DB with Crafty's own argon2 hasher. +- Crafty's own 3-strikes brute-force cooldown is **inert behind this sidecar**, verified live. Crafty + *records* failed attempts under `get_remote_ip()` (which honours `X-Forwarded-For` — the live + tracker key is the chain `"192.168.2.132, 10.0.0.123"`) but *checks* for a cooldown under + `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The two halves key + differently, so `get_recent_attempts()` always reads an empty list and `max_login_attempts: 3` + never fires. Good news for the operator (**repeated guesses cannot lock you out**), but it means + the only thing actually limiting who can reach the login form is the oauth2-proxy + `platform-admins` gate — worth knowing before anyone treats that cooldown as a control. ### Keycloak client diff --git a/platform/crafty/deployment.yaml b/platform/crafty/deployment.yaml index 4fdabe9..adcaa1d 100644 --- a/platform/crafty/deployment.yaml +++ b/platform/crafty/deployment.yaml @@ -150,7 +150,27 @@ spec: # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button. - --skip-provider-button=true - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback - - --cookie-domain=.${BASE_DOMAIN} + # ── COOKIE ISOLATION (bug fix 2026-09-29) ──────────────────────────────────────────── + # This sidecar and the SHARED forward-auth oauth2-proxy (platform/oauth2-proxy) are + # INDEPENDENT instances holding DIFFERENT cookie-secrets (verified in-cluster: the two + # `cookie-secret` values differ, so neither can decrypt the other's cookie). Both used + # to emit oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and `_oauth2_proxy_csrf`) + # scoped to `.${BASE_DOMAIN}`, so the browser kept ONE pair of cookies for BOTH proxies: + # whichever authenticated last overwrote the other's session, and the loser then logged + # "No valid authentication in request" and bounced to Keycloak -- which, thanks to the + # live Keycloak SSO session, silently redirected back and overwrote the cookie again. + # That mutual eviction presents EXACTLY as the reported symptom: "I signed in to + # Keycloak but I am still not signed in to the app." + # Two independent fixes, both applied here (only this side changes -- renaming the + # SHARED proxy's cookie would sign every other gated app out): + # 1. --cookie-name: a unique name, so the two sessions cannot alias at all. + # 2. --cookie-domain: HOST-scoped. A per-app reverse proxy has no reason to broadcast + # its session to every other host in the zone; the shared proxy keeps the wide + # scope because cross-host forward-auth is precisely its job. + # --whitelist-domain deliberately stays zone-wide: it governs permitted post-login + # REDIRECT targets, not cookie scope. + - --cookie-name=_oauth2_proxy_crafty + - --cookie-domain=crafty.${BASE_DOMAIN} - --whitelist-domain=.${BASE_DOMAIN} - --cookie-secure=true - --pass-access-token=true diff --git a/runbooks/README.md b/runbooks/README.md index c393ee8..9c859fd 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -115,6 +115,12 @@ Each affected runbook says so on its own page and gives the by-hand equivalent. plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`: token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured). Includes why a changed Secret does not restart anything, and why an expired token is harmless. +- **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password. + Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second + factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap + password exists **only** on the PVC (`default-creds.txt`, generated once on first boot, never in git). + Includes the argon2 reset for a lost password, why the 3-strikes lockout cannot fire behind the + sidecar, and why deleting `crafty.sqlite` to force a new password destroys every server definition. ### Virtualisation - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running. diff --git a/runbooks/crafty-admin-credential.md b/runbooks/crafty-admin-credential.md new file mode 100644 index 0000000..b6b8010 --- /dev/null +++ b/runbooks/crafty-admin-credential.md @@ -0,0 +1,179 @@ +# crafty-admin-credential — Crafty's own login rejects you after Keycloak sign-in + +**Not alert-driven.** This is an operator procedure, linked from +[`platform/crafty/README.md`](../platform/crafty/README.md). It covers getting into the Crafty panel the +first time, rotating that credential, and recovering when it is lost. + +> **Everything below prints or sets a LIVE credential.** Do not paste the output into a PR, an issue, a +> chat, or an AI agent transcript. Read it, use it, rotate it. (Repo rule §6: secrets never enter a +> transcript.) + +## What this means here + +`crafty.home.keithjackson.dev` sits behind **two** independent authentications, and only the first is +Keycloak's: + +1. the **oauth2-proxy sidecar** in the Crafty pod — Keycloak `homelab` realm, group `platform-admins`; +2. **Crafty's own local username/password login** — Crafty 4 has no OIDC and cannot consume the + sidecar's identity headers, so it shows its own login page to an already-authenticated user. + +So "I signed into Keycloak and I'm still not signed in" is the **expected** shape of this service, and +the real question is only ever *which* of the two rejected you. They fail differently: + +| Where you are | What you see | Which layer | +|---|---|---| +| Bounced to Keycloak repeatedly, never reach Crafty | sidecar log: `No valid authentication in request` | oauth2-proxy | +| Authenticated but `403` | not in `platform-admins` | oauth2-proxy | +| **Crafty's own login page, credentials refused** | `POST /api/v2/auth/login/` → **401** | **Crafty local** | + +The third row is the one that blocks setup, because **nothing in the UI tells you where that password +comes from**. Crafty generates it **once, on first boot only** — when the users table is empty +(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`) — and writes it to +`/crafty/app/config/default-creds.txt` (mode `0600`) on the `config` subPath of the `crafty-data` PVC. +It is **never in git** and **cannot be recovered from git**: a rebuild onto a fresh PVC generates a new +random password, and the old one is gone with the old volume. + +Two things that are *not* the problem, both verified live: + +- **Repeated wrong guesses cannot lock you out.** Crafty *records* failed attempts under + `get_remote_ip()` (honours `X-Forwarded-For`) but *checks* the 3-strikes cooldown under + `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The halves key + differently, so `max_login_attempts: 3` never fires. (It also means that throttle is not a control + you have — the `platform-admins` gate is what limits who reaches the form at all.) +- **Crafty's `set_passwd` console command is unreachable here.** It calls `getpass.getpass()` and needs + a TTY; the container runs `main.py -d -i` as a daemon. Recovery goes through the DB instead. + +## First commands + +`kubectl` does not exist on the workstation — the toolchain lives only in the pinned dev container, and +`dev.sh` picks a current-context **per invocation**, so always name the context. + +**1. Is the pod healthy, and is the password still the generated one?** + +```bash +./infra/devcontainer/dev.sh run bash -c ' +kubectl --context k3s-prod -n crafty get pods -l app.kubernetes.io/name=crafty +kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c " +import sqlite3 +c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True) +for r in c.execute(\"SELECT username,created,last_update,last_login FROM users WHERE user_id=1\"): + print(\"created :\",r[1]); print(\"last_update:\",r[2]); print(\"last_login :\",r[3]) +"' +``` + +Read `last_update` against `created`: **equal ⇒ the password has never been changed**, so +`default-creds.txt` is still live. Different ⇒ someone rotated it and that file is stale — skip to +*Reset*. + +**2. Read the bootstrap credential** (prints a live secret — see the warning at the top): + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/default-creds.txt' +``` + +The username is `admin`. Sign in at `https://crafty.home.keithjackson.dev` (Keycloak first, then this). + +**3. Rotate it immediately**, in the UI: user menu → your profile → change password. That is the only +supported path that keeps Crafty's own bookkeeping consistent. Then confirm it took — re-run command 1 +and check `last_update` has moved away from `created`. + +## Likely causes, most likely first + +1. **You never had the password.** By far the most common, and the reason this runbook exists. The + credential was only ever on the PVC. → commands 1–3 above. +2. **The password was rotated and not recorded.** `last_update` ≠ `created` while nobody knows the + value. `default-creds.txt` is now misleading, not authoritative. → *Reset* below. +3. **The PVC was rebuilt.** A fresh volume means a fresh random password *and* an empty panel — no + servers, no worlds (`platform/crafty/README.md`, "the servers themselves are NOT in git"). Read + `default-creds.txt` from the new pod; do not expect the old credential to work. +4. **You are actually stuck at the oauth2-proxy layer, not Crafty's.** Check which layer is talking + before touching any password: + ```bash + ./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n crafty logs deploy/crafty -c oauth2-proxy --tail=200 | grep -v kube-probe' + ``` + `[AuthSuccess] ... groups:[platform-admins]` followed by a `200` on `/login` means oauth2-proxy did + its job and the 401 is Crafty's. Repeated `No valid authentication in request` with no `AuthSuccess` + means the session cookie is not surviving — see *cookie isolation* in + `platform/crafty/README.md`; this sidecar must keep its own `--cookie-name` distinct from the shared + proxy's, or the two evict each other. +5. **`platform-admins` membership lapsed.** A `403` rather than a login page. Fix in Keycloak, not here. + +## Reset when the password is unknown + +Crafty stores an **argon2** hash in `users.password`. This writes a fresh random password, invalidates +existing sessions, and drops the new value into a `0600` file **inside the pod** — deliberately +mirroring Crafty's own `default-creds.txt` idiom so the secret never crosses argv, stdin or shell +history. (`dev.sh` does **not** forward stdin — a `kubectl exec -i` here hangs forever, which is why the +password is generated in-pod rather than passed in.) + +Two details that will otherwise cost you the afternoon: argon2 is installed **only** in Crafty's venv, +so it must be `/crafty/.venv/bin/python`, not the system `python3`; and this is a **mutating operator +action**, not an agent action. + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- /crafty/.venv/bin/python -c " +import os, secrets, sqlite3, string, datetime +from argon2 import PasswordHasher +pw = \"\".join(secrets.choice(string.ascii_letters + string.digits) for _ in range(24)) +now = datetime.datetime.now() +c = sqlite3.connect(\"/crafty/app/config/db/crafty.sqlite\", timeout=30) +cur = c.execute( + \"UPDATE users SET password=?, last_update=?, valid_tokens_from=? WHERE username=?\", + (PasswordHasher().hash(pw), str(now), + str(datetime.datetime.now(datetime.timezone.utc)), \"admin\"), +) +c.commit() +print(\"rows updated:\", cur.rowcount) +out = \"/crafty/app/config/reset-creds.txt\" +open(out, \"w\").write(pw + chr(10)) +os.chmod(out, 0o600) +print(\"new password written to\", out) +"' +``` + +Expect `rows updated: 1`. Then read it, sign in, rotate in the UI, and delete the file: + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/reset-creds.txt' +# after signing in and rotating: +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- rm -f /crafty/app/config/reset-creds.txt' +``` + +`rows updated: 0` means the username is not `admin` — list the accounts (names only, never the +`password` column) and retry against the right one: + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c " +import sqlite3 +c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True) +print([r[0] for r in c.execute(\"SELECT username FROM users\")]) +"' +``` + +`database is locked` means Crafty held the write lock; the `timeout=30` usually rides it out, otherwise +retry. The panel picks the new hash up on the next login attempt — no restart needed, and a restart +costs a Minecraft-server stop (`strategy: Recreate` on an RWO volume), so do not restart reflexively. + +## When to escalate or stop + +- **Stop** as soon as you can sign in, and **rotate immediately** — do not leave the generated + bootstrap password in place as the standing admin credential, and do not leave `reset-creds.txt` on + the PVC. +- **Do not `kubectl edit`/`patch`/`apply`** the Deployment to work around a login problem. Flux + reconciles `main`; hand-applied changes are drift. Manifest changes go through a branch and a PR. +- **Do not delete `crafty.sqlite`** to force a "fresh install" and a new generated password. It holds + every server definition — jar, ports, heap, `auto_start` — and losing it means recreating each server + by hand, re-accepting the Mojang EULA, and re-`PATCH`ing `auto_start`. The worlds under + `/crafty/servers//` are not in git and not recreatable. The reset above is always cheaper. +- **Escalate to the Keycloak side** (`runbooks/keycloak-down.md`, + `runbooks/keycloak-client-secrets.md`) if there is no `AuthSuccess` in the sidecar log at all — that + is an IdP or client-secret problem and no Crafty password will help. +- **If `default-creds.txt` is absent on a pod whose DB says the password was never changed**, stop and + hand off: that combination should not occur, and it suggests the `config` subPath is not the volume + you think it is. Record what you measured rather than guessing. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index 626ae5f..deafa5f 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -60,6 +60,9 @@ configMapGenerator: - cnpg-exporter=cnpg-exporter.md - cnpg-primary-spread=cnpg-primary-spread.md - cnpg-wal-growth=cnpg-wal-growth.md + # Operator procedure, not alert-linked (Crafty's local login is a second factor behind + # the Keycloak gate and its bootstrap password lives only on the PVC). + - crafty-admin-credential=crafty-admin-credential.md - denied-flows=denied-flows.md - flux-reconcile=flux-reconcile.md - flux-suspended=flux-suspended.md