home-network-k3s-impl/runbooks/crafty-admin-credential.md

crafty-admin-credential — Crafty's own login rejects you after Keycloak sign-in

Not alert-driven. This is an operator procedure, linked from platform/crafty/README.md. It covers getting into the Crafty panel the first time, rotating that credential, and recovering when it is lost.

Everything below prints or sets a LIVE credential. Do not paste the output into a PR, an issue, a chat, or an AI agent transcript. Read it, use it, rotate it. (Repo rule §6: secrets never enter a transcript.)

What this means here

crafty.home.keithjackson.dev sits behind two independent authentications, and only the first is Keycloak's:

1. the oauth2-proxy sidecar in the Crafty pod — Keycloak homelab realm, group platform-admins;
2. Crafty's own local username/password login — Crafty 4 has no OIDC and cannot consume the sidecar's identity headers, so it shows its own login page to an already-authenticated user.

So "I signed into Keycloak and I'm still not signed in" is the expected shape of this service, and the real question is only ever which of the two rejected you. They fail differently:

Where you are	What you see	Which layer
Bounced to Keycloak repeatedly, never reach Crafty	sidecar log: No valid authentication in request	oauth2-proxy
Authenticated but 403	not in platform-admins	oauth2-proxy
Crafty's own login page, credentials refused	POST /api/v2/auth/login/ → 401	Crafty local

The third row is the one that blocks setup, because nothing in the UI tells you where that password comes from. Crafty generates it once, on first boot only — when the users table is empty (main.py FRESH_INSTALL → DatabaseBuilder.default_settings()) — and writes it to /crafty/app/config/default-creds.txt (mode 0600) on the config subPath of the crafty-data PVC. It is never in git and cannot be recovered from git: a rebuild onto a fresh PVC generates a new random password, and the old one is gone with the old volume.

Two things that are not the problem, both verified live:

• Repeated wrong guesses cannot lock you out. Crafty records failed attempts under get_remote_ip() (honours X-Forwarded-For) but checks the 3-strikes cooldown under self.request.remote_ip, which behind the sidecar is always 127.0.0.1. The halves key differently, so max_login_attempts: 3 never fires. (It also means that throttle is not a control you have — the platform-admins gate is what limits who reaches the form at all.)
• Crafty's set_passwd console command is unreachable here. It calls getpass.getpass() and needs a TTY; the container runs main.py -d -i as a daemon. Recovery goes through the DB instead.

First commands

kubectl does not exist on the workstation — the toolchain lives only in the pinned dev container, and dev.sh picks a current-context per invocation, so always name the context.

1. Is the pod healthy, and is the password still the generated one?

./infra/devcontainer/dev.sh run bash -c '
kubectl --context k3s-prod -n crafty get pods -l app.kubernetes.io/name=crafty
kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
import sqlite3
c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
for r in c.execute(\"SELECT username,created,last_update,last_login FROM users WHERE user_id=1\"):
    print(\"created    :\",r[1]); print(\"last_update:\",r[2]); print(\"last_login :\",r[3])
"'

Read last_update against created: equal ⇒ the password has never been changed, so default-creds.txt is still live. Different ⇒ someone rotated it and that file is stale — skip to Reset.

2. Read the bootstrap credential (prints a live secret — see the warning at the top):

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/default-creds.txt'

The username is admin. Sign in at https://crafty.home.keithjackson.dev (Keycloak first, then this).

3. Rotate it immediately, in the UI: user menu → your profile → change password. That is the only supported path that keeps Crafty's own bookkeeping consistent. Then confirm it took — re-run command 1 and check last_update has moved away from created.

Likely causes, most likely first

1. You never had the password. By far the most common, and the reason this runbook exists. The credential was only ever on the PVC. → commands 1–3 above.
2. The password was rotated and not recorded. last_update ≠ created while nobody knows the value. default-creds.txt is now misleading, not authoritative. → Reset below.
3. The PVC was rebuilt. A fresh volume means a fresh random password and an empty panel — no servers, no worlds (platform/crafty/README.md, "the servers themselves are NOT in git"). Read default-creds.txt from the new pod; do not expect the old credential to work.
4. You are actually stuck at the oauth2-proxy layer, not Crafty's. Check which layer is talking before touching any password:

   ./infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n crafty logs deploy/crafty -c oauth2-proxy --tail=200 | grep -v kube-probe'
   

   [AuthSuccess] ... groups:[platform-admins] followed by a 200 on /login means oauth2-proxy did its job and the 401 is Crafty's. Repeated No valid authentication in request with no AuthSuccess means the session cookie is not surviving — see cookie isolation in platform/crafty/README.md; this sidecar must keep its own --cookie-name distinct from the shared proxy's, or the two evict each other.
5. platform-admins membership lapsed. A 403 rather than a login page. Fix in Keycloak, not here.

Reset when the password is unknown

Crafty stores an argon2 hash in users.password. This writes a fresh random password, invalidates existing sessions, and drops the new value into a 0600 file inside the pod — deliberately mirroring Crafty's own default-creds.txt idiom so the secret never crosses argv, stdin or shell history. (dev.sh does not forward stdin — a kubectl exec -i here hangs forever, which is why the password is generated in-pod rather than passed in.)

Two details that will otherwise cost you the afternoon: argon2 is installed only in Crafty's venv, so it must be /crafty/.venv/bin/python, not the system python3; and this is a mutating operator action, not an agent action.

./infra/devcontainer/dev.sh run bash -c \
 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- /crafty/.venv/bin/python -c "
import os, secrets, sqlite3, string, datetime
from argon2 import PasswordHasher
pw  = \"\".join(secrets.choice(string.ascii_letters + string.digits) for _ in range(24))
now = datetime.datetime.now()
c   = sqlite3.connect(\"/crafty/app/config/db/crafty.sqlite\", timeout=30)
cur = c.execute(
    \"UPDATE users SET password=?, last_update=?, valid_tokens_from=? WHERE username=?\",
    (PasswordHasher().hash(pw), str(now),
     str(datetime.datetime.now(datetime.timezone.utc)), \"admin\"),
)
c.commit()
print(\"rows updated:\", cur.rowcount)
out = \"/crafty/app/config/reset-creds.txt\"
open(out, \"w\").write(pw + chr(10))
os.chmod(out, 0o600)
print(\"new password written to\", out)
"'

Expect rows updated: 1. Then read it, sign in, rotate in the UI, and delete the file:

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/reset-creds.txt'
# after signing in and rotating:
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- rm -f /crafty/app/config/reset-creds.txt'

rows updated: 0 means the username is not admin — list the accounts (names only, never the password column) and retry against the right one:

./infra/devcontainer/dev.sh run bash -c \
 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
import sqlite3
c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
print([r[0] for r in c.execute(\"SELECT username FROM users\")])
"'

database is locked means Crafty held the write lock; the timeout=30 usually rides it out, otherwise retry. The panel picks the new hash up on the next login attempt — no restart needed, and a restart costs a Minecraft-server stop (strategy: Recreate on an RWO volume), so do not restart reflexively.

When to escalate or stop

• Stop as soon as you can sign in, and rotate immediately — do not leave the generated bootstrap password in place as the standing admin credential, and do not leave reset-creds.txt on the PVC.
• Do not kubectl edit/patch/apply the Deployment to work around a login problem. Flux reconciles main; hand-applied changes are drift. Manifest changes go through a branch and a PR.
• Do not delete crafty.sqlite to force a "fresh install" and a new generated password. It holds every server definition — jar, ports, heap, auto_start — and losing it means recreating each server by hand, re-accepting the Mojang EULA, and re-PATCHing auto_start. The worlds under /crafty/servers/<uuid>/ are not in git and not recreatable. The reset above is always cheaper.
• Escalate to the Keycloak side (runbooks/keycloak-down.md, runbooks/keycloak-client-secrets.md) if there is no AuthSuccess in the sidecar log at all — that is an IdP or client-secret problem and no Crafty password will help.
• If default-creds.txt is absent on a pod whose DB says the password was never changed, stop and hand off: that combination should not occur, and it suggests the config subPath is not the volume you think it is. Record what you measured rather than guessing.