# crafty-admin-credential — Crafty's own login rejects you after Keycloak sign-in

**Not alert-driven.** This is an operator procedure, linked from
[`platform/crafty/README.md`](../platform/crafty/README.md). It covers getting into the Crafty panel the
first time, rotating that credential, and recovering when it is lost.

> **Everything below prints or sets a LIVE credential.** Do not paste the output into a PR, an issue, a
> chat, or an AI agent transcript. Read it, use it, rotate it. (Repo rule §6: secrets never enter a
> transcript.)

## What this means here

`crafty.home.keithjackson.dev` sits behind **two** independent authentications, and only the first is
Keycloak's:

1. the **oauth2-proxy sidecar** in the Crafty pod — Keycloak `homelab` realm, group `platform-admins`;
2. **Crafty's own local username/password login** — Crafty 4 has no OIDC and cannot consume the
   sidecar's identity headers, so it shows its own login page to an already-authenticated user.

So "I signed into Keycloak and I'm still not signed in" is the **expected** shape of this service, and
the real question is only ever *which* of the two rejected you. They fail differently:

| Where you are | What you see | Which layer |
|---|---|---|
| Bounced to Keycloak repeatedly, never reach Crafty | sidecar log: `No valid authentication in request` | oauth2-proxy |
| Authenticated but `403` | not in `platform-admins` | oauth2-proxy |
| **Crafty's own login page, credentials refused** | `POST /api/v2/auth/login/` → **401** | **Crafty local** |

The third row is the one that blocks setup, because **nothing in the UI tells you where that password
comes from**. Crafty generates it **once, on first boot only** — when the users table is empty
(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`) — and writes it to
`/crafty/app/config/default-creds.txt` (mode `0600`) on the `config` subPath of the `crafty-data` PVC.
It is **never in git** and **cannot be recovered from git**: a rebuild onto a fresh PVC generates a new
random password, and the old one is gone with the old volume.

Two things that are *not* the problem, both verified live:

- **Repeated wrong guesses cannot lock you out.** Crafty *records* failed attempts under
  `get_remote_ip()` (honours `X-Forwarded-For`) but *checks* the 3-strikes cooldown under
  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The halves key
  differently, so `max_login_attempts: 3` never fires. (It also means that throttle is not a control
  you have — the `platform-admins` gate is what limits who reaches the form at all.)
- **Crafty's `set_passwd` console command is unreachable here.** It calls `getpass.getpass()` and needs
  a TTY; the container runs `main.py -d -i` as a daemon. Recovery goes through the DB instead.

## First commands

`kubectl` does not exist on the workstation — the toolchain lives only in the pinned dev container, and
`dev.sh` picks a current-context **per invocation**, so always name the context.

**1. Is the pod healthy, and is the password still the generated one?**

```bash
./infra/devcontainer/dev.sh run bash -c '
kubectl --context k3s-prod -n crafty get pods -l app.kubernetes.io/name=crafty
kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
import sqlite3
c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
for r in c.execute(\"SELECT username,created,last_update,last_login FROM users WHERE user_id=1\"):
    print(\"created    :\",r[1]); print(\"last_update:\",r[2]); print(\"last_login :\",r[3])
"'
```

Read `last_update` against `created`: **equal ⇒ the password has never been changed**, so
`default-creds.txt` is still live. Different ⇒ someone rotated it and that file is stale — skip to
*Reset*.

**2. Read the bootstrap credential** (prints a live secret — see the warning at the top):

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/default-creds.txt'
```

The username is `admin`. Sign in at `https://crafty.home.keithjackson.dev` (Keycloak first, then this).

**3. Rotate it immediately**, in the UI: user menu → your profile → change password. That is the only
supported path that keeps Crafty's own bookkeeping consistent. Then confirm it took — re-run command 1
and check `last_update` has moved away from `created`.

## Likely causes, most likely first

1. **You never had the password.** By far the most common, and the reason this runbook exists. The
   credential was only ever on the PVC. → commands 1–3 above.
2. **The password was rotated and not recorded.** `last_update` ≠ `created` while nobody knows the
   value. `default-creds.txt` is now misleading, not authoritative. → *Reset* below.
3. **The PVC was rebuilt.** A fresh volume means a fresh random password *and* an empty panel — no
   servers, no worlds (`platform/crafty/README.md`, "the servers themselves are NOT in git"). Read
   `default-creds.txt` from the new pod; do not expect the old credential to work.
4. **You are actually stuck at the oauth2-proxy layer, not Crafty's.** Check which layer is talking
   before touching any password:
   ```bash
   ./infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n crafty logs deploy/crafty -c oauth2-proxy --tail=200 | grep -v kube-probe'
   ```
   `[AuthSuccess] ... groups:[platform-admins]` followed by a `200` on `/login` means oauth2-proxy did
   its job and the 401 is Crafty's. Repeated `No valid authentication in request` with no `AuthSuccess`
   means the session cookie is not surviving — see *cookie isolation* in
   `platform/crafty/README.md`; this sidecar must keep its own `--cookie-name` distinct from the shared
   proxy's, or the two evict each other.
5. **`platform-admins` membership lapsed.** A `403` rather than a login page. Fix in Keycloak, not here.

## Reset when the password is unknown

Crafty stores an **argon2** hash in `users.password`. This writes a fresh random password, invalidates
existing sessions, and drops the new value into a `0600` file **inside the pod** — deliberately
mirroring Crafty's own `default-creds.txt` idiom so the secret never crosses argv, stdin or shell
history. (`dev.sh` does **not** forward stdin — a `kubectl exec -i` here hangs forever, which is why the
password is generated in-pod rather than passed in.)

Two details that will otherwise cost you the afternoon: argon2 is installed **only** in Crafty's venv,
so it must be `/crafty/.venv/bin/python`, not the system `python3`; and this is a **mutating operator
action**, not an agent action.

```bash
./infra/devcontainer/dev.sh run bash -c \
 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- /crafty/.venv/bin/python -c "
import os, secrets, sqlite3, string, datetime
from argon2 import PasswordHasher
pw  = \"\".join(secrets.choice(string.ascii_letters + string.digits) for _ in range(24))
now = datetime.datetime.now()
c   = sqlite3.connect(\"/crafty/app/config/db/crafty.sqlite\", timeout=30)
cur = c.execute(
    \"UPDATE users SET password=?, last_update=?, valid_tokens_from=? WHERE username=?\",
    (PasswordHasher().hash(pw), str(now),
     str(datetime.datetime.now(datetime.timezone.utc)), \"admin\"),
)
c.commit()
print(\"rows updated:\", cur.rowcount)
out = \"/crafty/app/config/reset-creds.txt\"
open(out, \"w\").write(pw + chr(10))
os.chmod(out, 0o600)
print(\"new password written to\", out)
"'
```

Expect `rows updated: 1`. Then read it, sign in, rotate in the UI, and delete the file:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/reset-creds.txt'
# after signing in and rotating:
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- rm -f /crafty/app/config/reset-creds.txt'
```

`rows updated: 0` means the username is not `admin` — list the accounts (names only, never the
`password` column) and retry against the right one:

```bash
./infra/devcontainer/dev.sh run bash -c \
 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
import sqlite3
c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
print([r[0] for r in c.execute(\"SELECT username FROM users\")])
"'
```

`database is locked` means Crafty held the write lock; the `timeout=30` usually rides it out, otherwise
retry. The panel picks the new hash up on the next login attempt — no restart needed, and a restart
costs a Minecraft-server stop (`strategy: Recreate` on an RWO volume), so do not restart reflexively.

## When to escalate or stop

- **Stop** as soon as you can sign in, and **rotate immediately** — do not leave the generated
  bootstrap password in place as the standing admin credential, and do not leave `reset-creds.txt` on
  the PVC.
- **Do not `kubectl edit`/`patch`/`apply`** the Deployment to work around a login problem. Flux
  reconciles `main`; hand-applied changes are drift. Manifest changes go through a branch and a PR.
- **Do not delete `crafty.sqlite`** to force a "fresh install" and a new generated password. It holds
  every server definition — jar, ports, heap, `auto_start` — and losing it means recreating each server
  by hand, re-accepting the Mojang EULA, and re-`PATCH`ing `auto_start`. The worlds under
  `/crafty/servers/<uuid>/` are not in git and not recreatable. The reset above is always cheaper.
- **Escalate to the Keycloak side** (`runbooks/keycloak-down.md`,
  `runbooks/keycloak-client-secrets.md`) if there is no `AuthSuccess` in the sidecar log at all — that
  is an IdP or client-secret problem and no Crafty password will help.
- **If `default-creds.txt` is absent on a pod whose DB says the password was never changed**, stop and
  hand off: that combination should not occur, and it suggests the `config` subPath is not the volume
  you think it is. Record what you measured rather than guessing.