jakman85andClaude Opus 5
dd1d5a6d0a
Reported: "crafty does not sign in after signing into keycloak, and there is
no way to set the service up without credentials."
MEASURED, not inferred. The sidecar log settles which layer refused:
GET / -> 302 to Keycloak
[AuthSuccess] ... groups:[platform-admins] <- oauth2-proxy is FINE
GET /oauth2/callback -> 302
GET /login?next=/panel/dashboard -> 200 <- Crafty's OWN login page
POST /api/v2/auth/login/ -> 401 <- Crafty refused
So the Keycloak gate was never broken. Crafty 4 has no OIDC and cannot consume
the sidecar's identity headers, so it shows its own local login to an already
authenticated user. That second factor is intended. What was NOT intended is
that nothing documented where its password comes from, which is exactly why
setup was impossible: Crafty generates it ONCE on first boot (users table
empty -> DatabaseBuilder.default_settings()) into /crafty/app/config/
default-creds.txt, mode 0600, on the PVC's `config` subPath. Never in git;
a fresh PVC generates a new one. The README asserted the double login was
"expected, not a bug" and then left the reader with no way forward.
Verified in-cluster that the existing credential is still usable, so the
service can be set up without touching the DB: users.last_update equals
users.created for `admin` (09/05/2026, 08:10:04), i.e. the password has never
been changed, and admin's last_login is 09/14 from 127.0.0.1 -- the API login
that created the "homelab" server. default-creds.txt is therefore live.
ALSO FIXED -- a real, separate defect found while proving the above, which
produces the reported symptom intermittently and would have been blamed on
Keycloak forever: this sidecar and the SHARED forward-auth oauth2-proxy both
emitted oauth2-proxy's DEFAULT cookie names (_oauth2_proxy,
_oauth2_proxy_csrf) scoped to .home.keithjackson.dev, while holding DIFFERENT
cookie-secrets (compared the two live Secrets: they differ, so neither can
decrypt the other's cookie). One cookie pair, two proxies: whichever
authenticated last clobbered the other's session; the loser logged "No valid
authentication in request", bounced to Keycloak, and -- SSO session still live
-- was redirected straight back, clobbering it again. Fixed on this side only
(renaming the shared proxy's cookie would sign out every other gated app):
a unique --cookie-name plus a host-scoped --cookie-domain, since a per-app
reverse proxy has no reason to broadcast its session zone-wide.
--whitelist-domain stays zone-wide: it governs redirect targets, not cookies.
Two traps measured rather than assumed, both now documented:
- The 3-strikes cooldown CANNOT fire behind this sidecar. Crafty records
attempts under get_remote_ip() (honours X-Forwarded-For -- the live
tracker key is the chain "192.168.2.132, 10.0.0.123") but checks the
cooldown under request.remote_ip, which is always 127.0.0.1 here. The
halves key differently, so get_recent_attempts() always reads empty.
Good (no lockout risk) and bad (that throttle is not a control we have).
- Crafty's `set_passwd` console command is unreachable in k8s: it calls
getpass.getpass() and needs a TTY; the container runs `main.py -d -i`.
Recovery has to go through the DB, and argon2 is installed ONLY in
/crafty/.venv/bin/python -- the system python3 lacks it.
Deliberately NOT done: an initContainer seeding default.json from a SOPS
Secret to make a rebuilt Crafty's password deterministic. Crafty does support
that hook (find_default_password reads app/config/default.json, min 8 chars),
but the image entrypoint pulls its default config ONLY when app/config is
EMPTY -- so writing a file there pre-boot suppresses the whole config
bootstrap. Buying reproducibility means co-owning image-internal bootstrap
logic that a 4.11 bump could silently change, to replace a one-command
retrieval. Recorded in DECISIONS-LOG as considered and rejected.
New runbook runbooks/crafty-admin-credential.md: tell the two auth layers
apart from the sidecar log, retrieve and rotate the bootstrap credential,
and reset a lost one via argon2 -- generating the password in-pod into a 0600
file rather than passing it in, because dev.sh does not forward stdin (a
`kubectl exec -i` through it hangs; verified). Registered in the configMapGenerator
so the URL actually resolves, not just the file exists.
Gate: toolbox ./ci/gate.sh -- lint/l0/render-golden/l1 green, spec-immutability
green. rc 3 overall: gitea-settings SKIPPED (no SETTINGS_API_TOKEN) and L2/L3
NOT RUN (CLUSTER_AVAILABLE unset). No runtime browser round-trip performed --
the sign-in fix is UNVERIFIED end-to-end and needs the human test in the README.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
186 lines
12 KiB
Markdown
186 lines
12 KiB
Markdown
# platform/crafty
|
||
|
||
**Crafty Controller 4** — a web panel that creates and manages multiple Minecraft servers (Java +
|
||
Bedrock) as **subprocesses inside its own container**. Its web UI is **Keycloak-gated** via a per-pod
|
||
**oauth2-proxy reverse-proxy sidecar** (the decided **Option B**, see
|
||
[`../oauth2-proxy/README.md`](../oauth2-proxy/README.md)). Game ports are exposed **un-gated** on a
|
||
Cilium L2 LoadBalancer (game protocols authenticate players themselves).
|
||
|
||
- **Owner:** WS-APP (auth wiring shared with WS-SEC)
|
||
- **Flux tier:** own non-gating `clusters/base/crafty.yaml` (`dependsOn platform-config`, `wait:false`)
|
||
- **Namespace:** `crafty` (PLATFORM namespace — `gateway-access=shared`, NOT `app-namespace=true`)
|
||
|
||
## Deploy summary
|
||
|
||
| Thing | Value |
|
||
|---|---|
|
||
| Image | `registry.gitlab.com/crafty-controller/crafty-4:4.10.8` (current stable; bare-semver tags, no `v`) |
|
||
| Web UI | HTTPS **self-signed** on `:8443` (reached only via the sidecar / same-pod) |
|
||
| Sidecar | `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1`, reverse-proxy on `:4180` |
|
||
| Data | ONE Longhorn RWO PVC `crafty-data` (20Gi), mounted 5× by subPath |
|
||
| PVC subPaths | `servers`→`/crafty/servers`, `backups`→`/crafty/backups`, `logs`→`/crafty/logs`, `import`→`/crafty/import`, `config`→`/crafty/app/config` |
|
||
| Identity | `runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true` (see below) |
|
||
| Web host | `crafty.home.keithjackson.dev` → shared Gateway → `crafty` Service :80 → sidecar :4180 → Crafty :8443 |
|
||
| Game VIP | **192.168.1.55** (Cilium L2 LoadBalancer; `.53`=Gateway, `.54`=unifi, `.55`=crafty) |
|
||
| Game ports | starter set: **TCP 25565-25567** (Java), **UDP 19132-19134** (Bedrock) |
|
||
|
||
### fsGroup / user decision (the seerr lesson, applied)
|
||
|
||
Crafty's Dockerfile does `useradd -g root -M crafty` (UID **1000**, primary group **root/GID 0**),
|
||
`chown -R crafty:root /crafty`, and makes `/crafty` group-writable — the OpenShift/K8s
|
||
"arbitrary-UID, root-group-perms" pattern. A fresh Longhorn PVC is `root:root` and not group-writable,
|
||
so the non-root crafty process could not create `/crafty/servers/*` etc. and would CrashLoop (exactly
|
||
seerr's failure at UID 1000 on a `root:root` PVC). Fix: **`fsGroup: 0`** makes the kubelet group-own
|
||
the mounted volume to GID 0, mark it group-writable+SGID, and add GID 0 as a supplemental group;
|
||
**`runAsGroup: 0`** + **`runAsUser: 1000`** pin the image's own `crafty:root` identity. Longhorn's
|
||
`CSIDriver fsGroupPolicy: ReadWriteOnceWithFSType` applies fsGroup to this RWO filesystem volume.
|
||
|
||
> **HUMAN runtime check:** confirm the pod writes its SQLite DB and can start a server. If the image's
|
||
> `crafty` UID is ever not 1000, adjust `runAsUser` (the launcher's non-root path runs directly as the
|
||
> current UID, so any non-root UID with GID 0 works, but 1000 matches `getpwuid` cleanly).
|
||
|
||
## Auth = Option B (per-app oauth2-proxy reverse-proxy)
|
||
|
||
Cilium 1.16.5 cannot inject `ext_authz` into the shared Gateway's machine-generated Envoy config, and
|
||
native Gateway-API ExternalAuth ships in no released Cilium yet (see `../oauth2-proxy/README.md`). So
|
||
Crafty's UI is gated by a **dedicated oauth2-proxy sidecar in the Crafty pod** (`--reverse-proxy`,
|
||
`--upstream=https://127.0.0.1:8443`, `--ssl-upstream-insecure-skip-verify=true`,
|
||
`--allowed-groups=platform-admins`, `--skip-provider-button`, cookie/redirect for
|
||
`crafty.home.keithjackson.dev`). The `crafty` Service targets the sidecar, never Crafty's `:8443`, so
|
||
the gate is **in the data path** (fail-closed: no valid `platform-admins` session ⇒ no upstream).
|
||
|
||
#### Cookie isolation (why the session used to vanish)
|
||
|
||
This sidecar and the **shared** forward-auth oauth2-proxy are independent instances with **different
|
||
`cookie-secret` values** (measured in-cluster — neither can decrypt the other's cookie). Both used to
|
||
emit oauth2-proxy's *default* cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
|
||
`.home.keithjackson.dev`, so the browser held **one** cookie pair for **both**: whichever proxy
|
||
authenticated last clobbered the other's session, and the loser logged
|
||
`No valid authentication in request`, bounced to Keycloak, and — because the Keycloak SSO session was
|
||
still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents
|
||
exactly as *"I signed in to Keycloak but I'm still not signed in."*
|
||
|
||
Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app):
|
||
`--cookie-name=_oauth2_proxy_crafty` and a **host-scoped** `--cookie-domain=crafty.home.keithjackson.dev`.
|
||
`--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
|
||
|
||
#### Crafty's own local login — this is where setup actually blocks
|
||
|
||
Crafty has **no OIDC**, so after oauth2-proxy admits the user they still meet Crafty's own
|
||
username/password page. That second factor is intended. What was **not** intended — and what made the
|
||
service impossible to set up — is that **nothing documented where that password comes from**.
|
||
|
||
Crafty generates it **once, on first boot only**, when the users table is empty
|
||
(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`), and writes it to
|
||
**`/crafty/app/config/default-creds.txt`** (mode `0600`, on the `config` subPath of the PVC). It is
|
||
**never** in git and **cannot** be recovered from git: a rebuild onto a fresh PVC generates a *new*
|
||
random password. Submitting anything else yields `POST /api/v2/auth/login/` → **401**, which is exactly
|
||
what a browser sees with no hint as to why.
|
||
|
||
**→ Retrieval, rotation, and lost-password recovery: [`../../runbooks/crafty-admin-credential.md`](../../runbooks/crafty-admin-credential.md).**
|
||
|
||
Two traps that runbook covers:
|
||
|
||
- Crafty's `set_passwd` console command is **unreachable** in Kubernetes — it calls
|
||
`getpass.getpass()` and so needs a TTY, while the container runs `main.py -d -i` (daemon).
|
||
Recovery therefore has to go through the DB with Crafty's own argon2 hasher.
|
||
- Crafty's own 3-strikes brute-force cooldown is **inert behind this sidecar**, verified live. Crafty
|
||
*records* failed attempts under `get_remote_ip()` (which honours `X-Forwarded-For` — the live
|
||
tracker key is the chain `"192.168.2.132, 10.0.0.123"`) but *checks* for a cooldown under
|
||
`self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The two halves key
|
||
differently, so `get_recent_attempts()` always reads an empty list and `max_login_attempts: 3`
|
||
never fires. Good news for the operator (**repeated guesses cannot lock you out**), but it means
|
||
the only thing actually limiting who can reach the login form is the oauth2-proxy
|
||
`platform-admins` gate — worth knowing before anyone treats that cooldown as a control.
|
||
|
||
### Keycloak client
|
||
|
||
Confidential auth-code client **`crafty`** in the `homelab` realm — redirect
|
||
`https://crafty.home.keithjackson.dev/oauth2/callback`, `webOrigins` the same host, a `groups`
|
||
membership mapper (bare names). Provisioned **live via kcadm** at authoring time (2026-08-28) and
|
||
captured as the durable GitOps source of truth in
|
||
[`../keycloak/keycloak-realm-import.yaml`](../keycloak/keycloak-realm-import.yaml) (secret injected via
|
||
the `CRAFTY_CLIENT_SECRET` placeholder).
|
||
|
||
### Secret locations (SOPS/age)
|
||
|
||
| Secret | File | Key |
|
||
|---|---|---|
|
||
| oauth2-proxy client id/secret + cookie secret | `crafty-oauth2-proxy-credentials.sops.yaml` (this dir) | `client-id`, `client-secret`, `cookie-secret` |
|
||
| Keycloak client secret (mirror) | `../keycloak/keycloak-realm-clients.sops.yaml` | `crafty-client-secret` |
|
||
|
||
`client-secret` here **must equal** `crafty-client-secret` there (cross-namespace mirror — a Secret
|
||
can't span namespaces). Both are age-encrypted to the repo `sops-age` recipient and Flux-decryptable
|
||
(verified: both decrypt and the two values match).
|
||
|
||
## Game-port L4 exposure
|
||
|
||
The Gateway is L7-HTTPS-only; raw TCP/UDP game ports can't ride it. `service-l4.yaml` exposes them as
|
||
two `type=LoadBalancer` Services (TCP + UDP) sharing the pinned VIP **192.168.1.55**
|
||
(`lbipam.cilium.io/sharing-key: crafty`), ARP-announced by the existing `default-l2` policy. This path
|
||
is **NOT auth-gated** — Minecraft Java (Mojang/Microsoft) and Bedrock (Xbox Live) authenticate players
|
||
themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → 192.168.1.55`.
|
||
|
||
### Adding more game ports (when the operator creates servers)
|
||
|
||
The operator has **not created any servers yet**; the starter set covers 3 Java + 3 Bedrock slots. To
|
||
add capacity:
|
||
|
||
1. In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP).
|
||
2. Add that port to the matching Service in `service-l4.yaml` (one
|
||
`{ name, port, targetPort, protocol }` line — K8s Services enumerate ports, no ranges).
|
||
3. Add it to the game-port ingress rule in `networkpolicy.yaml` (`crafty-allow-ingress`).
|
||
4. (Optional) add a `containerPort` in `deployment.yaml` for documentation.
|
||
5. Commit → Flux reconciles.
|
||
|
||
## Policies
|
||
|
||
`namespace.yaml` sets only `gateway-access=shared` (platform namespace, like headlamp/homepage), so
|
||
Kyverno does not auto-generate the app baseline. `networkpolicy.yaml` ships it explicitly (defense in
|
||
depth): default-deny (DNS-only egress) + Cilium allows — Gateway→sidecar `:4180`, world/LAN→game
|
||
ports, host→probe ports, and egress `world` (server-jar downloads + the OIDC path to Keycloak via the
|
||
Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subject to network policy.
|
||
|
||
## Verification
|
||
|
||
- `make -C ci gate` — green (hand-authored platform manifests; render-golden unaffected).
|
||
- `kustomize build` of `platform/crafty`, `clusters/base`, `clusters/dev` — clean.
|
||
|
||
### Human tests (post-deploy)
|
||
|
||
1. **Browser OIDC round-trip:** incognito → `https://crafty.home.keithjackson.dev` → expect a 302 to
|
||
`https://keycloak.home.keithjackson.dev/realms/homelab/...` (NOT the Crafty UI). Log in as a
|
||
**platform-admins** user (MFA/OTP enforced) → land on Crafty's own login → sign in with Crafty's
|
||
local admin. A **non-platform-admins** user → **403** (authenticated but not authorized).
|
||
2. **Pod health / fsGroup:** `kubectl -n crafty get pod` Ready 2/2; Crafty writes its DB under
|
||
`/crafty/app/config` and `/crafty/servers` without EACCES.
|
||
3. **Server-create + game reach:** in the Crafty UI create a Java server bound to `25565`; start it;
|
||
connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or `192.168.1.55:25565`)
|
||
from the LAN and confirm the world loads. (Bedrock: a server on `19132/udp` reached at the VIP.)
|
||
|
||
## Runtime state: the servers themselves are NOT in git
|
||
|
||
As of 2026-09-14 one Java server exists — **"homelab"**, Paper 26.2, TCP 25565, 2 GiB heap, autostart
|
||
on, UUID `13c7b85d-00bb-4a58-8c86-2433ca47891c`. It was created through Crafty's v2 API, and **Mojang's
|
||
EULA (<https://aka.ms/MinecraftEULA>) was accepted on the operator's behalf** to let it start; see the
|
||
DECISIONS-LOG entry of that date.
|
||
|
||
Everything in this directory describes only the *capacity* to run Minecraft servers. The servers are
|
||
application state on the `crafty-data` Longhorn PVC, in two places, and **Flux cannot recreate either**:
|
||
|
||
- `/crafty/app/config/db/` (`config` subPath) — Crafty's SQLite row: name, port, jar, launch command,
|
||
heap sizes, `auto_start` / `crash_detection`.
|
||
- `/crafty/servers/<uuid>/` (`servers` subPath) — `paper.jar`, `eula.txt`, `server.properties`, and
|
||
**the generated world**. The jar is re-downloadable; the world is not.
|
||
|
||
So a rebuild from git alone yields a healthy but **empty** Crafty with nothing listening on 25565.
|
||
Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH
|
||
`auto_start` to true — see the next note. The world starts over. **Treat this PVC as backup-relevant.**
|
||
|
||
### Trap: the create API silently ignores `autostart`
|
||
|
||
`POST /api/v2/servers` accepts and validates `autostart` and `crashdetection`, but never stores them —
|
||
the persisted columns are `auto_start` and `crash_detection`. After a create that asked for `true`,
|
||
both read back `false`. Set them afterwards with `PATCH /api/v2/servers/<uuid>` and **read them back to
|
||
confirm**, or the server will not come up after a pod restart.
|