Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
dd1d5a6d0ac383be6a57e763a32e0c40fdef3ce0
home-network-k3s-impl/platform/crafty
T
History
jakman85andClaude Opus 5 dd1d5a6d0a fix(crafty): the Keycloak gate works -- Crafty's own login was the wall 
Reported: "crafty does not sign in after signing into keycloak, and there is
no way to set the service up without credentials."

MEASURED, not inferred. The sidecar log settles which layer refused:

  GET /                        -> 302 to Keycloak
  [AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy is FINE
  GET /oauth2/callback         -> 302
  GET /login?next=/panel/dashboard -> 200           <- Crafty's OWN login page
  POST /api/v2/auth/login/     -> 401               <- Crafty refused

So the Keycloak gate was never broken. Crafty 4 has no OIDC and cannot consume
the sidecar's identity headers, so it shows its own local login to an already
authenticated user. That second factor is intended. What was NOT intended is
that nothing documented where its password comes from, which is exactly why
setup was impossible: Crafty generates it ONCE on first boot (users table
empty -> DatabaseBuilder.default_settings()) into /crafty/app/config/
default-creds.txt, mode 0600, on the PVC's `config` subPath. Never in git;
a fresh PVC generates a new one. The README asserted the double login was
"expected, not a bug" and then left the reader with no way forward.

Verified in-cluster that the existing credential is still usable, so the
service can be set up without touching the DB: users.last_update equals
users.created for `admin` (09/05/2026, 08:10:04), i.e. the password has never
been changed, and admin's last_login is 09/14 from 127.0.0.1 -- the API login
that created the "homelab" server. default-creds.txt is therefore live.

ALSO FIXED -- a real, separate defect found while proving the above, which
produces the reported symptom intermittently and would have been blamed on
Keycloak forever: this sidecar and the SHARED forward-auth oauth2-proxy both
emitted oauth2-proxy's DEFAULT cookie names (_oauth2_proxy,
_oauth2_proxy_csrf) scoped to .home.keithjackson.dev, while holding DIFFERENT
cookie-secrets (compared the two live Secrets: they differ, so neither can
decrypt the other's cookie). One cookie pair, two proxies: whichever
authenticated last clobbered the other's session; the loser logged "No valid
authentication in request", bounced to Keycloak, and -- SSO session still live
-- was redirected straight back, clobbering it again. Fixed on this side only
(renaming the shared proxy's cookie would sign out every other gated app):
a unique --cookie-name plus a host-scoped --cookie-domain, since a per-app
reverse proxy has no reason to broadcast its session zone-wide.
--whitelist-domain stays zone-wide: it governs redirect targets, not cookies.

Two traps measured rather than assumed, both now documented:
  - The 3-strikes cooldown CANNOT fire behind this sidecar. Crafty records
    attempts under get_remote_ip() (honours X-Forwarded-For -- the live
    tracker key is the chain "192.168.2.132, 10.0.0.123") but checks the
    cooldown under request.remote_ip, which is always 127.0.0.1 here. The
    halves key differently, so get_recent_attempts() always reads empty.
    Good (no lockout risk) and bad (that throttle is not a control we have).
  - Crafty's `set_passwd` console command is unreachable in k8s: it calls
    getpass.getpass() and needs a TTY; the container runs `main.py -d -i`.
    Recovery has to go through the DB, and argon2 is installed ONLY in
    /crafty/.venv/bin/python -- the system python3 lacks it.

Deliberately NOT done: an initContainer seeding default.json from a SOPS
Secret to make a rebuilt Crafty's password deterministic. Crafty does support
that hook (find_default_password reads app/config/default.json, min 8 chars),
but the image entrypoint pulls its default config ONLY when app/config is
EMPTY -- so writing a file there pre-boot suppresses the whole config
bootstrap. Buying reproducibility means co-owning image-internal bootstrap
logic that a 4.11 bump could silently change, to replace a one-command
retrieval. Recorded in DECISIONS-LOG as considered and rejected.

New runbook runbooks/crafty-admin-credential.md: tell the two auth layers
apart from the sidecar log, retrieve and rotate the bootstrap credential,
and reset a lost one via argon2 -- generating the password in-pod into a 0600
file rather than passing it in, because dev.sh does not forward stdin (a
`kubectl exec -i` through it hangs; verified). Registered in the configMapGenerator
so the URL actually resolves, not just the file exists.

Gate: toolbox ./ci/gate.sh -- lint/l0/render-golden/l1 green, spec-immutability
green. rc 3 overall: gitea-settings SKIPPED (no SETTINGS_API_TOKEN) and L2/L3
NOT RUN (CLUSTER_AVAILABLE unset). No runtime browser round-trip performed --
the sign-in fix is UNVERIFIED end-to-end and needs the human test in the README.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:31:36 -05:00
..
crafty-oauth2-proxy-credentials.sops.yaml
Onboard Crafty Controller (Minecraft panel) with Keycloak-gated web UI
2026-08-28 15:39:51 -05:00
crafty-oauth2-proxy-credentials.sops.yaml.example
Onboard Crafty Controller (Minecraft panel) with Keycloak-gated web UI
2026-08-28 15:39:51 -05:00
deployment.yaml
fix(crafty): the Keycloak gate works -- Crafty's own login was the wall
2026-09-29 19:31:36 -05:00
httproute.yaml
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
2026-09-04 22:43:54 -05:00
kustomization.yaml
Onboard Crafty Controller (Minecraft panel) with Keycloak-gated web UI
2026-08-28 15:39:51 -05:00
namespace.yaml
Onboard Crafty Controller (Minecraft panel) with Keycloak-gated web UI
2026-08-28 15:39:51 -05:00
networkpolicy.yaml
fix(net): allow SSO apps egress to keycloak backend (Cilium 1.20 CVE-2025-30162)
2026-08-30 23:34:03 -05:00
pvc.yaml
Onboard Crafty Controller (Minecraft panel) with Keycloak-gated web UI
2026-08-28 15:39:51 -05:00
README.md
fix(crafty): the Keycloak gate works -- Crafty's own login was the wall
2026-09-29 19:31:36 -05:00
service-l4.yaml
fix(l4): externalTrafficPolicy Cluster on unifi + crafty L4 VIPs
2026-09-14 01:32:47 -05:00
service.yaml
Onboard Crafty Controller (Minecraft panel) with Keycloak-gated web UI
2026-08-28 15:39:51 -05:00

README.md

platform/crafty

Crafty Controller 4 — a web panel that creates and manages multiple Minecraft servers (Java + Bedrock) as subprocesses inside its own container. Its web UI is Keycloak-gated via a per-pod oauth2-proxy reverse-proxy sidecar (the decided Option B, see ../oauth2-proxy/README.md). Game ports are exposed un-gated on a Cilium L2 LoadBalancer (game protocols authenticate players themselves).

  • Owner: WS-APP (auth wiring shared with WS-SEC)
  • Flux tier: own non-gating clusters/base/crafty.yaml (dependsOn platform-config, wait:false)
  • Namespace: crafty (PLATFORM namespace — gateway-access=shared, NOT app-namespace=true)

Deploy summary

Thing Value
Image registry.gitlab.com/crafty-controller/crafty-4:4.10.8 (current stable; bare-semver tags, no v)
Web UI HTTPS self-signed on :8443 (reached only via the sidecar / same-pod)
Sidecar quay.io/oauth2-proxy/oauth2-proxy:v7.8.1, reverse-proxy on :4180
Data ONE Longhorn RWO PVC crafty-data (20Gi), mounted 5× by subPath
PVC subPaths servers→/crafty/servers, backups→/crafty/backups, logs→/crafty/logs, import→/crafty/import, config→/crafty/app/config
Identity runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true (see below)
Web host crafty.home.keithjackson.dev → shared Gateway → crafty Service :80 → sidecar :4180 → Crafty :8443
Game VIP 192.168.1.55 (Cilium L2 LoadBalancer; .53=Gateway, .54=unifi, .55=crafty)
Game ports starter set: TCP 25565-25567 (Java), UDP 19132-19134 (Bedrock)

fsGroup / user decision (the seerr lesson, applied)

Crafty's Dockerfile does useradd -g root -M crafty (UID 1000, primary group root/GID 0), chown -R crafty:root /crafty, and makes /crafty group-writable — the OpenShift/K8s "arbitrary-UID, root-group-perms" pattern. A fresh Longhorn PVC is root:root and not group-writable, so the non-root crafty process could not create /crafty/servers/* etc. and would CrashLoop (exactly seerr's failure at UID 1000 on a root:root PVC). Fix: fsGroup: 0 makes the kubelet group-own the mounted volume to GID 0, mark it group-writable+SGID, and add GID 0 as a supplemental group; runAsGroup: 0 + runAsUser: 1000 pin the image's own crafty:root identity. Longhorn's CSIDriver fsGroupPolicy: ReadWriteOnceWithFSType applies fsGroup to this RWO filesystem volume.

HUMAN runtime check: confirm the pod writes its SQLite DB and can start a server. If the image's crafty UID is ever not 1000, adjust runAsUser (the launcher's non-root path runs directly as the current UID, so any non-root UID with GID 0 works, but 1000 matches getpwuid cleanly).

Auth = Option B (per-app oauth2-proxy reverse-proxy)

Cilium 1.16.5 cannot inject ext_authz into the shared Gateway's machine-generated Envoy config, and native Gateway-API ExternalAuth ships in no released Cilium yet (see ../oauth2-proxy/README.md). So Crafty's UI is gated by a dedicated oauth2-proxy sidecar in the Crafty pod (--reverse-proxy, --upstream=https://127.0.0.1:8443, --ssl-upstream-insecure-skip-verify=true, --allowed-groups=platform-admins, --skip-provider-button, cookie/redirect for crafty.home.keithjackson.dev). The crafty Service targets the sidecar, never Crafty's :8443, so the gate is in the data path (fail-closed: no valid platform-admins session ⇒ no upstream).

Cookie isolation (why the session used to vanish)

This sidecar and the shared forward-auth oauth2-proxy are independent instances with different cookie-secret values (measured in-cluster — neither can decrypt the other's cookie). Both used to emit oauth2-proxy's default cookie names (_oauth2_proxy, _oauth2_proxy_csrf) scoped to .home.keithjackson.dev, so the browser held one cookie pair for both: whichever proxy authenticated last clobbered the other's session, and the loser logged No valid authentication in request, bounced to Keycloak, and — because the Keycloak SSO session was still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents exactly as "I signed in to Keycloak but I'm still not signed in."

Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app): --cookie-name=_oauth2_proxy_crafty and a host-scoped --cookie-domain=crafty.home.keithjackson.dev. --whitelist-domain stays zone-wide — it governs post-login redirect targets, not cookie scope.

Crafty's own local login — this is where setup actually blocks

Crafty has no OIDC, so after oauth2-proxy admits the user they still meet Crafty's own username/password page. That second factor is intended. What was not intended — and what made the service impossible to set up — is that nothing documented where that password comes from.

Crafty generates it once, on first boot only, when the users table is empty (main.py FRESH_INSTALL → DatabaseBuilder.default_settings()), and writes it to /crafty/app/config/default-creds.txt (mode 0600, on the config subPath of the PVC). It is never in git and cannot be recovered from git: a rebuild onto a fresh PVC generates a new random password. Submitting anything else yields POST /api/v2/auth/login/ → 401, which is exactly what a browser sees with no hint as to why.

→ Retrieval, rotation, and lost-password recovery: ../../runbooks/crafty-admin-credential.md.

Two traps that runbook covers:

  • Crafty's set_passwd console command is unreachable in Kubernetes — it calls getpass.getpass() and so needs a TTY, while the container runs main.py -d -i (daemon). Recovery therefore has to go through the DB with Crafty's own argon2 hasher.
  • Crafty's own 3-strikes brute-force cooldown is inert behind this sidecar, verified live. Crafty records failed attempts under get_remote_ip() (which honours X-Forwarded-For — the live tracker key is the chain "192.168.2.132, 10.0.0.123") but checks for a cooldown under self.request.remote_ip, which behind the sidecar is always 127.0.0.1. The two halves key differently, so get_recent_attempts() always reads an empty list and max_login_attempts: 3 never fires. Good news for the operator (repeated guesses cannot lock you out), but it means the only thing actually limiting who can reach the login form is the oauth2-proxy platform-admins gate — worth knowing before anyone treats that cooldown as a control.

Keycloak client

Confidential auth-code client crafty in the homelab realm — redirect https://crafty.home.keithjackson.dev/oauth2/callback, webOrigins the same host, a groups membership mapper (bare names). Provisioned live via kcadm at authoring time (2026-08-28) and captured as the durable GitOps source of truth in ../keycloak/keycloak-realm-import.yaml (secret injected via the CRAFTY_CLIENT_SECRET placeholder).

Secret locations (SOPS/age)

Secret File Key
oauth2-proxy client id/secret + cookie secret crafty-oauth2-proxy-credentials.sops.yaml (this dir) client-id, client-secret, cookie-secret
Keycloak client secret (mirror) ../keycloak/keycloak-realm-clients.sops.yaml crafty-client-secret

client-secret here must equal crafty-client-secret there (cross-namespace mirror — a Secret can't span namespaces). Both are age-encrypted to the repo sops-age recipient and Flux-decryptable (verified: both decrypt and the two values match).

Game-port L4 exposure

The Gateway is L7-HTTPS-only; raw TCP/UDP game ports can't ride it. service-l4.yaml exposes them as two type=LoadBalancer Services (TCP + UDP) sharing the pinned VIP 192.168.1.55 (lbipam.cilium.io/sharing-key: crafty), ARP-announced by the existing default-l2 policy. This path is NOT auth-gated — Minecraft Java (Mojang/Microsoft) and Bedrock (Xbox Live) authenticate players themselves. external-dns publishes crafty-mc.home.keithjackson.dev → 192.168.1.55.

Adding more game ports (when the operator creates servers)

The operator has not created any servers yet; the starter set covers 3 Java + 3 Bedrock slots. To add capacity:

  1. In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP).
  2. Add that port to the matching Service in service-l4.yaml (one { name, port, targetPort, protocol } line — K8s Services enumerate ports, no ranges).
  3. Add it to the game-port ingress rule in networkpolicy.yaml (crafty-allow-ingress).
  4. (Optional) add a containerPort in deployment.yaml for documentation.
  5. Commit → Flux reconciles.

Policies

namespace.yaml sets only gateway-access=shared (platform namespace, like headlamp/homepage), so Kyverno does not auto-generate the app baseline. networkpolicy.yaml ships it explicitly (defense in depth): default-deny (DNS-only egress) + Cilium allows — Gateway→sidecar :4180, world/LAN→game ports, host→probe ports, and egress world (server-jar downloads + the OIDC path to Keycloak via the Gateway VIP). The sidecar→Crafty hop is pod-local (127.0.0.1) and not subject to network policy.

Verification

  • make -C ci gate — green (hand-authored platform manifests; render-golden unaffected).
  • kustomize build of platform/crafty, clusters/base, clusters/dev — clean.

Human tests (post-deploy)

  1. Browser OIDC round-trip: incognito → https://crafty.home.keithjackson.dev → expect a 302 to https://keycloak.home.keithjackson.dev/realms/homelab/... (NOT the Crafty UI). Log in as a platform-admins user (MFA/OTP enforced) → land on Crafty's own login → sign in with Crafty's local admin. A non-platform-admins user → 403 (authenticated but not authorized).
  2. Pod health / fsGroup: kubectl -n crafty get pod Ready 2/2; Crafty writes its DB under /crafty/app/config and /crafty/servers without EACCES.
  3. Server-create + game reach: in the Crafty UI create a Java server bound to 25565; start it; connect a Minecraft client to crafty-mc.home.keithjackson.dev:25565 (or 192.168.1.55:25565) from the LAN and confirm the world loads. (Bedrock: a server on 19132/udp reached at the VIP.)

Runtime state: the servers themselves are NOT in git

As of 2026-09-14 one Java server exists — "homelab", Paper 26.2, TCP 25565, 2 GiB heap, autostart on, UUID 13c7b85d-00bb-4a58-8c86-2433ca47891c. It was created through Crafty's v2 API, and Mojang's EULA (https://aka.ms/MinecraftEULA) was accepted on the operator's behalf to let it start; see the DECISIONS-LOG entry of that date.

Everything in this directory describes only the capacity to run Minecraft servers. The servers are application state on the crafty-data Longhorn PVC, in two places, and Flux cannot recreate either:

  • /crafty/app/config/db/ (config subPath) — Crafty's SQLite row: name, port, jar, launch command, heap sizes, auto_start / crash_detection.
  • /crafty/servers/<uuid>/ (servers subPath) — paper.jar, eula.txt, server.properties, and the generated world. The jar is re-downloadable; the world is not.

So a rebuild from git alone yields a healthy but empty Crafty with nothing listening on 25565. Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH auto_start to true — see the next note. The world starts over. Treat this PVC as backup-relevant.

Trap: the create API silently ignores autostart

POST /api/v2/servers accepts and validates autostart and crashdetection, but never stores them — the persisted columns are auto_start and crash_detection. After a create that asked for true, both read back false. Set them afterwards with PATCH /api/v2/servers/<uuid> and read them back to confirm, or the server will not come up after a pod restart.

Reference in new issue View Git Blame Copy Permalink