home-network-k3s-impl/platform/crafty/README.md

platform/crafty

Crafty Controller 4 — a web panel that creates and manages multiple Minecraft servers (Java + Bedrock) as subprocesses inside its own container. Its web UI is Keycloak-gated via a per-pod oauth2-proxy reverse-proxy sidecar (the decided Option B, see ../oauth2-proxy/README.md). Game ports are exposed un-gated on a Cilium L2 LoadBalancer (game protocols authenticate players themselves).

• Owner: WS-APP (auth wiring shared with WS-SEC)
• Flux tier: own non-gating clusters/base/crafty.yaml (dependsOn platform-config, wait:false)
• Namespace: crafty (PLATFORM namespace — gateway-access=shared, NOT app-namespace=true)

Deploy summary

Thing	Value
Image	registry.gitlab.com/crafty-controller/crafty-4:4.10.8 (current stable; bare-semver tags, no v)
Web UI	HTTPS self-signed on :8443 (reached only via the sidecar / same-pod)
Sidecar	quay.io/oauth2-proxy/oauth2-proxy:v7.8.1, reverse-proxy on :4180
Data	ONE Longhorn RWO PVC crafty-data (20Gi), mounted 5× by subPath
PVC subPaths	servers→/crafty/servers, backups→/crafty/backups, logs→/crafty/logs, import→/crafty/import, config→/crafty/app/config
Identity	runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true (see below)
Web host	crafty.home.keithjackson.dev → shared Gateway → crafty Service :80 → sidecar :4180 → Crafty :8443
Game VIP	192.168.1.55 (Cilium L2 LoadBalancer; .53=Gateway, .54=unifi, .55=crafty)
Game ports	starter set: TCP 25565-25567 (Java), UDP 19132-19134 (Bedrock)

fsGroup / user decision (the seerr lesson, applied)

Crafty's Dockerfile does useradd -g root -M crafty (UID 1000, primary group root/GID 0), chown -R crafty:root /crafty, and makes /crafty group-writable — the OpenShift/K8s "arbitrary-UID, root-group-perms" pattern. A fresh Longhorn PVC is root:root and not group-writable, so the non-root crafty process could not create /crafty/servers/* etc. and would CrashLoop (exactly seerr's failure at UID 1000 on a root:root PVC). Fix: fsGroup: 0 makes the kubelet group-own the mounted volume to GID 0, mark it group-writable+SGID, and add GID 0 as a supplemental group; runAsGroup: 0 + runAsUser: 1000 pin the image's own crafty:root identity. Longhorn's CSIDriver fsGroupPolicy: ReadWriteOnceWithFSType applies fsGroup to this RWO filesystem volume.

HUMAN runtime check: confirm the pod writes its SQLite DB and can start a server. If the image's crafty UID is ever not 1000, adjust runAsUser (the launcher's non-root path runs directly as the current UID, so any non-root UID with GID 0 works, but 1000 matches getpwuid cleanly).

Auth = Option B (per-app oauth2-proxy reverse-proxy)

Cilium 1.16.5 cannot inject ext_authz into the shared Gateway's machine-generated Envoy config, and native Gateway-API ExternalAuth ships in no released Cilium yet (see ../oauth2-proxy/README.md). So Crafty's UI is gated by a dedicated oauth2-proxy sidecar in the Crafty pod (--reverse-proxy, --upstream=https://127.0.0.1:8443, --ssl-upstream-insecure-skip-verify=true, --allowed-groups=platform-admins, --skip-provider-button, cookie/redirect for crafty.home.keithjackson.dev). The crafty Service targets the sidecar, never Crafty's :8443, so the gate is in the data path (fail-closed: no valid platform-admins session ⇒ no upstream).

Note: Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs in with Crafty's local admin credentials (set on first boot). This is expected, not a bug.

Keycloak client

Confidential auth-code client crafty in the homelab realm — redirect https://crafty.home.keithjackson.dev/oauth2/callback, webOrigins the same host, a groups membership mapper (bare names). Provisioned live via kcadm at authoring time (2026-08-28) and captured as the durable GitOps source of truth in ../keycloak/keycloak-realm-import.yaml (secret injected via the CRAFTY_CLIENT_SECRET placeholder).

Secret locations (SOPS/age)

Secret	File	Key
oauth2-proxy client id/secret + cookie secret	crafty-oauth2-proxy-credentials.sops.yaml (this dir)	client-id, client-secret, cookie-secret
Keycloak client secret (mirror)	../keycloak/keycloak-realm-clients.sops.yaml	crafty-client-secret

client-secret here must equal crafty-client-secret there (cross-namespace mirror — a Secret can't span namespaces). Both are age-encrypted to the repo sops-age recipient and Flux-decryptable (verified: both decrypt and the two values match).

Game-port L4 exposure

The Gateway is L7-HTTPS-only; raw TCP/UDP game ports can't ride it. service-l4.yaml exposes them as two type=LoadBalancer Services (TCP + UDP) sharing the pinned VIP 192.168.1.55 (lbipam.cilium.io/sharing-key: crafty), ARP-announced by the existing default-l2 policy. This path is NOT auth-gated — Minecraft Java (Mojang/Microsoft) and Bedrock (Xbox Live) authenticate players themselves. external-dns publishes crafty-mc.home.keithjackson.dev → 192.168.1.55.

Adding more game ports (when the operator creates servers)

As of 2026-09-29 two servers exist (see Runtime state below): Java homelab on 25565 and Bedrock K3s-Bedrock on 19132. The starter set covers 3 Java + 3 Bedrock slots. To add capacity:

1. In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP).
2. Add that port to the matching Service in service-l4.yaml (one { name, port, targetPort, protocol } line — K8s Services enumerate ports, no ranges).
3. Add it to the game-port ingress rule in networkpolicy.yaml (crafty-allow-ingress).
4. (Optional) add a containerPort in deployment.yaml for documentation.
5. Commit → Flux reconciles.

Bedrock on UDP 19132 — the transport trap (MEASURED on prod 2026-09-29)

A Bedrock server that Crafty created, started, and reports as running can still have nothing bound to 19132/udp, with every layer in this repo provably correct. This is the first thing to check when "Bedrock players cannot connect" — before the Service, the policy, the lease or the VIP.

Measured on prod 2026-09-29 against server K3s-Bedrock (c5837301-12b1-404e-b50e-70eafb37226b, DB type=minecraft-bedrock, server_port=19132, process ./bedrock_server alive):

• /proc/net/udp and /proc/net/udp6 inside the crafty container held no socket on 19132 (0x4ABC) at all — only ephemeral UDP sockets. Port 19132 was bound as TCP (/proc/net/tcp6, …:4ABC, state 0A/LISTEN), not UDP.
• Cause, from the server's own server.properties: transport=nethernet — the value Microsoft's BDS archive ships by default. Its own inline docs: # Allowed values: "raknet" or "nethernet", and against server-portv6: # This property is ignored when transport=nethernet and a dual-stack socket will be opened on server-port instead. NetherNet is the WebRTC/Xbox-signalled transport; it never opens a RakNet UDP listener, so nothing binds 19132/udp however server-port is set.
• Crafty does not manage this property — transport does not appear anywhere under /crafty/app/classes/ — and Crafty's only Bedrock reachability probe is raknet_ping.py, so the panel spams WARNING app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping to 127.0.0.1:19132 every 5–30 s against a server it started successfully. That warning is the signature of this trap, and it comes from 127.0.0.1 — i.e. it indicts the server, not the LAN.
• The platform path was verified sound at the same time: crafty-l4-udp live as externalTrafficPolicy: Cluster with status.loadBalancer.ingress[0].ip = the VIP, a non-empty EndpointSlice, cilium-dbg service list showing <VIP>:19132/UDP → 10.0.0.130:19132/UDP (active) on every node, and the crafty endpoint's datapath policy map carrying Allow Ingress reserved:world 19132/UDP with 26 packets / 27660 bytes already counted through it. Real player packets reached the pod and were allowed; the application had no socket to take them.

Operator fix — application state on the PVC, not git and not Flux: stop the server in Crafty, set transport=raknet in /crafty/servers/<uuid>/server.properties, start it, and confirm a UDP listener on 19132 exists (grep 4ABC /proc/net/udp /proc/net/udp6, or the raknet_ping warnings stopping) before suspecting the network. (That raknet restores the UDP bind is read from the shipped property documentation, not yet observed on this cluster — see the DECISIONS-LOG entry of 2026-09-29.)

Bedrock realities that are NOT platform defects but present identically

• LAN / Friends-tab discovery can never work here. Bedrock discovery is a UDP broadcast on 19132 inside one L2 segment; this server sits behind a LoadBalancer VIP (and SNAT), not on the LAN, so it will never appear in a client's Friends/LAN list. Players must use Add Server and type the address and port by hand: crafty-mc.home.keithjackson.dev (or the VIP) with port 19132. external-dns publishes that name from the -tcp Service, so it resolves to the shared VIP and is correct for Bedrock too — only the port differs from Java's 25565.
• Consoles cannot reach this at all. Switch/Xbox/PlayStation Bedrock clients have no arbitrary-server-address field; only Windows/mobile/Fire do. Reaching it from a console needs a DNS redirect of a featured-server hostname, which is out of scope for service-l4.yaml.
• enable-lan-visibility=true (the shipped default) collides with Bedrock slots 1 and 2. Under transport=raknet it makes a server bind the default ports 19132/19133 even when server-port is set to something else — the shipped comment says exactly that. So a second Bedrock server put on 19133 or 19134 will fight the first for 19132. Set enable-lan-visibility=false on every Bedrock server after the first; LAN discovery cannot work here anyway (above).
• server-portv6 is not exposed and is not meant to be. service-l4.yaml is IPv4 single-stack, so the default server-portv6=19133 is unreachable from the LAN. Harmless — but do not read 19133 in server.properties as "slot 1 is in use".

Policies

namespace.yaml sets only gateway-access=shared (platform namespace, like headlamp/homepage), so Kyverno does not auto-generate the app baseline. networkpolicy.yaml ships it explicitly (defense in depth): default-deny (DNS-only egress) + Cilium allows — Gateway→sidecar :4180, world/LAN→game ports, host→probe ports, and egress world (server-jar downloads + the OIDC path to Keycloak via the Gateway VIP). The sidecar→Crafty hop is pod-local (127.0.0.1) and not subject to network policy.

Verification

• make -C ci gate — green (hand-authored platform manifests; render-golden unaffected).
• kustomize build of platform/crafty, clusters/base, clusters/dev — clean.

Human tests (post-deploy)

1. Browser OIDC round-trip: incognito → https://crafty.home.keithjackson.dev → expect a 302 to https://keycloak.home.keithjackson.dev/realms/homelab/... (NOT the Crafty UI). Log in as a platform-admins user (MFA/OTP enforced) → land on Crafty's own login → sign in with Crafty's local admin. A non-platform-admins user → 403 (authenticated but not authorized).
2. Pod health / fsGroup: kubectl -n crafty get pod Ready 2/2; Crafty writes its DB under /crafty/app/config and /crafty/servers without EACCES.
3. Server-create + game reach (Java): in the Crafty UI create a Java server bound to 25565; start it; connect a Minecraft client to crafty-mc.home.keithjackson.dev:25565 (or 192.168.1.55:25565) from the LAN and confirm the world loads.
4. Server-create + game reach (Bedrock): creating and starting the server is not sufficient — check transport=raknet and a live UDP listener on 19132 first (see the transport trap above), then Add Server by hand in a Windows/mobile client at crafty-mc.home.keithjackson.dev port 19132. It will never show up in the LAN/Friends tab, and a console cannot reach it.

Runtime state: the servers themselves are NOT in git

As of 2026-09-29 two servers exist (both measured live, select * from servers):

Name	Type	Port	UUID	auto_start
homelab	minecraft-java (Paper 26.2, 2 GiB heap)	TCP 25565	13c7b85d-00bb-4a58-8c86-2433ca47891c	1
K3s-Bedrock	minecraft-bedrock	UDP 19132	c5837301-12b1-404e-b50e-70eafb37226b	0 ⚠

K3s-Bedrock was created 2026-09-29 20:51 and has auto_start=0 and crash_detection=0 — the create-API trap documented at the bottom of this file, live again. It will not come back after a pod restart, and merging any change to deployment.yaml restarts the pod (strategy: Recreate). PATCH both to true and read them back.

homelab was created through Crafty's v2 API, and Mojang's EULA (https://aka.ms/MinecraftEULA) was accepted on the operator's behalf to let it start; see the DECISIONS-LOG entry of that date.

Everything in this directory describes only the capacity to run Minecraft servers. The servers are application state on the crafty-data Longhorn PVC, in two places, and Flux cannot recreate either:

• /crafty/app/config/db/ (config subPath) — Crafty's SQLite row: name, port, jar, launch command, heap sizes, auto_start / crash_detection.
• /crafty/servers/<uuid>/ (servers subPath) — paper.jar, eula.txt, server.properties, and the generated world. The jar is re-downloadable; the world is not.

So a rebuild from git alone yields a healthy but empty Crafty with nothing listening on 25565. Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH auto_start to true — see the next note. The world starts over. Treat this PVC as backup-relevant.

Trap: the create API silently ignores autostart

POST /api/v2/servers accepts and validates autostart and crashdetection, but never stores them — the persisted columns are auto_start and crash_detection. After a create that asked for true, both read back false. Set them afterwards with PATCH /api/v2/servers/<uuid> and read them back to confirm, or the server will not come up after a pod restart.