home-network-k3s-impl/platform/crafty

platform/crafty

Crafty Controller 4 — a web panel that creates and manages multiple Minecraft servers (Java + Bedrock) as subprocesses inside its own container. Its web UI is Keycloak-gated via a per-pod oauth2-proxy reverse-proxy sidecar (the decided Option B, see ../oauth2-proxy/README.md). Game ports are exposed un-gated on a Cilium L2 LoadBalancer (game protocols authenticate players themselves).

• Owner: WS-APP (auth wiring shared with WS-SEC)
• Flux tier: own non-gating clusters/base/crafty.yaml (dependsOn platform-config, wait:false)
• Namespace: crafty (PLATFORM namespace — gateway-access=shared, NOT app-namespace=true)

Deploy summary

Thing	Value
Image	registry.gitlab.com/crafty-controller/crafty-4:4.10.8 (current stable; bare-semver tags, no v)
Web UI	HTTPS self-signed on :8443 (reached only via the sidecar / same-pod)
Sidecar	quay.io/oauth2-proxy/oauth2-proxy:v7.15.6, reverse-proxy on :4180
Data	ONE Longhorn RWO PVC crafty-data (20Gi), mounted 5× by subPath
PVC subPaths	servers→/crafty/servers, backups→/crafty/backups, logs→/crafty/logs, import→/crafty/import, config→/crafty/app/config
Identity	runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true (see below)
Web host	crafty.home.keithjackson.dev → shared Gateway → crafty Service :80 → sidecar :4180 → Crafty :8443
Game VIP	${CRAFTY_L4_VIP} (Cilium L2 LoadBalancer) — prod 192.168.2.22 (live EXTERNAL-IP, measured 2026-09-29); base/dev 192.168.1.55
Game ports	starter set: TCP 25565-25567 (Java), UDP 19132-19134 (Bedrock)

fsGroup / user decision (the seerr lesson, applied)

Crafty's Dockerfile does useradd -g root -M crafty (UID 1000, primary group root/GID 0), chown -R crafty:root /crafty, and makes /crafty group-writable — the OpenShift/K8s "arbitrary-UID, root-group-perms" pattern. A fresh Longhorn PVC is root:root and not group-writable, so the non-root crafty process could not create /crafty/servers/* etc. and would CrashLoop (exactly seerr's failure at UID 1000 on a root:root PVC). Fix: fsGroup: 0 makes the kubelet group-own the mounted volume to GID 0, mark it group-writable+SGID, and add GID 0 as a supplemental group; runAsGroup: 0 + runAsUser: 1000 pin the image's own crafty:root identity. Longhorn's CSIDriver fsGroupPolicy: ReadWriteOnceWithFSType applies fsGroup to this RWO filesystem volume.

HUMAN runtime check: confirm the pod writes its SQLite DB and can start a server. If the image's crafty UID is ever not 1000, adjust runAsUser (the launcher's non-root path runs directly as the current UID, so any non-root UID with GID 0 works, but 1000 matches getpwuid cleanly).

Auth = Option B (per-app oauth2-proxy reverse-proxy)

Cilium 1.16.5 cannot inject ext_authz into the shared Gateway's machine-generated Envoy config, and native Gateway-API ExternalAuth ships in no released Cilium yet (see ../oauth2-proxy/README.md). So Crafty's UI is gated by a dedicated oauth2-proxy sidecar in the Crafty pod (--reverse-proxy, --upstream=https://127.0.0.1:8443, --ssl-upstream-insecure-skip-verify=true, --allowed-groups=platform-admins, --skip-provider-button, cookie/redirect for crafty.home.keithjackson.dev). The crafty Service targets the sidecar, never Crafty's :8443, so the gate is in the data path (fail-closed: no valid platform-admins session ⇒ no upstream).

Cookie isolation (why the session used to vanish)

This sidecar and the shared forward-auth oauth2-proxy are independent instances with different cookie-secret values (measured in-cluster — neither can decrypt the other's cookie). Both used to emit oauth2-proxy's default cookie names (_oauth2_proxy, _oauth2_proxy_csrf) scoped to .home.keithjackson.dev, so the browser held one cookie pair for both: whichever proxy authenticated last clobbered the other's session, and the loser logged No valid authentication in request, bounced to Keycloak, and — because the Keycloak SSO session was still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents exactly as "I signed in to Keycloak but I'm still not signed in."

Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app): --cookie-name=_oauth2_proxy_crafty and a host-scoped --cookie-domain=crafty.home.keithjackson.dev. --whitelist-domain stays zone-wide — it governs post-login redirect targets, not cookie scope.

Crafty's own local login — this is where setup actually blocks

Crafty has no OIDC, so after oauth2-proxy admits the user they still meet Crafty's own username/password page. That second factor is intended. What was not intended — and what made the service impossible to set up — is that nothing documented where that password comes from.

Crafty generates it once, on first boot only, when the users table is empty (main.py FRESH_INSTALL → DatabaseBuilder.default_settings()), and writes it to /crafty/app/config/default-creds.txt (mode 0600, on the config subPath of the PVC). It is never in git and cannot be recovered from git: a rebuild onto a fresh PVC generates a new random password. Submitting anything else yields POST /api/v2/auth/login/ → 401, which is exactly what a browser sees with no hint as to why.

→ Retrieval, rotation, and lost-password recovery: ../../runbooks/crafty-admin-credential.md.

Two traps that runbook covers:

• Crafty's set_passwd console command is unreachable in Kubernetes — it calls getpass.getpass() and so needs a TTY, while the container runs main.py -d -i (daemon). Recovery therefore has to go through the DB with Crafty's own argon2 hasher.
• Crafty's own 3-strikes brute-force cooldown is inert behind this sidecar, verified live. Crafty records failed attempts under get_remote_ip() (which honours X-Forwarded-For — the live tracker key is the chain "192.168.2.132, 10.0.0.123") but checks for a cooldown under self.request.remote_ip, which behind the sidecar is always 127.0.0.1. The two halves key differently, so get_recent_attempts() always reads an empty list and max_login_attempts: 3 never fires. Good news for the operator (repeated guesses cannot lock you out), but it means the only thing actually limiting who can reach the login form is the oauth2-proxy platform-admins gate — worth knowing before anyone treats that cooldown as a control.

Keycloak client

Confidential auth-code client crafty in the homelab realm — redirect https://crafty.home.keithjackson.dev/oauth2/callback, webOrigins the same host, a groups membership mapper (bare names). Provisioned live via kcadm at authoring time (2026-08-28) and captured as the durable GitOps source of truth in ../keycloak/keycloak-realm-import.yaml (secret injected via the CRAFTY_CLIENT_SECRET placeholder).

Secret locations (SOPS/age)

Secret	File	Key
oauth2-proxy client id/secret + cookie secret	crafty-oauth2-proxy-credentials.sops.yaml (this dir)	client-id, client-secret, cookie-secret
Keycloak client secret (mirror)	../keycloak/keycloak-realm-clients.sops.yaml	crafty-client-secret

client-secret here must equal crafty-client-secret there (cross-namespace mirror — a Secret can't span namespaces). Both are age-encrypted to the repo sops-age recipient and Flux-decryptable (verified: both decrypt and the two values match).

Game-port L4 exposure

The Gateway is L7-HTTPS-only; raw TCP/UDP game ports can't ride it. service-l4.yaml exposes them as two type=LoadBalancer Services (TCP + UDP) sharing the pinned VIP ${CRAFTY_L4_VIP} (prod 192.168.2.22) (lbipam.cilium.io/sharing-key: crafty), ARP-announced by the existing default-l2 policy. This path is NOT auth-gated — Minecraft Java (Mojang/Microsoft) and Bedrock (Xbox Live) authenticate players themselves. external-dns publishes crafty-mc.home.keithjackson.dev → ${CRAFTY_L4_VIP}.

Adding more game ports (when the operator creates servers)

As of 2026-09-29 two servers exist (see Runtime state below): Java homelab on 25565 and Bedrock K3s-Bedrock on 19132. The starter set covers 3 Java + 3 Bedrock slots. To add capacity:

1. In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP).
2. Add that port to the matching Service in service-l4.yaml (one { name, port, targetPort, protocol } line — K8s Services enumerate ports, no ranges).
3. Add it to the game-port ingress rule in networkpolicy.yaml (crafty-allow-ingress).
4. (Optional) add a containerPort in deployment.yaml for documentation.
5. Commit → Flux reconciles.

Bedrock on UDP 19132 — the transport trap, and the ERROR banner that is NOT a refusal (MEASURED on prod 2026-09-29)

Two traps in sequence. The first hides the UDP socket; the second frightens you into undoing the fix. As of 2026-09-29 21:34 this server is reachable on 19132/udp and answering, measured from off-cluster. Read both traps before changing anything.

Trap 1 — transport=nethernet (the shipped default) opens no RakNet UDP listener

Measured 20:54–21:32 against K3s-Bedrock (c5837301-12b1-404e-b50e-70eafb37226b, DB type=minecraft-bedrock, server_port=19132, process ./bedrock_server alive):

• /proc/net/udp and /proc/net/udp6 held no socket on 19132 (0x4ABC) at all. Port 19132 was bound as TCP (/proc/net/tcp6, …:4ABC, state 0A/LISTEN), not UDP.
• That is nethernet working as designed, and the shipped bedrock_server_how_to.html — the vendor's own doc, sitting on the PVC beside the binary — says exactly why: nethernet "listens on a single dual-stack TCP socket on server-port for an HTTP-based signaling handshake, then negotiates a UDP connection with each client that carries the gameplay traffic."
• So nethernet could never have worked here anyway, on two independent counts: service-l4.yaml exposes 19132 only in the UDP Service (crafty-l4-tcp carries 25565-25567 and nothing else), and crafty-allow-ingress allows 19132 UDP and not 19132 TCP. The signaling port was unreachable at both the Service and the policy layer. The server's own packet-statistics.txt for that run agrees: Total Received: 0, 0B over 2293 s.
• Crafty does not manage this property (transport appears nowhere under /crafty/app/classes/) and its only Bedrock probe is raknet_ping.py, so the panel logged WARNING app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping to 127.0.0.1:19132 every ~5 s — 155 such lines — against a server it had started successfully.

Trap 2 — under transport=raknet this build prints a scary ERROR banner and then works anyway

The operator set transport=raknet at 21:33:31 (/crafty/logs/audit.log: Edited file /crafty/servers/c5837301-…/server.properties) and started the server at 21:33:58. BDS 1.26.52 printed, at ERROR level:

[ERROR] ================ TRANSPORT TYPE ERROR  ===================
[ERROR] Your current connection type is not set to NetherNet. In this release, NetherNet is the only
        supported transport type.
[ERROR] Players will not be able to connect to your game without NetherNet.
[ERROR] To switch, set 'transport=nethernet' in server.properties.

That banner is advisory, not fatal. Do not revert transport on seeing it. Every claim it makes about connectivity was false on this cluster within five seconds of it being printed:

• The process kept running (./bedrock_server, pid 47006) and bound the RakNet sockets: /proc/net/udp 00000000:4ABC (19132, IPv4) and /proc/net/udp6 …:4ABD (19133, IPv6), both owned by bedrock_server — /proc/47006/fd/25 and fd/26 → those socket inodes. The earlier TCP bind on 4ABC is gone.

• Crafty's warning changed kind at that instant: last Got timeout at 21:34:04,306, first Bedrock ping returned a string with too few fields at 21:34:09,854. That distinction is load-bearing — read raknet_ping.py: Got timeout is emitted only when sock.recv raises TimeoutError (nothing came back at all), whereas too few fields is reachable only after a 0x1C Unconnected Pong has arrived and its magic has validated. The server is answering pings.

• End-to-end, from off-cluster: a RakNet Unconnected Ping sent from the LAN to the prod VIP 192.168.2.22:19132 returns a 139-byte 0x1c Unconnected Pong in ~12 ms, magic valid:

  MCPE;Dedicated Server;2193;1.26.52;0;10;11676629557768398479;Bedrock level;Survival;1;19132;19133;0;0;0;
  

  Edition MCPE, protocol 2193, version 1.26.52, 0/10 players. Players can reach this server now, through the VIP, the Service, and the CiliumNetworkPolicy, with no manifest change.

• The vendor doc on disk contradicts the banner and matches the observed behaviour. transport is documented as "Allowed values: raknet, nethernet", and: "raknet — The traditional Bedrock UDP transport. The server listens directly on server-port (IPv4) and server-portv6 (IPv6), and clients connect to that address and port." RakNet is also still fully compiled into the binary — strings bedrock_server yields RakNetConnector, createUniqueRakPeer, RakPeerInterface and the vendored build path …/src-deps/raknet/raknet/RakNetSocket2.cpp.

So "NetherNet is the only supported transport type" is a support statement, not a capability statement: Microsoft declines to support RakNet and ships it working. Treat the banner as a deprecation notice with an unknown expiry — a future BDS may genuinely remove RakNet, and that is the day this exposure needs GeyserMC (see If RakNet is ever really removed below).

The shipped-comment trap, stated plainly. server.properties lists raknet as allowed and the binary logs an ERROR saying it is not supported. Both are the vendor, and they disagree. The tiebreak is neither prose: it is /proc/net/udp and a pong on the wire. Whenever this file says a Bedrock server is or is not reachable, it is because someone pinged it — not because a comment said so.

The steady-state too few fields warning is benign and is now the expected signature

Crafty will log Bedrock ping returned a string with too few fields - missing values [None] every ~5 s forever. That means the ping SUCCEEDED. BDS 1.26.52's pong carries 16 ;-separated fields; Crafty 4.10.8's pingpong_string_fields table knows 14 and expects the last to be an empty terminator. It instead finds '0', so last_enumeration stops at index 12 and the parser reports the one None terminator entry as missing. Every field that matters (edition, MOTD, protocol, version name, player count, max players, uuid, game mode, both ports) parsed correctly. Do not chase this warning. The failure signature is the other message, Got timeout.

"You are not invited to play on this server" — the join gate is the allowlist, not the network (MEASURED 2026-09-29 21:56; RESOLVED and confirmed 22:03)

A Bedrock client reported "Unable to connect / You are not invited to play on this server". That is an authorization refusal, and it is the expected, guaranteed behaviour of this server's current configuration. Measured:

Property / file	Value	Source
allow-list	true (line 37)	server.properties
allowlist.json	[] — 3 bytes, mtime 20:51, never edited since server create	cat allowlist.json
online-mode	true (line 31)	server.properties
transport	raknet (line 71), file mtime still 21:33:31	server.properties
TCP 19132 listener	none (NO-TCP-4ABC)	grep 4ABC /proc/net/tcp /proc/net/tcp6
UDP 19132 listener	present, 00000000:4ABC, pid 47006	grep 4ABC /proc/net/udp

allow-list=true with an empty allowlist.json refuses every player, unconditionally. The shipped bedrock_server_how_to.html states it: "If the allow-list property is enabled in server.properties then the server will only allow selected users to connect. To allow a user to connect you need to know their Xbox Live Gamertag." An empty list selects nobody. This is not a network problem, not a transport problem, and not a platform problem — and no amount of Service or policy work will change it.

CONFIRMED: one gamertag added, and a player joined

This is no longer a hypothesis. The fix was applied and it worked. (The join itself is operator-reported — 2026-09-29, the operator added the player's gamertag and the Bedrock client connected and joined. I did not watch it.) Two independent things corroborate it directly on disk:

• allowlist.json went from [] to one entry, and that entry's xuid is populated (keys=['ignoresPlayerLimit','name','xuid'], file 75 bytes, mtime 22:03:17). Per the vendor doc, xuid is "Optional. If it's not set then it will be populated when someone with a matching name connects." The operator wrote only the name — audit.log records Edited file …/allowlist.json at 22:02:13 — so the XUID was written by the server, on connect, at 22:03:17. That is server-authored evidence of an Xbox-Live-authenticated join. (Observed: the file state and both timestamps. Reasoned from the vendor doc: that a populated XUID implies a connect.)
• The RakNet path still answers after the restart this involved: a pong from 192.168.2.22:19132 at 22:06, MCPE;…;1.26.52;0;10;… (players=0 — the player had left by then), new server_guid matching the new pid.

So the causal chain is closed: allow-list=true + allowlist.json == [] was the wall, and nothing else was. The earlier mapping of the client's "You are not invited" string to the empty allowlist was reasoned from the vendor doc (there is no BDS log on disk to read it from); adding exactly one gamertag and getting exactly one successful join confirms that mapping. No transport change, no manifest change, and no network change was involved in the fix.

Note the route the operator actually used: they edited allowlist.json directly and then restarted the server (audit.log: edit 22:02:13 → stop_server 22:02:37 → start_server 22:02:41; pid 47006 → 47875). That is route 2 below, and a restart is a valid substitute for allowlist reload. Route 1 (allowlist add in the console) avoids the restart and is still preferred.

This is also positive evidence that the RakNet path works end to end. A refusal authored by our server required the client to reach our server and get an answer back. The only listener on 19132 is the RakNet UDP socket — there is no TCP 19132 listener at all, and NetherNet requires a TCP signalling socket on server-port. So the client cannot have used NetherNet against this server; it arrived over 19132/udp, through crafty-l4-udp and the VIP. The 19132-19134/UDP exposure is not inert — it is the path that carried both the pong and this refusal. (Observed: the listeners, the absent TCP socket, transport=raknet. Reasoned from those: that the refusal traversed that path. Not captured: the actual player flow — see the caveat below.)

Caveat: BDS console output is NOT on disk, so a join attempt cannot be correlated from the filesystem

K3s-Bedrock's log_path in crafty.sqlite is the empty string, and Dedicated_Server.txt is 0 bytes. Crafty does not persist a Bedrock server's stdout anywhere on the PVC — the console exists only in the panel UI. commander.log between 21:39 and 21:56 contains nothing but homelab crash-detection lines and the raknet_ping chatter: no join, no gamertag, no XUID, no allowlist denial. That absence is not evidence about where the refusal came from; it is evidence that Crafty logs no BDS stdout at all. To see a rejected join you must watch the panel console live. UNVERIFIED, and only observable in the UI.

How a Bedrock player actually gets authorized here

Identity is an Xbox Live Gamertag (online-mode=true forces XBL auth, which is what gives BDS a gamertag/XUID to check). allowlist.json is a JSON array; per the vendor doc the keys are name (the gamertag), optional xuid ("populated when someone with a matching name connects"), and ignoresPlayerLimit. So the operator adds one entry per player, keyed on gamertag — there is no group, role, or Keycloak integration, and none of this touches the oauth2-proxy/Keycloak gate, which protects only the web panel.

Three ways to do it, all operator actions. Routes 1 and 2 are both proven — route 2 is the one that produced the confirmed join above:

1. Preferred — the panel console. In Crafty, open K3s-Bedrock → Console and send: allowlist add <Gamertag> (quote it if it contains a space: allowlist add "Example Name"). This writes allowlist.json and takes effect immediately, with no restart.

2. Edit the file, then reload. Write /crafty/servers/c5837301-…/allowlist.json, then send allowlist reload in the console — "After you've modified the file you need to run the command allowlist reload to make sure that the server knows about your new change." Shape:

   [
     { "ignoresPlayerLimit": false, "name": "FirstPlayerGamertag" },
     { "ignoresPlayerLimit": false, "name": "SecondPlayerGamertag" }
   ]
   

3. Turn the allowlist off entirely — allow-list=false in server.properties (restart), or allowlist off in the console. Note the vendor's warning: allowlist on|off "does not change the value in the server.properties file", so a console off silently reverts on the next start. Not recommended: the game port is deliberately un-gated at the network layer (see Game-port L4 exposure), so the allowlist is the only thing deciding who may join. Keep it true and enumerate players.

allowlist.json is application state on the PVC, exactly like transport=raknet. Neither is in git and Flux can restore neither. Be precise about what breaks them, because the two cases differ:

• A pod or server restart preserves both — observed: the 22:02:48 restart (pid 47006 → 47875) kept transport=raknet (server.properties mtime unchanged at 21:33:31) and kept the allowlist entry. What a pod restart does break is that K3s-Bedrock has auto_start=0, so the Bedrock server simply does not come back until someone starts it by hand.
• A server recreate or reinstall silently resets both — a fresh server.properties ships transport=nethernet (nothing bound on 19132/udp) and a fresh allowlist.json is [] (everyone refused). Neither change announces itself, and the server will look healthy and started in the panel while being unreachable and unjoinable.

Treat both files as backup-relevant, and re-run the 5-step connect procedure above after any Bedrock recreate or BDS update.

Why this does not change the transport recommendation

transport=raknet is correct, and with the allowlist entry in place the combination is proven end-to-end by a player who has joined — not merely the best-reasoned option. The allowlist refusal was orthogonal to transport all along: it would have refused players identically under NetherNet. Switching to NetherNet would add work (TCP 19132 in crafty-l4-tcp and in crafty-allow-ingress, plus server-udp-ports pinned and exposed) and would admit not one extra player. The ranking below stands, and option 1 is now settled rather than provisional.

Decision (PROVEN, not provisional): transport=raknet stays; the 19132-19134/UDP exposure is correct and live

No manifest change is needed or wanted. service-l4.yaml, networkpolicy.yaml and the containerPorts were already shaped for exactly RakNet, and that shape is now proven by a player who has connected and joined over 192.168.2.22:19132 — not assumed, and not merely reasoned. The options that would have required manifest work are all not recommended:

Option	Manifest consequence	Verdict
Keep transport=raknet + allowlist entries (this is what works)	none — the existing Service, policy and ports were already correct	Done; proven. A player has joined over the VIP. Zero manifest change, vanilla parity, real Bedrock world.
GeyserMC + Floodgate on the homelab Paper server	none either (Geyser also binds 19132/udp), but it would collide with the live BDS on 19132 — one of them must move	Hold in reserve. Only worth it if RakNet is really removed, or if you want one shared world. Cost: Bedrock players join the Java world, not a Bedrock world; Java-only mechanics and some content differ; it is another moving part to patch.
Pin/downgrade BDS to a pre-nethernet release	none	Unnecessary — the current build already speaks RakNet. Also fragile: Microsoft removes old BDS downloads, and Crafty's Bedrock installer tracks latest. UNVERIFIED whether Crafty can pin a version at all.
A third-party server (PocketMine-MP, Nukkit)	none	No. Loses vanilla parity and the world format; a large change to solve a problem that is not occurring.
Accept NetherNet	19132-19134/UDP becomes inert; nethernet would instead need TCP 19132 added to crafty-l4-tcp and to crafty-allow-ingress, plus server-udp-ports pinned and exposed for the negotiated media path	No — and note it is not currently in use: transport=raknet and there is no TCP 19132 listener. Strictly more manifest work than the option that already works, it admits no extra player (the allowlist is the gate), and WebRTC media behind a shared VIP and SNAT is unproven here.

If RakNet is ever really removed

Then BDS opens no UDP listener and this exposure has nothing to bind to — an architecture problem, not a config one. The migration is GeyserMC + Floodgate as a plugin on the existing homelab Paper server, because Geyser speaks RakNet on 19132/udp itself: the VIP, the Service, the policy and the containerPorts all stay byte-for-byte as they are, and Bedrock clients keep connecting to crafty-mc.home.keithjackson.dev:19132. What changes is the world: Bedrock players land in the Java world via Floodgate (no Xbox-authenticated Bedrock account needed on the Java side), the Bedrock server is retired, and K3s-Bedrock stops the day Geyser starts — they cannot share 19132. Installing it is a PVC write plus a Paper restart, i.e. an operator action: drop Geyser-Spigot.jar and floodgate-spigot.jar into /crafty/servers/13c7b85d-…/plugins/ and restart homelab, then set Geyser's bedrock.port: 19132 and clone-remote-port: false in its generated config.yml. Geyser's own Bedrock-version support is external and unmeasured here — check that it supports the Bedrock client version in use before committing.

Bedrock realities that are NOT platform defects but present identically

• LAN / Friends-tab discovery can never work here. Bedrock discovery is a UDP broadcast on 19132 inside one L2 segment; this server sits behind a LoadBalancer VIP (and SNAT), not on the LAN, so it will never appear in a client's Friends/LAN list. Players must use Add Server and type the address and port by hand: crafty-mc.home.keithjackson.dev (or the VIP) with port 19132. external-dns publishes that name from the -tcp Service, so it resolves to the shared VIP and is correct for Bedrock too — only the port differs from Java's 25565.
• Consoles cannot reach this at all. Switch/Xbox/PlayStation Bedrock clients have no arbitrary-server-address field; only Windows/mobile/Fire do. Reaching it from a console needs a DNS redirect of a featured-server hostname, which is out of scope for service-l4.yaml.
• enable-lan-visibility=true (the shipped default) collides with Bedrock slots 1 and 2 — now OBSERVED, not merely documented. Under transport=raknet it makes a server bind the default ports 19132 and 19133 even when server-port/server-portv6 say otherwise; the shipped bedrock_server_how_to.html says so, and on 2026-09-29 a RakNet ping to 192.168.2.22:19133 returned the same pong, from the same server_guid, as :19132 — one server occupying two of the three Bedrock slots. :19134 correctly timed out. So a second Bedrock server put on 19133 will fight the first. Set enable-lan-visibility=false on every Bedrock server after the first; LAN discovery cannot work here anyway (above).
• server-portv6 is the IPv6 listener and is not exposed; 19133 is nonetheless in use. service-l4.yaml is IPv4 single-stack, so the IPv6 socket …:4ABD that BDS opens for server-portv6=19133 is unreachable from the LAN. But because enable-lan-visibility=true (above) the server also answers on IPv4 19133, which the UDP Service does expose — measured. So 19133 in server.properties genuinely does mean "slot 1 is taken", contrary to what this file previously said.

Policies

namespace.yaml sets only gateway-access=shared (platform namespace, like headlamp/homepage), so Kyverno does not auto-generate the app baseline. networkpolicy.yaml ships it explicitly (defense in depth): default-deny (DNS-only egress) + Cilium allows — Gateway→sidecar :4180, world/LAN→game ports, host→probe ports, and egress world (server-jar downloads + the OIDC path to Keycloak via the Gateway VIP). The sidecar→Crafty hop is pod-local (127.0.0.1) and not subject to network policy.

Verification

• make -C ci gate — green (hand-authored platform manifests; render-golden unaffected).
• kustomize build of platform/crafty, clusters/base, clusters/dev — clean.

Human tests (post-deploy)

1. Browser OIDC round-trip: incognito → https://crafty.home.keithjackson.dev → expect a 302 to https://keycloak.home.keithjackson.dev/realms/homelab/... (NOT the Crafty UI). Log in as a platform-admins user (MFA/OTP enforced) → land on Crafty's own login → sign in with Crafty's local admin. A non-platform-admins user → 403 (authenticated but not authorized).

2. Pod health / fsGroup: kubectl -n crafty get pod Ready 2/2; Crafty writes its DB under /crafty/app/config and /crafty/servers without EACCES.

3. Server-create + game reach (Java): in the Crafty UI create a Java server bound to 25565; start it; connect a Minecraft client to crafty-mc.home.keithjackson.dev:25565 (or 192.168.2.22:25565) from the LAN and confirm the world loads.

4. Bedrock: the connect procedure (VERIFIED end-to-end 2026-09-29 — a player has joined). Creating and starting the server is not sufficient. Do these in order; each step has a distinct failure signature:

   1. transport=raknet in server.properties. The vendor default is nethernet, which opens no UDP listener at all.
   2. A live UDP listener: grep 4ABC /proc/net/udp /proc/net/udp6 → a wildcard bind. Under nethernet you instead see 4ABC on TCP, which nothing here exposes.
   3. A pong from off-cluster: a RakNet Unconnected Ping to 192.168.2.22:19132 returns a 0x1c packet whose MOTD string starts MCPE;. This proves the VIP, Service and policy path.
   4. An allowlist entry for the player's Xbox Gamertag. allow-list=true with allowlist.json == [] refuses every player with "You are not invited to play on this server" while the server is otherwise perfectly reachable. In the panel console: allowlist add <Gamertag>.
   5. Add Server by hand in a Windows/mobile client: crafty-mc.home.keithjackson.dev, port 19132. It will never appear in the LAN/Friends tab, and a console cannot reach it at all.

   Two log messages appear on a fully working server and must be ignored: BDS's ================ TRANSPORT TYPE ERROR =================== banner (advisory — see Trap 2), and Crafty's perpetual Bedrock ping returned a string with too few fields. Neither indicates a fault; chasing them is what cost two sessions. The real failure signatures are Got timeout while issuing bedrock ping (no listener) and "You are not invited" (allowlist).

Runtime state: the servers themselves are NOT in git

As of 2026-09-29 two servers exist (both measured live, select * from servers):

Name	Type	Port	UUID	auto_start
homelab	minecraft-java (Paper 26.2, 2 GiB heap)	TCP 25565	13c7b85d-00bb-4a58-8c86-2433ca47891c	1
K3s-Bedrock	minecraft-bedrock	UDP 19132	c5837301-12b1-404e-b50e-70eafb37226b	0 ⚠

K3s-Bedrock was created 2026-09-29 20:51 and still has auto_start=0 and crash_detection=0 — re-measured 2026-09-29 21:41 straight out of crafty.sqlite (read-only), so the create-API trap documented at the bottom of this file is still live and still unfixed. It will not come back after a pod restart, and merging any change to deployment.yaml restarts the pod (strategy: Recreate). PATCH both to true and read them back.

homelab was created through Crafty's v2 API, and Mojang's EULA (https://aka.ms/MinecraftEULA) was accepted on the operator's behalf to let it start; see the DECISIONS-LOG entry of that date.

Everything in this directory describes only the capacity to run Minecraft servers. The servers are application state on the crafty-data Longhorn PVC, in two places, and Flux cannot recreate either:

• /crafty/app/config/db/ (config subPath) — Crafty's SQLite row: name, port, jar, launch command, heap sizes, auto_start / crash_detection.
• /crafty/servers/<uuid>/ (servers subPath) — paper.jar, eula.txt, server.properties, and the generated world. The jar is re-downloadable; the world is not. transport=raknet lives here too. It is the one setting that makes Bedrock reachable at all (see the transport trap), it is not in git, Flux cannot restore it, and a freshly created or reinstalled Bedrock server gets the vendor default transport=nethernet back — i.e. it silently regresses to "nothing bound on 19132/udp". Re-apply and re-ping after any Bedrock server recreate or BDS update.

So a rebuild from git alone yields a healthy but empty Crafty with nothing listening on 25565. Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH auto_start to true — see the next note. The world starts over. Treat this PVC as backup-relevant.

Trap: the create API silently ignores autostart

POST /api/v2/servers accepts and validates autostart and crashdetection, but never stores them — the persisted columns are auto_start and crash_detection. After a create that asked for true, both read back false. Set them afterwards with PATCH /api/v2/servers/<uuid> and read them back to confirm, or the server will not come up after a pod restart.