# platform/crafty

**Crafty Controller 4** — a web panel that creates and manages multiple Minecraft servers (Java +
Bedrock) as **subprocesses inside its own container**. Its web UI is **Keycloak-gated** via a per-pod
**oauth2-proxy reverse-proxy sidecar** (the decided **Option B**, see
[`../oauth2-proxy/README.md`](../oauth2-proxy/README.md)). Game ports are exposed **un-gated** on a
Cilium L2 LoadBalancer (game protocols authenticate players themselves).

- **Owner:** WS-APP (auth wiring shared with WS-SEC)
- **Flux tier:** own non-gating `clusters/base/crafty.yaml` (`dependsOn platform-config`, `wait:false`)
- **Namespace:** `crafty` (PLATFORM namespace — `gateway-access=shared`, NOT `app-namespace=true`)

## Deploy summary

| Thing | Value |
|---|---|
| Image | `registry.gitlab.com/crafty-controller/crafty-4:4.10.8` (current stable; bare-semver tags, no `v`) |
| Web UI | HTTPS **self-signed** on `:8443` (reached only via the sidecar / same-pod) |
| Sidecar | `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1`, reverse-proxy on `:4180` |
| Data | ONE Longhorn RWO PVC `crafty-data` (20Gi), mounted 5× by subPath |
| PVC subPaths | `servers`→`/crafty/servers`, `backups`→`/crafty/backups`, `logs`→`/crafty/logs`, `import`→`/crafty/import`, `config`→`/crafty/app/config` |
| Identity | `runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true` (see below) |
| Web host | `crafty.home.keithjackson.dev` → shared Gateway → `crafty` Service :80 → sidecar :4180 → Crafty :8443 |
| Game VIP | `${CRAFTY_L4_VIP}` (Cilium L2 LoadBalancer) — **prod `192.168.2.22`** (live `EXTERNAL-IP`, measured 2026-09-29); base/dev `192.168.1.55` |
| Game ports | starter set: **TCP 25565-25567** (Java), **UDP 19132-19134** (Bedrock) |

### fsGroup / user decision (the seerr lesson, applied)

Crafty's Dockerfile does `useradd -g root -M crafty` (UID **1000**, primary group **root/GID 0**),
`chown -R crafty:root /crafty`, and makes `/crafty` group-writable — the OpenShift/K8s
"arbitrary-UID, root-group-perms" pattern. A fresh Longhorn PVC is `root:root` and not group-writable,
so the non-root crafty process could not create `/crafty/servers/*` etc. and would CrashLoop (exactly
seerr's failure at UID 1000 on a `root:root` PVC). Fix: **`fsGroup: 0`** makes the kubelet group-own
the mounted volume to GID 0, mark it group-writable+SGID, and add GID 0 as a supplemental group;
**`runAsGroup: 0`** + **`runAsUser: 1000`** pin the image's own `crafty:root` identity. Longhorn's
`CSIDriver fsGroupPolicy: ReadWriteOnceWithFSType` applies fsGroup to this RWO filesystem volume.

> **HUMAN runtime check:** confirm the pod writes its SQLite DB and can start a server. If the image's
> `crafty` UID is ever not 1000, adjust `runAsUser` (the launcher's non-root path runs directly as the
> current UID, so any non-root UID with GID 0 works, but 1000 matches `getpwuid` cleanly).

## Auth = Option B (per-app oauth2-proxy reverse-proxy)

Cilium 1.16.5 cannot inject `ext_authz` into the shared Gateway's machine-generated Envoy config, and
native Gateway-API ExternalAuth ships in no released Cilium yet (see `../oauth2-proxy/README.md`). So
Crafty's UI is gated by a **dedicated oauth2-proxy sidecar in the Crafty pod** (`--reverse-proxy`,
`--upstream=https://127.0.0.1:8443`, `--ssl-upstream-insecure-skip-verify=true`,
`--allowed-groups=platform-admins`, `--skip-provider-button`, cookie/redirect for
`crafty.home.keithjackson.dev`). The `crafty` Service targets the sidecar, never Crafty's `:8443`, so
the gate is **in the data path** (fail-closed: no valid `platform-admins` session ⇒ no upstream).

> **Note:** Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the
> user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs
> in with Crafty's local admin credentials (set on first boot). This is expected, not a bug.

### Keycloak client

Confidential auth-code client **`crafty`** in the `homelab` realm — redirect
`https://crafty.home.keithjackson.dev/oauth2/callback`, `webOrigins` the same host, a `groups`
membership mapper (bare names). Provisioned **live via kcadm** at authoring time (2026-08-28) and
captured as the durable GitOps source of truth in
[`../keycloak/keycloak-realm-import.yaml`](../keycloak/keycloak-realm-import.yaml) (secret injected via
the `CRAFTY_CLIENT_SECRET` placeholder).

### Secret locations (SOPS/age)

| Secret | File | Key |
|---|---|---|
| oauth2-proxy client id/secret + cookie secret | `crafty-oauth2-proxy-credentials.sops.yaml` (this dir) | `client-id`, `client-secret`, `cookie-secret` |
| Keycloak client secret (mirror) | `../keycloak/keycloak-realm-clients.sops.yaml` | `crafty-client-secret` |

`client-secret` here **must equal** `crafty-client-secret` there (cross-namespace mirror — a Secret
can't span namespaces). Both are age-encrypted to the repo `sops-age` recipient and Flux-decryptable
(verified: both decrypt and the two values match).

## Game-port L4 exposure

The Gateway is L7-HTTPS-only; raw TCP/UDP game ports can't ride it. `service-l4.yaml` exposes them as
two `type=LoadBalancer` Services (TCP + UDP) sharing the pinned VIP `${CRAFTY_L4_VIP}` (**prod
`192.168.2.22`**)
(`lbipam.cilium.io/sharing-key: crafty`), ARP-announced by the existing `default-l2` policy. This path
is **NOT auth-gated** — Minecraft Java (Mojang/Microsoft) and Bedrock (Xbox Live) authenticate players
themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → ${CRAFTY_L4_VIP}`.

### Adding more game ports (when the operator creates servers)

As of 2026-09-29 **two** servers exist (see *Runtime state* below): Java `homelab` on 25565 and
Bedrock `K3s-Bedrock` on 19132. The starter set covers 3 Java + 3 Bedrock slots. To add capacity:

1. In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP).
2. Add that port to the matching Service in `service-l4.yaml` (one
   `{ name, port, targetPort, protocol }` line — K8s Services enumerate ports, no ranges).
3. Add it to the game-port ingress rule in `networkpolicy.yaml` (`crafty-allow-ingress`).
4. (Optional) add a `containerPort` in `deployment.yaml` for documentation.
5. Commit → Flux reconciles.

### Bedrock on UDP 19132 — the transport trap, and the ERROR banner that is NOT a refusal (MEASURED on prod 2026-09-29)

Two traps in sequence. The first hides the UDP socket; the second frightens you into undoing the fix.
As of 2026-09-29 21:34 this server is **reachable on `19132/udp` and answering**, measured from
off-cluster. Read both traps before changing anything.

#### Trap 1 — `transport=nethernet` (the shipped default) opens no RakNet UDP listener

Measured 20:54–21:32 against `K3s-Bedrock` (`c5837301-12b1-404e-b50e-70eafb37226b`, DB
`type=minecraft-bedrock`, `server_port=19132`, process `./bedrock_server` alive):

- `/proc/net/udp` and `/proc/net/udp6` held **no socket on 19132** (`0x4ABC`) at all. Port 19132 was
  bound as **TCP** (`/proc/net/tcp6`, `…:4ABC`, state `0A`/LISTEN), not UDP.
- That is nethernet working as designed, and the **shipped `bedrock_server_how_to.html`** — the
  vendor's own doc, sitting on the PVC beside the binary — says exactly why: nethernet *"listens on a
  single dual-stack **TCP** socket on `server-port` for an HTTP-based signaling handshake, then
  negotiates a UDP connection with each client that carries the gameplay traffic."*
- **So nethernet could never have worked here anyway, on two independent counts:** `service-l4.yaml`
  exposes 19132 **only in the UDP Service** (`crafty-l4-tcp` carries 25565-25567 and nothing else), and
  `crafty-allow-ingress` allows `19132 UDP` and **not** `19132 TCP`. The signaling port was unreachable
  at both the Service and the policy layer. The server's own `packet-statistics.txt` for that run
  agrees: `Total Received: 0, 0B` over 2293 s.
- Crafty does not manage this property (`transport` appears nowhere under `/crafty/app/classes/`) and
  its *only* Bedrock probe is `raknet_ping.py`, so the panel logged `WARNING
  app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping to 127.0.0.1:19132`
  every ~5 s — **155 such lines** — against a server it had started successfully.

#### Trap 2 — under `transport=raknet` this build prints a scary ERROR banner and then works anyway

The operator set `transport=raknet` at 21:33:31 (`/crafty/logs/audit.log`: `Edited file
/crafty/servers/c5837301-…/server.properties`) and started the server at 21:33:58. BDS **1.26.52**
printed, at ERROR level:

```
[ERROR] ================ TRANSPORT TYPE ERROR  ===================
[ERROR] Your current connection type is not set to NetherNet. In this release, NetherNet is the only
        supported transport type.
[ERROR] Players will not be able to connect to your game without NetherNet.
[ERROR] To switch, set 'transport=nethernet' in server.properties.
```

**That banner is advisory, not fatal. Do not revert `transport` on seeing it.** Every claim it makes
about connectivity was false on this cluster within five seconds of it being printed:

- The process **kept running** (`./bedrock_server`, pid 47006) and **bound the RakNet sockets**:
  `/proc/net/udp` `00000000:4ABC` (19132, IPv4) and `/proc/net/udp6` `…:4ABD` (19133, IPv6), both owned
  by `bedrock_server` — `/proc/47006/fd/25` and `fd/26` → those socket inodes. The earlier **TCP** bind
  on `4ABC` is **gone**.
- Crafty's warning **changed kind** at that instant: last `Got timeout` at `21:34:04,306`, first
  `Bedrock ping returned a string with too few fields` at `21:34:09,854`. That distinction is
  load-bearing — read `raknet_ping.py`: `Got timeout` is emitted *only* when `sock.recv` raises
  `TimeoutError` (nothing came back at all), whereas `too few fields` is reachable *only after* a
  `0x1C` Unconnected Pong has arrived **and its magic has validated**. The server is answering pings.
- **End-to-end, from off-cluster:** a RakNet Unconnected Ping sent from the LAN to the prod VIP
  `192.168.2.22:19132` returns a 139-byte `0x1c` Unconnected Pong in ~12 ms, magic valid:

      MCPE;Dedicated Server;2193;1.26.52;0;10;11676629557768398479;Bedrock level;Survival;1;19132;19133;0;0;0;

  Edition `MCPE`, protocol `2193`, version **`1.26.52`**, `0/10` players. **Players can reach this
  server now**, through the VIP, the Service, and the CiliumNetworkPolicy, with no manifest change.
- The vendor doc on disk contradicts the banner and matches the observed behaviour. `transport` is
  documented as *"Allowed values: raknet, nethernet"*, and: *"**raknet** — The traditional Bedrock UDP
  transport. The server listens directly on `server-port` (IPv4) and `server-portv6` (IPv6), and
  clients connect to that address and port."* RakNet is also still fully compiled into the binary —
  `strings bedrock_server` yields `RakNetConnector`, `createUniqueRakPeer`, `RakPeerInterface` and the
  vendored build path `…/src-deps/raknet/raknet/RakNetSocket2.cpp`.

So *"NetherNet is the only supported transport type"* is a **support statement, not a capability
statement**: Microsoft declines to support RakNet and ships it working. Treat the banner as a
deprecation notice with an **unknown expiry** — a future BDS may genuinely remove RakNet, and that is
the day this exposure needs GeyserMC (see *If RakNet is ever really removed* below).

> **The shipped-comment trap, stated plainly.** `server.properties` lists `raknet` as allowed and the
> binary logs an ERROR saying it is not supported. Both are the vendor, and they disagree. The tiebreak
> is neither prose: it is `/proc/net/udp` and a pong on the wire. Whenever this file says a Bedrock
> server is or is not reachable, it is because someone pinged it — not because a comment said so.

#### The steady-state `too few fields` warning is benign and is now the expected signature

Crafty will log `Bedrock ping returned a string with too few fields - missing values [None]` every
~5 s forever. **That means the ping SUCCEEDED.** BDS 1.26.52's pong carries 16 `;`-separated fields;
Crafty 4.10.8's `pingpong_string_fields` table knows 14 and expects the last to be an empty
terminator. It instead finds `'0'`, so `last_enumeration` stops at index 12 and the parser reports the
one `None` terminator entry as missing. Every field that matters (edition, MOTD, protocol, version
name, player count, max players, uuid, game mode, both ports) parsed correctly. **Do not chase this
warning.** The failure signature is the *other* message, `Got timeout`.

#### Decision: `transport=raknet` stays; the 19132-19134/UDP exposure is correct and live

No manifest change is needed or wanted. `service-l4.yaml`, `networkpolicy.yaml` and the
`containerPort`s were already shaped for exactly RakNet, and that shape is now proven end-to-end
rather than assumed. The three options that would have required manifest work are all **not
recommended** today:

| Option | Manifest consequence | Verdict |
|---|---|---|
| **Keep `transport=raknet`** (recommended) | **none** — the existing Service, policy and ports are already correct and now proven | **Take this.** Zero change, works today, vanilla parity, Bedrock clients join a real Bedrock world. |
| **GeyserMC + Floodgate** on the `homelab` Paper server | none either (Geyser also binds `19132/udp`), but it would **collide** with the live BDS on 19132 — one of them must move | Hold in reserve. Only worth it if RakNet is really removed, or if you *want* one shared world. Cost: Bedrock players join the **Java** world, not a Bedrock world; Java-only mechanics and some content differ; it is another moving part to patch. |
| **Pin/downgrade BDS** to a pre-nethernet release | none | Unnecessary — the *current* build already speaks RakNet. Also fragile: Microsoft removes old BDS downloads, and Crafty's Bedrock installer tracks latest. UNVERIFIED whether Crafty can pin a version at all. |
| **A third-party server** (PocketMine-MP, Nukkit) | none | No. Loses vanilla parity and the world format; a large change to solve a problem that is not occurring. |
| **Accept NetherNet** | 19132-19134/UDP becomes **inert**; nethernet would instead need **TCP** 19132 added to `crafty-l4-tcp` *and* to `crafty-allow-ingress`, plus `server-udp-ports` pinned and exposed for the negotiated media path | No. Strictly more manifest work than the option that already works, and the WebRTC media path behind a shared VIP and SNAT is unproven here. |

#### If RakNet is ever really removed

Then BDS opens no UDP listener and this exposure has nothing to bind to — an architecture problem, not
a config one. The migration is **GeyserMC + Floodgate as a plugin on the existing `homelab` Paper
server**, because Geyser speaks RakNet on `19132/udp` itself: the VIP, the Service, the policy and the
`containerPort`s all stay **byte-for-byte as they are**, and Bedrock clients keep connecting to
`crafty-mc.home.keithjackson.dev:19132`. What changes is the world: Bedrock players land in the Java
world via Floodgate (no Xbox-authenticated Bedrock account needed on the Java side), the Bedrock server
is retired, and `K3s-Bedrock` stops the day Geyser starts — **they cannot share 19132**. Installing it
is a PVC write plus a Paper restart, i.e. an **operator action**: drop `Geyser-Spigot.jar` and
`floodgate-spigot.jar` into `/crafty/servers/13c7b85d-…/plugins/` and restart `homelab`, then set
Geyser's `bedrock.port: 19132` and `clone-remote-port: false` in its generated `config.yml`.
Geyser's own Bedrock-version support is **external and unmeasured here** — check that it supports the
Bedrock client version in use before committing.

### Bedrock realities that are NOT platform defects but present identically

- **LAN / Friends-tab discovery can never work here.** Bedrock discovery is a UDP broadcast on 19132
  inside one L2 segment; this server sits behind a LoadBalancer VIP (and SNAT), not on the LAN, so it
  will **never** appear in a client's Friends/LAN list. Players must use **Add Server** and type the
  address and port by hand: `crafty-mc.home.keithjackson.dev` (or the VIP) with port **19132**.
  `external-dns` publishes that name from the **-tcp** Service, so it resolves to the shared VIP and
  is correct for Bedrock too — only the port differs from Java's 25565.
- **Consoles cannot reach this at all.** Switch/Xbox/PlayStation Bedrock clients have no
  arbitrary-server-address field; only Windows/mobile/Fire do. Reaching it from a console needs a DNS
  redirect of a featured-server hostname, which is out of scope for `service-l4.yaml`.
- **`enable-lan-visibility=true` (the shipped default) collides with Bedrock slots 1 and 2 — now
  OBSERVED, not merely documented.** Under `transport=raknet` it makes a server bind the **default**
  ports 19132 *and* 19133 even when `server-port`/`server-portv6` say otherwise; the shipped
  `bedrock_server_how_to.html` says so, and on 2026-09-29 a RakNet ping to
  `192.168.2.22:19133` returned **the same pong, from the same `server_guid`, as `:19132`** — one
  server occupying two of the three Bedrock slots. `:19134` correctly timed out. So a second Bedrock
  server put on 19133 will fight the first. Set `enable-lan-visibility=false` on every Bedrock server
  after the first; LAN discovery cannot work here anyway (above).
- **`server-portv6` is the IPv6 listener and is not exposed; `19133` is nonetheless in use.**
  `service-l4.yaml` is IPv4 single-stack, so the *IPv6* socket `…:4ABD` that BDS opens for
  `server-portv6=19133` is unreachable from the LAN. But because `enable-lan-visibility=true` (above)
  the server *also* answers on **IPv4 19133**, which the UDP Service does expose — measured. So
  `19133` in `server.properties` genuinely does mean "slot 1 is taken", contrary to what this file
  previously said.

## Policies

`namespace.yaml` sets only `gateway-access=shared` (platform namespace, like headlamp/homepage), so
Kyverno does not auto-generate the app baseline. `networkpolicy.yaml` ships it explicitly (defense in
depth): default-deny (DNS-only egress) + Cilium allows — Gateway→sidecar `:4180`, world/LAN→game
ports, host→probe ports, and egress `world` (server-jar downloads + the OIDC path to Keycloak via the
Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subject to network policy.

## Verification

- `make -C ci gate` — green (hand-authored platform manifests; render-golden unaffected).
- `kustomize build` of `platform/crafty`, `clusters/base`, `clusters/dev` — clean.

### Human tests (post-deploy)

1. **Browser OIDC round-trip:** incognito → `https://crafty.home.keithjackson.dev` → expect a 302 to
   `https://keycloak.home.keithjackson.dev/realms/homelab/...` (NOT the Crafty UI). Log in as a
   **platform-admins** user (MFA/OTP enforced) → land on Crafty's own login → sign in with Crafty's
   local admin. A **non-platform-admins** user → **403** (authenticated but not authorized).
2. **Pod health / fsGroup:** `kubectl -n crafty get pod` Ready 2/2; Crafty writes its DB under
   `/crafty/app/config` and `/crafty/servers` without EACCES.
3. **Server-create + game reach (Java):** in the Crafty UI create a Java server bound to `25565`;
   start it; connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or
   `192.168.2.22:25565`) from the LAN and confirm the world loads.
4. **Server-create + game reach (Bedrock):** creating and starting the server is **not sufficient**.
   Check, in this order (see *the transport trap* above): `transport=raknet` in `server.properties`;
   a live **UDP** listener (`grep 4ABC /proc/net/udp /proc/net/udp6` → a wildcard bind); and a real
   pong from off-cluster. **Ignore the `TRANSPORT TYPE ERROR` banner in the console and ignore
   Crafty's `too few fields` warning — both appear on a fully working server.** The authoritative
   test is a RakNet Unconnected Ping to `192.168.2.22:19132` returning a `0x1c` packet whose MOTD
   string starts `MCPE;`. Then **Add Server** by hand in a Windows/mobile client at
   `crafty-mc.home.keithjackson.dev` port `19132`. It will never show up in the LAN/Friends tab, and a
   console cannot reach it.

## Runtime state: the servers themselves are NOT in git

As of 2026-09-29 **two** servers exist (both measured live, `select * from servers`):

| Name | Type | Port | UUID | `auto_start` |
|---|---|---|---|---|
| `homelab` | `minecraft-java` (Paper 26.2, 2 GiB heap) | TCP 25565 | `13c7b85d-00bb-4a58-8c86-2433ca47891c` | **1** |
| `K3s-Bedrock` | `minecraft-bedrock` | UDP 19132 | `c5837301-12b1-404e-b50e-70eafb37226b` | **0** ⚠ |

`K3s-Bedrock` was created 2026-09-29 20:51 and **still** has `auto_start=0` **and**
`crash_detection=0` — re-measured 2026-09-29 21:41 straight out of `crafty.sqlite` (read-only), so the
create-API trap documented at the bottom of this file is still live and still unfixed. **It will not
come back after a pod restart**, and merging any change to `deployment.yaml` restarts the pod
(`strategy: Recreate`). PATCH both to true and read them back.

`homelab` was created through Crafty's v2 API, and **Mojang's
EULA (<https://aka.ms/MinecraftEULA>) was accepted on the operator's behalf** to let it start; see the
DECISIONS-LOG entry of that date.

Everything in this directory describes only the *capacity* to run Minecraft servers. The servers are
application state on the `crafty-data` Longhorn PVC, in two places, and **Flux cannot recreate either**:

- `/crafty/app/config/db/` (`config` subPath) — Crafty's SQLite row: name, port, jar, launch command,
  heap sizes, `auto_start` / `crash_detection`.
- `/crafty/servers/<uuid>/` (`servers` subPath) — `paper.jar`, `eula.txt`, `server.properties`, and
  **the generated world**. The jar is re-downloadable; the world is not.
  **`transport=raknet` lives here too.** It is the one setting that makes Bedrock reachable at all (see
  *the transport trap*), it is not in git, Flux cannot restore it, and a freshly created or reinstalled
  Bedrock server gets the vendor default `transport=nethernet` back — i.e. it silently regresses to
  "nothing bound on 19132/udp". Re-apply and re-ping after any Bedrock server recreate or BDS update.

So a rebuild from git alone yields a healthy but **empty** Crafty with nothing listening on 25565.
Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH
`auto_start` to true — see the next note. The world starts over. **Treat this PVC as backup-relevant.**

### Trap: the create API silently ignores `autostart`

`POST /api/v2/servers` accepts and validates `autostart` and `crashdetection`, but never stores them —
the persisted columns are `auto_start` and `crash_detection`. After a create that asked for `true`,
both read back `false`. Set them afterwards with `PATCH /api/v2/servers/<uuid>` and **read them back to
confirm**, or the server will not come up after a pod restart.