home-network-k3s-impl/platform/crafty/README.md

# platform/crafty

**Crafty Controller 4** — a web panel that creates and manages multiple Minecraft servers (Java +
Bedrock) as **subprocesses inside its own container**. Its web UI is **Keycloak-gated** via a per-pod
**oauth2-proxy reverse-proxy sidecar** (the decided **Option B**, see
[`../oauth2-proxy/README.md`](../oauth2-proxy/README.md)). Game ports are exposed **un-gated** on a
Cilium L2 LoadBalancer (game protocols authenticate players themselves).

- **Owner:** WS-APP (auth wiring shared with WS-SEC)
- **Flux tier:** own non-gating `clusters/base/crafty.yaml` (`dependsOn platform-config`, `wait:false`)
- **Namespace:** `crafty` (PLATFORM namespace — `gateway-access=shared`, NOT `app-namespace=true`)

## Deploy summary

| Thing | Value |
|---|---|
| Image | `registry.gitlab.com/crafty-controller/crafty-4:4.10.8` (current stable; bare-semver tags, no `v`) |
| Web UI | HTTPS **self-signed** on `:8443` (reached only via the sidecar / same-pod) |
| Sidecar | `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1`, reverse-proxy on `:4180` |
| Data | ONE Longhorn RWO PVC `crafty-data` (20Gi), mounted 5× by subPath |
| PVC subPaths | `servers`→`/crafty/servers`, `backups`→`/crafty/backups`, `logs`→`/crafty/logs`, `import`→`/crafty/import`, `config`→`/crafty/app/config` |
| Identity | `runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true` (see below) |
| Web host | `crafty.home.keithjackson.dev` → shared Gateway → `crafty` Service :80 → sidecar :4180 → Crafty :8443 |
| Game VIP | `${CRAFTY_L4_VIP}` (Cilium L2 LoadBalancer) — **prod `192.168.2.22`** (live `EXTERNAL-IP`, measured 2026-09-29); base/dev `192.168.1.55` |
| Game ports | starter set: **TCP 25565-25567** (Java), **UDP 19132-19134** (Bedrock) |

### fsGroup / user decision (the seerr lesson, applied)

Crafty's Dockerfile does `useradd -g root -M crafty` (UID **1000**, primary group **root/GID 0**),
`chown -R crafty:root /crafty`, and makes `/crafty` group-writable — the OpenShift/K8s
"arbitrary-UID, root-group-perms" pattern. A fresh Longhorn PVC is `root:root` and not group-writable,
so the non-root crafty process could not create `/crafty/servers/*` etc. and would CrashLoop (exactly
seerr's failure at UID 1000 on a `root:root` PVC). Fix: **`fsGroup: 0`** makes the kubelet group-own
the mounted volume to GID 0, mark it group-writable+SGID, and add GID 0 as a supplemental group;
**`runAsGroup: 0`** + **`runAsUser: 1000`** pin the image's own `crafty:root` identity. Longhorn's
`CSIDriver fsGroupPolicy: ReadWriteOnceWithFSType` applies fsGroup to this RWO filesystem volume.

> **HUMAN runtime check:** confirm the pod writes its SQLite DB and can start a server. If the image's
> `crafty` UID is ever not 1000, adjust `runAsUser` (the launcher's non-root path runs directly as the
> current UID, so any non-root UID with GID 0 works, but 1000 matches `getpwuid` cleanly).

## Auth = Option B (per-app oauth2-proxy reverse-proxy)

Cilium 1.16.5 cannot inject `ext_authz` into the shared Gateway's machine-generated Envoy config, and
native Gateway-API ExternalAuth ships in no released Cilium yet (see `../oauth2-proxy/README.md`). So
Crafty's UI is gated by a **dedicated oauth2-proxy sidecar in the Crafty pod** (`--reverse-proxy`,
`--upstream=https://127.0.0.1:8443`, `--ssl-upstream-insecure-skip-verify=true`,
`--allowed-groups=platform-admins`, `--skip-provider-button`, cookie/redirect for
`crafty.home.keithjackson.dev`). The `crafty` Service targets the sidecar, never Crafty's `:8443`, so
the gate is **in the data path** (fail-closed: no valid `platform-admins` session ⇒ no upstream).

#### Cookie isolation (why the session used to vanish)

This sidecar and the **shared** forward-auth oauth2-proxy are independent instances with **different
`cookie-secret` values** (measured in-cluster — neither can decrypt the other's cookie). Both used to
emit oauth2-proxy's *default* cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
`.home.keithjackson.dev`, so the browser held **one** cookie pair for **both**: whichever proxy
authenticated last clobbered the other's session, and the loser logged
`No valid authentication in request`, bounced to Keycloak, and — because the Keycloak SSO session was
still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents
exactly as *"I signed in to Keycloak but I'm still not signed in."*

Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app):
`--cookie-name=_oauth2_proxy_crafty` and a **host-scoped** `--cookie-domain=crafty.home.keithjackson.dev`.
`--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.

#### Crafty's own local login — this is where setup actually blocks

Crafty has **no OIDC**, so after oauth2-proxy admits the user they still meet Crafty's own
username/password page. That second factor is intended. What was **not** intended — and what made the
service impossible to set up — is that **nothing documented where that password comes from**.

Crafty generates it **once, on first boot only**, when the users table is empty
(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`), and writes it to
**`/crafty/app/config/default-creds.txt`** (mode `0600`, on the `config` subPath of the PVC). It is
**never** in git and **cannot** be recovered from git: a rebuild onto a fresh PVC generates a *new*
random password. Submitting anything else yields `POST /api/v2/auth/login/` → **401**, which is exactly
what a browser sees with no hint as to why.

**→ Retrieval, rotation, and lost-password recovery: [`../../runbooks/crafty-admin-credential.md`](../../runbooks/crafty-admin-credential.md).**

Two traps that runbook covers:

- Crafty's `set_passwd` console command is **unreachable** in Kubernetes — it calls
  `getpass.getpass()` and so needs a TTY, while the container runs `main.py -d -i` (daemon).
  Recovery therefore has to go through the DB with Crafty's own argon2 hasher.
- Crafty's own 3-strikes brute-force cooldown is **inert behind this sidecar**, verified live. Crafty
  *records* failed attempts under `get_remote_ip()` (which honours `X-Forwarded-For` — the live
  tracker key is the chain `"192.168.2.132, 10.0.0.123"`) but *checks* for a cooldown under
  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The two halves key
  differently, so `get_recent_attempts()` always reads an empty list and `max_login_attempts: 3`
  never fires. Good news for the operator (**repeated guesses cannot lock you out**), but it means
  the only thing actually limiting who can reach the login form is the oauth2-proxy
  `platform-admins` gate — worth knowing before anyone treats that cooldown as a control.

### Keycloak client

Confidential auth-code client **`crafty`** in the `homelab` realm — redirect
`https://crafty.home.keithjackson.dev/oauth2/callback`, `webOrigins` the same host, a `groups`
membership mapper (bare names). Provisioned **live via kcadm** at authoring time (2026-08-28) and
captured as the durable GitOps source of truth in
[`../keycloak/keycloak-realm-import.yaml`](../keycloak/keycloak-realm-import.yaml) (secret injected via
the `CRAFTY_CLIENT_SECRET` placeholder).

### Secret locations (SOPS/age)

| Secret | File | Key |
|---|---|---|
| oauth2-proxy client id/secret + cookie secret | `crafty-oauth2-proxy-credentials.sops.yaml` (this dir) | `client-id`, `client-secret`, `cookie-secret` |
| Keycloak client secret (mirror) | `../keycloak/keycloak-realm-clients.sops.yaml` | `crafty-client-secret` |

`client-secret` here **must equal** `crafty-client-secret` there (cross-namespace mirror — a Secret
can't span namespaces). Both are age-encrypted to the repo `sops-age` recipient and Flux-decryptable
(verified: both decrypt and the two values match).

## Game-port L4 exposure

The Gateway is L7-HTTPS-only; raw TCP/UDP game ports can't ride it. `service-l4.yaml` exposes them as
two `type=LoadBalancer` Services (TCP + UDP) sharing the pinned VIP `${CRAFTY_L4_VIP}` (**prod
`192.168.2.22`**)
(`lbipam.cilium.io/sharing-key: crafty`), ARP-announced by the existing `default-l2` policy. This path
is **NOT auth-gated** — Minecraft Java (Mojang/Microsoft) and Bedrock (Xbox Live) authenticate players
themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → ${CRAFTY_L4_VIP}`.

### Adding more game ports (when the operator creates servers)

As of 2026-09-29 **two** servers exist (see *Runtime state* below): Java `homelab` on 25565 and
Bedrock `K3s-Bedrock` on 19132. The starter set covers 3 Java + 3 Bedrock slots. To add capacity:

1. In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP).
2. Add that port to the matching Service in `service-l4.yaml` (one
   `{ name, port, targetPort, protocol }` line — K8s Services enumerate ports, no ranges).
3. Add it to the game-port ingress rule in `networkpolicy.yaml` (`crafty-allow-ingress`).
4. (Optional) add a `containerPort` in `deployment.yaml` for documentation.
5. Commit → Flux reconciles.

### Bedrock on UDP 19132 — the transport trap, and the ERROR banner that is NOT a refusal (MEASURED on prod 2026-09-29)

Two traps in sequence. The first hides the UDP socket; the second frightens you into undoing the fix.
As of 2026-09-29 21:34 this server is **reachable on `19132/udp` and answering**, measured from
off-cluster. Read both traps before changing anything.

#### Trap 1 — `transport=nethernet` (the shipped default) opens no RakNet UDP listener

Measured 20:54–21:32 against `K3s-Bedrock` (`c5837301-12b1-404e-b50e-70eafb37226b`, DB
`type=minecraft-bedrock`, `server_port=19132`, process `./bedrock_server` alive):

- `/proc/net/udp` and `/proc/net/udp6` held **no socket on 19132** (`0x4ABC`) at all. Port 19132 was
  bound as **TCP** (`/proc/net/tcp6`, `…:4ABC`, state `0A`/LISTEN), not UDP.
- That is nethernet working as designed, and the **shipped `bedrock_server_how_to.html`** — the
  vendor's own doc, sitting on the PVC beside the binary — says exactly why: nethernet *"listens on a
  single dual-stack **TCP** socket on `server-port` for an HTTP-based signaling handshake, then
  negotiates a UDP connection with each client that carries the gameplay traffic."*
- **So nethernet could never have worked here anyway, on two independent counts:** `service-l4.yaml`
  exposes 19132 **only in the UDP Service** (`crafty-l4-tcp` carries 25565-25567 and nothing else), and
  `crafty-allow-ingress` allows `19132 UDP` and **not** `19132 TCP`. The signaling port was unreachable
  at both the Service and the policy layer. The server's own `packet-statistics.txt` for that run
  agrees: `Total Received: 0, 0B` over 2293 s.
- Crafty does not manage this property (`transport` appears nowhere under `/crafty/app/classes/`) and
  its *only* Bedrock probe is `raknet_ping.py`, so the panel logged `WARNING
  app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping to 127.0.0.1:19132`
  every ~5 s — **155 such lines** — against a server it had started successfully.

#### Trap 2 — under `transport=raknet` this build prints a scary ERROR banner and then works anyway

The operator set `transport=raknet` at 21:33:31 (`/crafty/logs/audit.log`: `Edited file
/crafty/servers/c5837301-…/server.properties`) and started the server at 21:33:58. BDS **1.26.52**
printed, at ERROR level:

```
[ERROR] ================ TRANSPORT TYPE ERROR  ===================
[ERROR] Your current connection type is not set to NetherNet. In this release, NetherNet is the only
        supported transport type.
[ERROR] Players will not be able to connect to your game without NetherNet.
[ERROR] To switch, set 'transport=nethernet' in server.properties.
```

**That banner is advisory, not fatal. Do not revert `transport` on seeing it.** Every claim it makes
about connectivity was false on this cluster within five seconds of it being printed:

- The process **kept running** (`./bedrock_server`, pid 47006) and **bound the RakNet sockets**:
  `/proc/net/udp` `00000000:4ABC` (19132, IPv4) and `/proc/net/udp6` `…:4ABD` (19133, IPv6), both owned
  by `bedrock_server` — `/proc/47006/fd/25` and `fd/26` → those socket inodes. The earlier **TCP** bind
  on `4ABC` is **gone**.
- Crafty's warning **changed kind** at that instant: last `Got timeout` at `21:34:04,306`, first
  `Bedrock ping returned a string with too few fields` at `21:34:09,854`. That distinction is
  load-bearing — read `raknet_ping.py`: `Got timeout` is emitted *only* when `sock.recv` raises
  `TimeoutError` (nothing came back at all), whereas `too few fields` is reachable *only after* a
  `0x1C` Unconnected Pong has arrived **and its magic has validated**. The server is answering pings.
- **End-to-end, from off-cluster:** a RakNet Unconnected Ping sent from the LAN to the prod VIP
  `192.168.2.22:19132` returns a 139-byte `0x1c` Unconnected Pong in ~12 ms, magic valid:

      MCPE;Dedicated Server;2193;1.26.52;0;10;11676629557768398479;Bedrock level;Survival;1;19132;19133;0;0;0;

  Edition `MCPE`, protocol `2193`, version **`1.26.52`**, `0/10` players. **Players can reach this
  server now**, through the VIP, the Service, and the CiliumNetworkPolicy, with no manifest change.
- The vendor doc on disk contradicts the banner and matches the observed behaviour. `transport` is
  documented as *"Allowed values: raknet, nethernet"*, and: *"**raknet** — The traditional Bedrock UDP
  transport. The server listens directly on `server-port` (IPv4) and `server-portv6` (IPv6), and
  clients connect to that address and port."* RakNet is also still fully compiled into the binary —
  `strings bedrock_server` yields `RakNetConnector`, `createUniqueRakPeer`, `RakPeerInterface` and the
  vendored build path `…/src-deps/raknet/raknet/RakNetSocket2.cpp`.

So *"NetherNet is the only supported transport type"* is a **support statement, not a capability
statement**: Microsoft declines to support RakNet and ships it working. Treat the banner as a
deprecation notice with an **unknown expiry** — a future BDS may genuinely remove RakNet, and that is
the day this exposure needs GeyserMC (see *If RakNet is ever really removed* below).

> **The shipped-comment trap, stated plainly.** `server.properties` lists `raknet` as allowed and the
> binary logs an ERROR saying it is not supported. Both are the vendor, and they disagree. The tiebreak
> is neither prose: it is `/proc/net/udp` and a pong on the wire. Whenever this file says a Bedrock
> server is or is not reachable, it is because someone pinged it — not because a comment said so.

#### The steady-state `too few fields` warning is benign and is now the expected signature

Crafty will log `Bedrock ping returned a string with too few fields - missing values [None]` every
~5 s forever. **That means the ping SUCCEEDED.** BDS 1.26.52's pong carries 16 `;`-separated fields;
Crafty 4.10.8's `pingpong_string_fields` table knows 14 and expects the last to be an empty
terminator. It instead finds `'0'`, so `last_enumeration` stops at index 12 and the parser reports the
one `None` terminator entry as missing. Every field that matters (edition, MOTD, protocol, version
name, player count, max players, uuid, game mode, both ports) parsed correctly. **Do not chase this
warning.** The failure signature is the *other* message, `Got timeout`.

### "You are not invited to play on this server" — the join gate is the allowlist, not the network (MEASURED 2026-09-29 21:56; **RESOLVED and confirmed 22:03**)

A Bedrock client reported **"Unable to connect / You are not invited to play on this server"**. That is
an **authorization** refusal, and it is the expected, guaranteed behaviour of this server's current
configuration. Measured:

| Property / file | Value | Source |
|---|---|---|
| `allow-list` | **`true`** (line 37) | `server.properties` |
| `allowlist.json` | **`[]`** — 3 bytes, mtime 20:51, never edited since server create | `cat allowlist.json` |
| `online-mode` | `true` (line 31) | `server.properties` |
| `transport` | **`raknet`** (line 71), file mtime **still 21:33:31** | `server.properties` |
| TCP 19132 listener | **none** (`NO-TCP-4ABC`) | `grep 4ABC /proc/net/tcp /proc/net/tcp6` |
| UDP 19132 listener | present, `00000000:4ABC`, pid 47006 | `grep 4ABC /proc/net/udp` |

**`allow-list=true` with an empty `allowlist.json` refuses every player, unconditionally.** The shipped
`bedrock_server_how_to.html` states it: *"If the `allow-list` property is enabled in server.properties
then the server will only allow selected users to connect. To allow a user to connect you need to know
their Xbox Live **Gamertag**."* An empty list selects nobody. This is not a network problem, not a
transport problem, and not a platform problem — and no amount of Service or policy work will change it.

#### CONFIRMED: one gamertag added, and a player joined

**This is no longer a hypothesis. The fix was applied and it worked.** *(The join itself is
**operator-reported** — 2026-09-29, the operator added the player's gamertag and the Bedrock client
connected and joined. I did not watch it.)* Two independent things corroborate it **directly on disk**:

- `allowlist.json` went from `[]` to **one entry**, and that entry's **`xuid` is populated**
  (`keys=['ignoresPlayerLimit','name','xuid']`, file 75 bytes, mtime **22:03:17**). Per the vendor doc,
  `xuid` is *"Optional. If it's not set then it will be populated when someone with a matching name
  **connects**."* The operator wrote only the name — `audit.log` records `Edited file
  …/allowlist.json` at **22:02:13** — so **the XUID was written by the server, on connect, at 22:03:17**.
  That is server-authored evidence of an Xbox-Live-authenticated join. *(Observed: the file state and
  both timestamps. Reasoned from the vendor doc: that a populated XUID implies a connect.)*
- The RakNet path still answers after the restart this involved: a pong from `192.168.2.22:19132` at
  22:06, `MCPE;…;1.26.52;0;10;…` (`players=0` — the player had left by then), new `server_guid` matching
  the new pid.

**So the causal chain is closed: `allow-list=true` + `allowlist.json == []` was the wall, and nothing
else was.** The earlier mapping of the client's *"You are not invited"* string to the empty allowlist was
**reasoned** from the vendor doc (there is no BDS log on disk to read it from); adding exactly one
gamertag and getting exactly one successful join **confirms that mapping**. No transport change, no
manifest change, and no network change was involved in the fix.

> **Note the route the operator actually used:** they **edited `allowlist.json` directly** and then
> **restarted** the server (`audit.log`: edit 22:02:13 → `stop_server` 22:02:37 → `start_server`
> 22:02:41; pid 47006 → **47875**). That is route 2 below, and a restart is a valid substitute for
> `allowlist reload`. Route 1 (`allowlist add` in the console) avoids the restart and is still preferred.

**This is also positive evidence that the RakNet path works end to end.** A refusal authored by *our*
server required the client to reach our server and get an answer back. The only listener on 19132 is
the **RakNet UDP** socket — there is **no TCP 19132 listener at all**, and NetherNet requires a TCP
signalling socket on `server-port`. So the client cannot have used NetherNet against this server; it
arrived over `19132/udp`, through `crafty-l4-udp` and the VIP. **The 19132-19134/UDP exposure is not
inert — it is the path that carried both the pong and this refusal.** *(Observed: the listeners, the
absent TCP socket, `transport=raknet`. Reasoned from those: that the refusal traversed that path. Not
captured: the actual player flow — see the caveat below.)*

#### Caveat: BDS console output is NOT on disk, so a join attempt cannot be correlated from the filesystem

`K3s-Bedrock`'s `log_path` in `crafty.sqlite` is the **empty string**, and `Dedicated_Server.txt` is
**0 bytes**. Crafty does not persist a Bedrock server's stdout anywhere on the PVC — the console exists
only in the panel UI. `commander.log` between 21:39 and 21:56 contains nothing but `homelab`
crash-detection lines and the `raknet_ping` chatter: **no join, no gamertag, no XUID, no allowlist
denial.** That absence is **not** evidence about where the refusal came from; it is evidence that
Crafty logs no BDS stdout at all. **To see a rejected join you must watch the panel console live.**
UNVERIFIED, and only observable in the UI.

#### How a Bedrock player actually gets authorized here

Identity is an **Xbox Live Gamertag** (`online-mode=true` forces XBL auth, which is what gives BDS a
gamertag/XUID to check). `allowlist.json` is a JSON array; per the vendor doc the keys are `name` (the
gamertag), optional `xuid` (*"populated when someone with a matching name connects"*), and
`ignoresPlayerLimit`. So the operator adds **one entry per player, keyed on gamertag** — there is no
group, role, or Keycloak integration, and **none of this touches the oauth2-proxy/Keycloak gate**,
which protects only the web panel.

Three ways to do it, all **operator actions**. **Routes 1 and 2 are both proven** — route 2 is the one
that produced the confirmed join above:

1. **Preferred — the panel console.** In Crafty, open `K3s-Bedrock` → Console and send:
   `allowlist add <Gamertag>` (quote it if it contains a space: `allowlist add "Example Name"`).
   This writes `allowlist.json` **and** takes effect immediately, with no restart.
2. **Edit the file, then reload.** Write `/crafty/servers/c5837301-…/allowlist.json`, then send
   `allowlist reload` in the console — *"After you've modified the file you need to run the command
   `allowlist reload` to make sure that the server knows about your new change."* Shape:

   ```json
   [
     { "ignoresPlayerLimit": false, "name": "FirstPlayerGamertag" },
     { "ignoresPlayerLimit": false, "name": "SecondPlayerGamertag" }
   ]
   ```

3. **Turn the allowlist off entirely** — `allow-list=false` in `server.properties` (restart), or
   `allowlist off` in the console. Note the vendor's warning: `allowlist on|off` *"does not change the
   value in the server.properties file"*, so a console `off` silently reverts on the next start.
   **Not recommended:** the game port is deliberately un-gated at the network layer (see *Game-port L4
   exposure*), so the allowlist is the **only** thing deciding who may join. Keep it `true` and enumerate
   players.

> **`allowlist.json` is application state on the PVC**, exactly like `transport=raknet`. Neither is in
> git and **Flux can restore neither**. Be precise about what breaks them, because the two cases differ:
>
> - **A pod or server *restart* preserves both** — observed: the 22:02:48 restart (pid 47006 → 47875)
>   kept `transport=raknet` (`server.properties` mtime unchanged at 21:33:31) and kept the allowlist
>   entry. What a **pod** restart *does* break is that `K3s-Bedrock` has `auto_start=0`, so the Bedrock
>   server simply **does not come back** until someone starts it by hand.
> - **A server *recreate* or reinstall silently resets both** — a fresh `server.properties` ships
>   `transport=nethernet` (nothing bound on 19132/udp) and a fresh `allowlist.json` is `[]` (everyone
>   refused). Neither change announces itself, and the server will look healthy and started in the panel
>   while being unreachable *and* unjoinable.
>
> **Treat both files as backup-relevant, and re-run the 5-step connect procedure above after any Bedrock
> recreate or BDS update.**

#### Why this does not change the transport recommendation

`transport=raknet` is correct, and with the allowlist entry in place the combination is **proven
end-to-end by a player who has joined** — not merely the best-reasoned option. The allowlist refusal was
orthogonal to transport all along: it would have refused players identically under NetherNet. Switching
to NetherNet would *add* work (TCP 19132 in `crafty-l4-tcp` **and** in `crafty-allow-ingress`, plus
`server-udp-ports` pinned and exposed) and would admit **not one extra player**. The ranking below stands,
and option 1 is now settled rather than provisional.

#### Decision (PROVEN, not provisional): `transport=raknet` stays; the 19132-19134/UDP exposure is correct and live

No manifest change is needed or wanted. `service-l4.yaml`, `networkpolicy.yaml` and the
`containerPort`s were already shaped for exactly RakNet, and that shape is now **proven by a player who
has connected and joined over `192.168.2.22:19132`** — not assumed, and not merely reasoned. The options
that would have required manifest work are all **not recommended**:

| Option | Manifest consequence | Verdict |
|---|---|---|
| **Keep `transport=raknet`** + allowlist entries (**this is what works**) | **none** — the existing Service, policy and ports were already correct | **Done; proven.** A player has joined over the VIP. Zero manifest change, vanilla parity, real Bedrock world. |
| **GeyserMC + Floodgate** on the `homelab` Paper server | none either (Geyser also binds `19132/udp`), but it would **collide** with the live BDS on 19132 — one of them must move | Hold in reserve. Only worth it if RakNet is really removed, or if you *want* one shared world. Cost: Bedrock players join the **Java** world, not a Bedrock world; Java-only mechanics and some content differ; it is another moving part to patch. |
| **Pin/downgrade BDS** to a pre-nethernet release | none | Unnecessary — the *current* build already speaks RakNet. Also fragile: Microsoft removes old BDS downloads, and Crafty's Bedrock installer tracks latest. UNVERIFIED whether Crafty can pin a version at all. |
| **A third-party server** (PocketMine-MP, Nukkit) | none | No. Loses vanilla parity and the world format; a large change to solve a problem that is not occurring. |
| **Accept NetherNet** | 19132-19134/UDP becomes **inert**; nethernet would instead need **TCP** 19132 added to `crafty-l4-tcp` *and* to `crafty-allow-ingress`, plus `server-udp-ports` pinned and exposed for the negotiated media path | No — and note it is **not currently in use**: `transport=raknet` and there is no TCP 19132 listener. Strictly more manifest work than the option that already works, it admits no extra player (the allowlist is the gate), and WebRTC media behind a shared VIP and SNAT is unproven here. |

#### If RakNet is ever really removed

Then BDS opens no UDP listener and this exposure has nothing to bind to — an architecture problem, not
a config one. The migration is **GeyserMC + Floodgate as a plugin on the existing `homelab` Paper
server**, because Geyser speaks RakNet on `19132/udp` itself: the VIP, the Service, the policy and the
`containerPort`s all stay **byte-for-byte as they are**, and Bedrock clients keep connecting to
`crafty-mc.home.keithjackson.dev:19132`. What changes is the world: Bedrock players land in the Java
world via Floodgate (no Xbox-authenticated Bedrock account needed on the Java side), the Bedrock server
is retired, and `K3s-Bedrock` stops the day Geyser starts — **they cannot share 19132**. Installing it
is a PVC write plus a Paper restart, i.e. an **operator action**: drop `Geyser-Spigot.jar` and
`floodgate-spigot.jar` into `/crafty/servers/13c7b85d-…/plugins/` and restart `homelab`, then set
Geyser's `bedrock.port: 19132` and `clone-remote-port: false` in its generated `config.yml`.
Geyser's own Bedrock-version support is **external and unmeasured here** — check that it supports the
Bedrock client version in use before committing.

### Bedrock realities that are NOT platform defects but present identically

- **LAN / Friends-tab discovery can never work here.** Bedrock discovery is a UDP broadcast on 19132
  inside one L2 segment; this server sits behind a LoadBalancer VIP (and SNAT), not on the LAN, so it
  will **never** appear in a client's Friends/LAN list. Players must use **Add Server** and type the
  address and port by hand: `crafty-mc.home.keithjackson.dev` (or the VIP) with port **19132**.
  `external-dns` publishes that name from the **-tcp** Service, so it resolves to the shared VIP and
  is correct for Bedrock too — only the port differs from Java's 25565.
- **Consoles cannot reach this at all.** Switch/Xbox/PlayStation Bedrock clients have no
  arbitrary-server-address field; only Windows/mobile/Fire do. Reaching it from a console needs a DNS
  redirect of a featured-server hostname, which is out of scope for `service-l4.yaml`.
- **`enable-lan-visibility=true` (the shipped default) collides with Bedrock slots 1 and 2 — now
  OBSERVED, not merely documented.** Under `transport=raknet` it makes a server bind the **default**
  ports 19132 *and* 19133 even when `server-port`/`server-portv6` say otherwise; the shipped
  `bedrock_server_how_to.html` says so, and on 2026-09-29 a RakNet ping to
  `192.168.2.22:19133` returned **the same pong, from the same `server_guid`, as `:19132`** — one
  server occupying two of the three Bedrock slots. `:19134` correctly timed out. So a second Bedrock
  server put on 19133 will fight the first. Set `enable-lan-visibility=false` on every Bedrock server
  after the first; LAN discovery cannot work here anyway (above).
- **`server-portv6` is the IPv6 listener and is not exposed; `19133` is nonetheless in use.**
  `service-l4.yaml` is IPv4 single-stack, so the *IPv6* socket `…:4ABD` that BDS opens for
  `server-portv6=19133` is unreachable from the LAN. But because `enable-lan-visibility=true` (above)
  the server *also* answers on **IPv4 19133**, which the UDP Service does expose — measured. So
  `19133` in `server.properties` genuinely does mean "slot 1 is taken", contrary to what this file
  previously said.

## Policies

`namespace.yaml` sets only `gateway-access=shared` (platform namespace, like headlamp/homepage), so
Kyverno does not auto-generate the app baseline. `networkpolicy.yaml` ships it explicitly (defense in
depth): default-deny (DNS-only egress) + Cilium allows — Gateway→sidecar `:4180`, world/LAN→game
ports, host→probe ports, and egress `world` (server-jar downloads + the OIDC path to Keycloak via the
Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subject to network policy.

## Verification

- `make -C ci gate` — green (hand-authored platform manifests; render-golden unaffected).
- `kustomize build` of `platform/crafty`, `clusters/base`, `clusters/dev` — clean.

### Human tests (post-deploy)

1. **Browser OIDC round-trip:** incognito → `https://crafty.home.keithjackson.dev` → expect a 302 to
   `https://keycloak.home.keithjackson.dev/realms/homelab/...` (NOT the Crafty UI). Log in as a
   **platform-admins** user (MFA/OTP enforced) → land on Crafty's own login → sign in with Crafty's
   local admin. A **non-platform-admins** user → **403** (authenticated but not authorized).
2. **Pod health / fsGroup:** `kubectl -n crafty get pod` Ready 2/2; Crafty writes its DB under
   `/crafty/app/config` and `/crafty/servers` without EACCES.
3. **Server-create + game reach (Java):** in the Crafty UI create a Java server bound to `25565`;
   start it; connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or
   `192.168.2.22:25565`) from the LAN and confirm the world loads.
4. **Bedrock: the connect procedure (VERIFIED end-to-end 2026-09-29 — a player has joined).**
   Creating and starting the server is **not sufficient**. Do these in order; each step has a distinct
   failure signature:

   1. **`transport=raknet`** in `server.properties`. The vendor default is `nethernet`, which opens
      **no UDP listener at all**.
   2. **A live UDP listener:** `grep 4ABC /proc/net/udp /proc/net/udp6` → a wildcard bind. Under
      `nethernet` you instead see `4ABC` on **TCP**, which nothing here exposes.
   3. **A pong from off-cluster:** a RakNet Unconnected Ping to `192.168.2.22:19132` returns a `0x1c`
      packet whose MOTD string starts `MCPE;`. This proves the VIP, Service and policy path.
   4. **An allowlist entry for the player's Xbox Gamertag.** `allow-list=true` with
      `allowlist.json == []` refuses **every** player with *"You are not invited to play on this
      server"* while the server is otherwise perfectly reachable. In the panel console:
      `allowlist add <Gamertag>`.
   5. **Add Server** by hand in a Windows/mobile client: `crafty-mc.home.keithjackson.dev`, port
      `19132`. It will never appear in the LAN/Friends tab, and a console cannot reach it at all.

   **Two log messages appear on a fully working server and must be ignored:** BDS's
   `================ TRANSPORT TYPE ERROR ===================` banner (advisory — see *Trap 2*), and
   Crafty's perpetual `Bedrock ping returned a string with too few fields`. Neither indicates a fault;
   chasing them is what cost two sessions. The **real** failure signatures are `Got timeout while
   issuing bedrock ping` (no listener) and *"You are not invited"* (allowlist).

## Runtime state: the servers themselves are NOT in git

As of 2026-09-29 **two** servers exist (both measured live, `select * from servers`):

| Name | Type | Port | UUID | `auto_start` |
|---|---|---|---|---|
| `homelab` | `minecraft-java` (Paper 26.2, 2 GiB heap) | TCP 25565 | `13c7b85d-00bb-4a58-8c86-2433ca47891c` | **1** |
| `K3s-Bedrock` | `minecraft-bedrock` | UDP 19132 | `c5837301-12b1-404e-b50e-70eafb37226b` | **0** ⚠ |

`K3s-Bedrock` was created 2026-09-29 20:51 and **still** has `auto_start=0` **and**
`crash_detection=0` — re-measured 2026-09-29 21:41 straight out of `crafty.sqlite` (read-only), so the
create-API trap documented at the bottom of this file is still live and still unfixed. **It will not
come back after a pod restart**, and merging any change to `deployment.yaml` restarts the pod
(`strategy: Recreate`). PATCH both to true and read them back.

`homelab` was created through Crafty's v2 API, and **Mojang's
EULA (<https://aka.ms/MinecraftEULA>) was accepted on the operator's behalf** to let it start; see the
DECISIONS-LOG entry of that date.

Everything in this directory describes only the *capacity* to run Minecraft servers. The servers are
application state on the `crafty-data` Longhorn PVC, in two places, and **Flux cannot recreate either**:

- `/crafty/app/config/db/` (`config` subPath) — Crafty's SQLite row: name, port, jar, launch command,
  heap sizes, `auto_start` / `crash_detection`.
- `/crafty/servers/<uuid>/` (`servers` subPath) — `paper.jar`, `eula.txt`, `server.properties`, and
  **the generated world**. The jar is re-downloadable; the world is not.
  **`transport=raknet` lives here too.** It is the one setting that makes Bedrock reachable at all (see
  *the transport trap*), it is not in git, Flux cannot restore it, and a freshly created or reinstalled
  Bedrock server gets the vendor default `transport=nethernet` back — i.e. it silently regresses to
  "nothing bound on 19132/udp". Re-apply and re-ping after any Bedrock server recreate or BDS update.

So a rebuild from git alone yields a healthy but **empty** Crafty with nothing listening on 25565.
Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH
`auto_start` to true — see the next note. The world starts over. **Treat this PVC as backup-relevant.**

### Trap: the create API silently ignores `autostart`

`POST /api/v2/servers` accepts and validates `autostart` and `crashdetection`, but never stores them —
the persisted columns are `auto_start` and `crash_detection`. After a create that asked for `true`,
both read back `false`. Set them afterwards with `PATCH /api/v2/servers/<uuid>` and **read them back to
confirm**, or the server will not come up after a pod restart.