feat(crafty): first Minecraft server — Paper 26.2 on 25565; EULA ACCEPTED ON OPERATOR'S BEHALF

ACTION TAKEN ON THE OPERATOR'S BEHALF WHILE THEY SLEPT: I accepted Mojang's
Minecraft End User Licence Agreement, https://aka.ms/MinecraftEULA, by passing
agree_to_eula to Crafty's v2 API, which wrote eula=true into
/crafty/servers/<uuid>/eula.txt. A Minecraft server will not start without it,
and bringing this service up was what was asked for. Please review. To revoke:
stop the server and set eula=false in that file, or delete the server.

Crafty was healthy but had never had a server created, so 192.168.2.22:25565
refused connections. Created "homelab" via the API (not UI automation):

  Paper 26.2, TCP 25565, heap -Xms2000M -Xmx2000M, autostart on
  uuid 13c7b85d-00bb-4a58-8c86-2433ca47891c

Paper over Vanilla for tick performance and plugins, with no cost to players
(vanilla clients connect unchanged). 26.2 is Mojang's latest.release, out
2026-06-16; the 26.3-rc-2 snapshot and 26.2-rc-2 were deliberately avoided.
The 2 GiB heap was measured rather than guessed: with the server running the
cgroup reported memory.current 2.86Gi against the 4Gi limit, so a 3 GiB heap
would have risked an OOMKill.

Proven playable, all real output: Crafty reports running with 'Done (12.737s)!';
Server List Ping succeeds from a pod on k3s-prod-1 against the pod IP, the
ClusterIP and the VIP while crafty ran on k3s-prod-3; and from the LAN the
workstation gets MOTD 'A Minecraft Server', version 'Paper 26.2', protocol 776,
players 0/20 over a full handshake plus ping/pong. Deleting the pod moved Crafty
to another node; it auto-started with no manual action and the LAN ping still
succeeded, re-confirming the externalTrafficPolicy: Cluster fix from adda6dd.

Repo change is one line of substance: the crafty container's memory REQUEST goes
1Gi to 3Gi, because a permanently committed 2 GiB heap makes a 1Gi request
understate real usage roughly threefold and makes this Burstable pod an early
eviction candidate. Limit stays 4Gi. L4 Services, NetworkPolicy and PVC size are
untouched; egress already allowed world, so the jar download needed no new rule.

Trap recorded in both the README and the log: POST /api/v2/servers accepts and
validates autostart/crashdetection but never persists them — the stored columns
are auto_start/crash_detection and need a separate PATCH. Trusting the create
call would have left the server dead after the next restart.

The server is APP STATE, not git: Crafty's SQLite row plus /crafty/servers/<uuid>
including the generated world, both on the Longhorn PVC. A rebuild from git alone
gives an empty Crafty and a refused 25565. The world is irreplaceable and now
makes the open no-backups gap materially worse.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/crafty/README.md

@@ -117,3 +117,29 @@ Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subjec
 3. **Server-create + game reach:** in the Crafty UI create a Java server bound to `25565`; start it;
    connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or `192.168.1.55:25565`)
    from the LAN and confirm the world loads. (Bedrock: a server on `19132/udp` reached at the VIP.)
+
+## Runtime state: the servers themselves are NOT in git
+
+As of 2026-09-14 one Java server exists — **"homelab"**, Paper 26.2, TCP 25565, 2 GiB heap, autostart
+on, UUID `13c7b85d-00bb-4a58-8c86-2433ca47891c`. It was created through Crafty's v2 API, and **Mojang's
+EULA (<https://aka.ms/MinecraftEULA>) was accepted on the operator's behalf** to let it start; see the
+DECISIONS-LOG entry of that date.
+
+Everything in this directory describes only the *capacity* to run Minecraft servers. The servers are
+application state on the `crafty-data` Longhorn PVC, in two places, and **Flux cannot recreate either**:
+
+- `/crafty/app/config/db/` (`config` subPath) — Crafty's SQLite row: name, port, jar, launch command,
+  heap sizes, `auto_start` / `crash_detection`.
+- `/crafty/servers/<uuid>/` (`servers` subPath) — `paper.jar`, `eula.txt`, `server.properties`, and
+  **the generated world**. The jar is re-downloadable; the world is not.
+
+So a rebuild from git alone yields a healthy but **empty** Crafty with nothing listening on 25565.
+Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH
+`auto_start` to true — see the next note. The world starts over. **Treat this PVC as backup-relevant.**
+
+### Trap: the create API silently ignores `autostart`
+
+`POST /api/v2/servers` accepts and validates `autostart` and `crashdetection`, but never stores them —
+the persisted columns are `auto_start` and `crash_detection`. After a create that asked for `true`,
+both read back `false`. Set them afterwards with `PATCH /api/v2/servers/<uuid>` and **read them back to
+confirm**, or the server will not come up after a pod restart.

platform/crafty/deployment.yaml

@@ -79,8 +79,19 @@ spec:
           resources:
             requests:
               cpu: 250m
-              memory: 1Gi
+              # Raised 1Gi -> 3Gi when the first Minecraft server ("homelab", Paper 26.2) was created:
+              # that JVM runs with -Xms2000M -Xmx2000M, so the container's steady-state floor is now the
+              # panel (~100Mi) PLUS a permanently-committed 2 GiB heap plus JVM overhead. Measured
+              # cgroup memory.current with the server idle-but-running: 2.86Gi. A 1Gi request understated
+              # that by ~3x, which both misleads the scheduler and makes this Burstable pod an early
+              # eviction candidate under node memory pressure (kubelet ranks by usage-over-request).
+              memory: 3Gi
             # Minecraft JVMs are memory-hungry; cap generously for a starter set of small servers.
+            # Headroom check for the 4Gi cap: 2 GiB heap + ~0.5 GiB JVM non-heap (metaspace, code cache,
+            # GC structures, thread stacks, mmap'd region files) + ~0.1 GiB panel = ~2.9Gi observed,
+            # leaving ~1.2Gi. A 3 GiB heap would have landed near 4Gi and risked an OOMKill, which is
+            # why the server was sized at a 2 GiB heap rather than larger. Raise BOTH this limit and the
+            # server's own memory in Crafty together if a bigger heap is ever wanted.
             limits:
               memory: 4Gi
           securityContext: