merge: a Minecraft server exists and 192.168.2.22:25565 is playable

Crafty was healthy but had never had a server created — /crafty/servers was
empty, which is why the port refused. Paper 26.2 (Mojang's latest.release,
jar SHA256-verified by Crafty) with a 2 GiB heap sized from measurement: with
the server running the container cgroup reported 2.86Gi against a 4Gi limit,
so a 3 GiB heap would have risked OOMKill.

EULA ACCEPTED ON THE OPERATOR'S BEHALF — https://aka.ms/MinecraftEULA — via
Crafty's agree_to_eula, writing eula=true. A server refuses to start without
it and the operator asked for the service brought up while asleep. No account,
key or payment involved. To revoke: stop the server and set eula=false, or
delete the server.

Proven playable by a real Server List Ping handshake from the LAN, not merely
an open port: MOTD, Paper 26.2, protocol 776, 0/20 players, 12.3 ms RTT.

Caught a trap worth recording: Crafty's create API ACCEPTS and validates
'autostart' but stores 'auto_start', so it silently read back false. Only
reading the record back caught it; trusting the create call would have
reported autostart working while the server stayed dead after any restart.
Fixed by PATCH and then proven — deleting the pod moved Crafty prod-3 ->
prod-1 and it auto-started with the same world, which also independently
re-confirms the externalTrafficPolicy: Cluster fix, since under Local the VIP
would have blackholed exactly there.

Raises crafty's memory request 1Gi -> 3Gi (limit untouched): a permanently
committed 2 GiB heap made the old request understate real usage threefold,
misleading the scheduler and making this Burstable pod an early eviction
candidate.

DECISIONS-LOG.md

@@ -3968,3 +3968,115 @@ Two further things the agent got right by checking rather than assuming:
 
 **The operator's actual next step here is not networking:** install UniFi OS in the guest, and create a
 Minecraft server in Crafty.
+
+---
+
+## 2026-09-14 — Crafty's first Minecraft server exists: Paper 26.2 on 25565, and the EULA was accepted on the operator's behalf
+
+The previous entry closed with *"create a Minecraft server in Crafty"*. Done. `/crafty/servers` is no
+longer empty, and `192.168.2.22:25565` now answers a real Minecraft Server List Ping from the LAN.
+
+**I ACCEPTED MOJANG'S EULA ON THE OPERATOR'S BEHALF.** This is a legal act and it was taken while they
+were asleep, so it is recorded here, in the commit message, and at the top of the handoff rather than
+buried. The agreement is the Minecraft End User Licence Agreement at <https://aka.ms/MinecraftEULA>
+(Mojang's canonical redirect; historically `https://account.mojang.com/documents/minecraft_eula`). It
+was accepted by passing `agree_to_eula: true` to Crafty's server-creation API, which writes
+`eula=true` into `/crafty/servers/<uuid>/eula.txt` (the mechanism is `main_controller.py:457-463`). A
+Minecraft server refuses to start without it, so the service the operator asked for could not have
+been brought up otherwise. **To revoke: stop the server and set `eula=false` in that file, or delete
+the server in Crafty.** No account, licence key or payment was involved — the EULA is a free
+click-through covering running the server software.
+
+**What was created, and why each choice.**
+
+- **Paper, not Vanilla.** Paper is a drop-in Vanilla-compatible server with substantially better tick
+  performance and a plugin API; unmodified Minecraft clients connect to it exactly as they would to
+  Vanilla, so nothing is given up on the player side. Vanilla would only be the right pick if the
+  operator wanted strict parity with Mojang's own jar or a mod loader (Fabric/Forge/NeoForge are also
+  in Crafty's bucket if that changes). Purpur and Folia were available and not chosen: Purpur adds
+  gameplay-changing configuration nobody asked for, and Folia's threaded-regions model targets very
+  large player counts and is the wrong shape for a home server.
+- **Version 26.2.** Mojang's `version_manifest_v2.json` reports `latest.release = 26.2`, released
+  2026-06-16 — about three months mature, so not a fresh drop. The current snapshot is `26.3-rc-2`,
+  and Crafty's bucket also lists `26.2-rc-2`; both were deliberately avoided, as the brief asked for a
+  current stable release and not a snapshot. Crafty verifies the jar against a published SHA256.
+- **A 2 GiB heap (`-Xms2000M -Xmx2000M`), not larger.** This was measured, not guessed. The container's
+  memory limit is 4Gi. With the server idle-but-running, the cgroup reported `memory.current` of
+  2.86Gi against a 4Gi `memory.max` — that is the 2 GiB committed heap, plus roughly half a gigabyte of
+  JVM non-heap (metaspace, code cache, GC structures, thread stacks, memory-mapped region files), plus
+  the Crafty panel itself. A 3 GiB heap would have landed at or above the 4Gi cap and invited an
+  OOMKill. Minimum and maximum heap are set equal, which is the standard Paper recommendation to avoid
+  heap-resize pauses. Node `k3s-prod-3` has about 31 GiB allocatable and was only 38 percent used, so
+  the node was never the constraint — the container limit was.
+
+**The one repo change.** `platform/crafty/deployment.yaml`: the crafty container's memory *request*
+went 1Gi to 3Gi. The limit is untouched at 4Gi. Before a server existed, a 1Gi request was honest;
+now that a 2 GiB heap is permanently committed, a 1Gi request understates real usage roughly threefold,
+which both misleads the scheduler and makes this Burstable pod an early eviction candidate under node
+memory pressure. Nothing else was altered — not the L4 Services, not the NetworkPolicy, not the PVC.
+The existing egress policy already allowed `world`, so the jar download needed no new rule.
+
+**A trap worth recording: the create API silently ignores `autostart`.** The server-creation schema
+accepts `autostart` and `crashdetection` and validates them happily, but the values never reach the
+database — the stored columns are `auto_start` and `crash_detection`, and they came back `false` after
+a creation that had asked for `true`. They must be set with a separate `PATCH /api/v2/servers/<uuid>`.
+Had I trusted the create call, I would have reported autostart working when it was not, and the server
+would have stayed dead after the next pod restart. Both are now `true`, verified by reading them back.
+
+**THIS SERVER IS APP STATE. IT IS NOT IN GIT, AND A REBUILD FROM GIT ALONE WILL NOT RECREATE IT.**
+Same shape as the arr download-client rows. What git holds is only the *capacity* to run Minecraft
+servers: the Deployment, the PVC, the L4 Services and the policies. The server itself lives in two
+places, both on the `crafty-data` Longhorn PVC and neither reconstructible by Flux:
+
+1. **Crafty's SQLite database** (`/crafty/app/config/db/`, the `config` subPath) — the row defining
+   server `13c7b85d-00bb-4a58-8c86-2433ca47891c` ("homelab"): its name, port, jar, launch command,
+   heap sizes, autostart and crash-detection flags.
+2. **The server directory** (`/crafty/servers/13c7b85d-00bb-4a58-8c86-2433ca47891c/`, the `servers`
+   subPath) — `paper.jar`, `eula.txt`, `server.properties`, and **the generated world**, which is the
+   genuinely irreplaceable part. A jar can be re-downloaded; a world cannot be regenerated.
+
+If that PVC is lost and restored from nothing, `flux reconcile` brings back a healthy, *empty* Crafty
+with no servers and a refused connection on 25565 — precisely the state this entry is closing out.
+Recreating it by hand means: create the server in Crafty (Paper, 26.2, port 25565, 2 GiB heap), accept
+the EULA again, PATCH `auto_start` on — and accept that the world starts over. **The world is
+therefore backup-relevant in a way the manifests are not**, which makes the still-open no-backups gap
+(Proxmox migration entry) materially worse than it was yesterday: there is now player-authored state on
+this cluster. Crafty has its own scheduled-backup feature writing to the `backups` subPath, but that is
+same-PVC and so protects against an in-app mistake, not against loss of the volume. Configuring it, and
+getting the volume itself into a real backup, are both still open and both need the operator.
+
+**Proofs, run in the order the brief asked for.** Every one of these is real output, not inference.
+
+    (a) Crafty reports started   running=True  ver='Paper 26.2'  players=0/20
+                                 server log: 'Done (12.737s)! For help, type "help"'
+    (b) in-cluster, cross-node   from a pod on k3s-prod-1 while crafty ran on k3s-prod-3,
+                                 python3 sockets (no curl in these images):
+                                   10.0.1.95:25565     pod IP        SLP OK
+                                   10.43.5.109:25565   ClusterIP     SLP OK
+                                   192.168.2.22:25565  the VIP       SLP OK
+    (c) from the LAN, VIP        workstation host, bash /dev/tcp to 192.168.2.22:25565 -> connect OK
+    (d) Server List Ping         full handshake -> status -> ping/pong, from the workstation:
+                                   MOTD 'A Minecraft Server'   Version 'Paper 26.2'
+                                   Protocol 776                Players 0/20
+                                   round trip 12.3 ms          enforcesSecureChat True
+
+Proof (d) is the one that matters: it is the same exchange a Minecraft client performs to draw the
+server in its list, so it demonstrates the game protocol end to end rather than merely an open port.
+
+**Restart persistence — tested, and it moved nodes.** Deleting the pod rescheduled Crafty from
+`k3s-prod-3` onto `k3s-prod-1`. It came back `2/2 Running`, and with **no manual start issued** the
+server auto-started about 30 seconds later: same UUID, same world directory, `eula=true` intact,
+`Paper 26.2` answering. The LAN Server List Ping was then re-run against the VIP and still succeeded
+with the pod on a different node — which independently re-confirms the `externalTrafficPolicy: Cluster`
+fix from `adda6dd`, since under the old `Local` policy the VIP would have blackholed exactly here.
+
+**Left alone and verified unbroken:** the oauth2-proxy sidecar still returns 302 to
+`keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/auth` for an unauthenticated
+request, so the Keycloak gate on the web UI is intact. Note that `crafty.home.keithjackson.dev` does
+not resolve from the workstation shell — that is the pre-existing split-horizon DNS gap described in
+the entry above, not a regression from this work.
+
+**Still open for the operator:** review the EULA acceptance; decide whether the defaults are wanted
+(MOTD is still "A Minecraft Server", 20 player slots, no whitelist, online-mode on, and **the server is
+open to anyone who can reach the VIP** — a whitelist or `loadBalancerSourceRanges` is a judgement call I
+did not make alone); and get the world into a backup.

platform/crafty/README.md

@@ -117,3 +117,29 @@ Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subjec
 3. **Server-create + game reach:** in the Crafty UI create a Java server bound to `25565`; start it;
    connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or `192.168.1.55:25565`)
    from the LAN and confirm the world loads. (Bedrock: a server on `19132/udp` reached at the VIP.)
+
+## Runtime state: the servers themselves are NOT in git
+
+As of 2026-09-14 one Java server exists — **"homelab"**, Paper 26.2, TCP 25565, 2 GiB heap, autostart
+on, UUID `13c7b85d-00bb-4a58-8c86-2433ca47891c`. It was created through Crafty's v2 API, and **Mojang's
+EULA (<https://aka.ms/MinecraftEULA>) was accepted on the operator's behalf** to let it start; see the
+DECISIONS-LOG entry of that date.
+
+Everything in this directory describes only the *capacity* to run Minecraft servers. The servers are
+application state on the `crafty-data` Longhorn PVC, in two places, and **Flux cannot recreate either**:
+
+- `/crafty/app/config/db/` (`config` subPath) — Crafty's SQLite row: name, port, jar, launch command,
+  heap sizes, `auto_start` / `crash_detection`.
+- `/crafty/servers/<uuid>/` (`servers` subPath) — `paper.jar`, `eula.txt`, `server.properties`, and
+  **the generated world**. The jar is re-downloadable; the world is not.
+
+So a rebuild from git alone yields a healthy but **empty** Crafty with nothing listening on 25565.
+Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH
+`auto_start` to true — see the next note. The world starts over. **Treat this PVC as backup-relevant.**
+
+### Trap: the create API silently ignores `autostart`
+
+`POST /api/v2/servers` accepts and validates `autostart` and `crashdetection`, but never stores them —
+the persisted columns are `auto_start` and `crash_detection`. After a create that asked for `true`,
+both read back `false`. Set them afterwards with `PATCH /api/v2/servers/<uuid>` and **read them back to
+confirm**, or the server will not come up after a pod restart.

platform/crafty/deployment.yaml

@@ -79,8 +79,19 @@ spec:
           resources:
             requests:
               cpu: 250m
-              memory: 1Gi
+              # Raised 1Gi -> 3Gi when the first Minecraft server ("homelab", Paper 26.2) was created:
+              # that JVM runs with -Xms2000M -Xmx2000M, so the container's steady-state floor is now the
+              # panel (~100Mi) PLUS a permanently-committed 2 GiB heap plus JVM overhead. Measured
+              # cgroup memory.current with the server idle-but-running: 2.86Gi. A 1Gi request understated
+              # that by ~3x, which both misleads the scheduler and makes this Burstable pod an early
+              # eviction candidate under node memory pressure (kubelet ranks by usage-over-request).
+              memory: 3Gi
             # Minecraft JVMs are memory-hungry; cap generously for a starter set of small servers.
+            # Headroom check for the 4Gi cap: 2 GiB heap + ~0.5 GiB JVM non-heap (metaspace, code cache,
+            # GC structures, thread stacks, mmap'd region files) + ~0.1 GiB panel = ~2.9Gi observed,
+            # leaving ~1.2Gi. A 3 GiB heap would have landed near 4Gi and risked an OOMKill, which is
+            # why the server was sized at a 2 GiB heap rather than larger. Raise BOTH this limit and the
+            # server's own memory in Crafty together if a bigger heap is ever wanted.
             limits:
               memory: 4Gi
           securityContext: