diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index f0a1a00..43cd1fa 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3968,3 +3968,115 @@ Two further things the agent got right by checking rather than assuming: **The operator's actual next step here is not networking:** install UniFi OS in the guest, and create a Minecraft server in Crafty. + +--- + +## 2026-09-14 — Crafty's first Minecraft server exists: Paper 26.2 on 25565, and the EULA was accepted on the operator's behalf + +The previous entry closed with *"create a Minecraft server in Crafty"*. Done. `/crafty/servers` is no +longer empty, and `192.168.2.22:25565` now answers a real Minecraft Server List Ping from the LAN. + +**I ACCEPTED MOJANG'S EULA ON THE OPERATOR'S BEHALF.** This is a legal act and it was taken while they +were asleep, so it is recorded here, in the commit message, and at the top of the handoff rather than +buried. The agreement is the Minecraft End User Licence Agreement at +(Mojang's canonical redirect; historically `https://account.mojang.com/documents/minecraft_eula`). It +was accepted by passing `agree_to_eula: true` to Crafty's server-creation API, which writes +`eula=true` into `/crafty/servers//eula.txt` (the mechanism is `main_controller.py:457-463`). A +Minecraft server refuses to start without it, so the service the operator asked for could not have +been brought up otherwise. **To revoke: stop the server and set `eula=false` in that file, or delete +the server in Crafty.** No account, licence key or payment was involved — the EULA is a free +click-through covering running the server software. + +**What was created, and why each choice.** + +- **Paper, not Vanilla.** Paper is a drop-in Vanilla-compatible server with substantially better tick + performance and a plugin API; unmodified Minecraft clients connect to it exactly as they would to + Vanilla, so nothing is given up on the player side. Vanilla would only be the right pick if the + operator wanted strict parity with Mojang's own jar or a mod loader (Fabric/Forge/NeoForge are also + in Crafty's bucket if that changes). Purpur and Folia were available and not chosen: Purpur adds + gameplay-changing configuration nobody asked for, and Folia's threaded-regions model targets very + large player counts and is the wrong shape for a home server. +- **Version 26.2.** Mojang's `version_manifest_v2.json` reports `latest.release = 26.2`, released + 2026-06-16 — about three months mature, so not a fresh drop. The current snapshot is `26.3-rc-2`, + and Crafty's bucket also lists `26.2-rc-2`; both were deliberately avoided, as the brief asked for a + current stable release and not a snapshot. Crafty verifies the jar against a published SHA256. +- **A 2 GiB heap (`-Xms2000M -Xmx2000M`), not larger.** This was measured, not guessed. The container's + memory limit is 4Gi. With the server idle-but-running, the cgroup reported `memory.current` of + 2.86Gi against a 4Gi `memory.max` — that is the 2 GiB committed heap, plus roughly half a gigabyte of + JVM non-heap (metaspace, code cache, GC structures, thread stacks, memory-mapped region files), plus + the Crafty panel itself. A 3 GiB heap would have landed at or above the 4Gi cap and invited an + OOMKill. Minimum and maximum heap are set equal, which is the standard Paper recommendation to avoid + heap-resize pauses. Node `k3s-prod-3` has about 31 GiB allocatable and was only 38 percent used, so + the node was never the constraint — the container limit was. + +**The one repo change.** `platform/crafty/deployment.yaml`: the crafty container's memory *request* +went 1Gi to 3Gi. The limit is untouched at 4Gi. Before a server existed, a 1Gi request was honest; +now that a 2 GiB heap is permanently committed, a 1Gi request understates real usage roughly threefold, +which both misleads the scheduler and makes this Burstable pod an early eviction candidate under node +memory pressure. Nothing else was altered — not the L4 Services, not the NetworkPolicy, not the PVC. +The existing egress policy already allowed `world`, so the jar download needed no new rule. + +**A trap worth recording: the create API silently ignores `autostart`.** The server-creation schema +accepts `autostart` and `crashdetection` and validates them happily, but the values never reach the +database — the stored columns are `auto_start` and `crash_detection`, and they came back `false` after +a creation that had asked for `true`. They must be set with a separate `PATCH /api/v2/servers/`. +Had I trusted the create call, I would have reported autostart working when it was not, and the server +would have stayed dead after the next pod restart. Both are now `true`, verified by reading them back. + +**THIS SERVER IS APP STATE. IT IS NOT IN GIT, AND A REBUILD FROM GIT ALONE WILL NOT RECREATE IT.** +Same shape as the arr download-client rows. What git holds is only the *capacity* to run Minecraft +servers: the Deployment, the PVC, the L4 Services and the policies. The server itself lives in two +places, both on the `crafty-data` Longhorn PVC and neither reconstructible by Flux: + +1. **Crafty's SQLite database** (`/crafty/app/config/db/`, the `config` subPath) — the row defining + server `13c7b85d-00bb-4a58-8c86-2433ca47891c` ("homelab"): its name, port, jar, launch command, + heap sizes, autostart and crash-detection flags. +2. **The server directory** (`/crafty/servers/13c7b85d-00bb-4a58-8c86-2433ca47891c/`, the `servers` + subPath) — `paper.jar`, `eula.txt`, `server.properties`, and **the generated world**, which is the + genuinely irreplaceable part. A jar can be re-downloaded; a world cannot be regenerated. + +If that PVC is lost and restored from nothing, `flux reconcile` brings back a healthy, *empty* Crafty +with no servers and a refused connection on 25565 — precisely the state this entry is closing out. +Recreating it by hand means: create the server in Crafty (Paper, 26.2, port 25565, 2 GiB heap), accept +the EULA again, PATCH `auto_start` on — and accept that the world starts over. **The world is +therefore backup-relevant in a way the manifests are not**, which makes the still-open no-backups gap +(Proxmox migration entry) materially worse than it was yesterday: there is now player-authored state on +this cluster. Crafty has its own scheduled-backup feature writing to the `backups` subPath, but that is +same-PVC and so protects against an in-app mistake, not against loss of the volume. Configuring it, and +getting the volume itself into a real backup, are both still open and both need the operator. + +**Proofs, run in the order the brief asked for.** Every one of these is real output, not inference. + + (a) Crafty reports started running=True ver='Paper 26.2' players=0/20 + server log: 'Done (12.737s)! For help, type "help"' + (b) in-cluster, cross-node from a pod on k3s-prod-1 while crafty ran on k3s-prod-3, + python3 sockets (no curl in these images): + 10.0.1.95:25565 pod IP SLP OK + 10.43.5.109:25565 ClusterIP SLP OK + 192.168.2.22:25565 the VIP SLP OK + (c) from the LAN, VIP workstation host, bash /dev/tcp to 192.168.2.22:25565 -> connect OK + (d) Server List Ping full handshake -> status -> ping/pong, from the workstation: + MOTD 'A Minecraft Server' Version 'Paper 26.2' + Protocol 776 Players 0/20 + round trip 12.3 ms enforcesSecureChat True + +Proof (d) is the one that matters: it is the same exchange a Minecraft client performs to draw the +server in its list, so it demonstrates the game protocol end to end rather than merely an open port. + +**Restart persistence — tested, and it moved nodes.** Deleting the pod rescheduled Crafty from +`k3s-prod-3` onto `k3s-prod-1`. It came back `2/2 Running`, and with **no manual start issued** the +server auto-started about 30 seconds later: same UUID, same world directory, `eula=true` intact, +`Paper 26.2` answering. The LAN Server List Ping was then re-run against the VIP and still succeeded +with the pod on a different node — which independently re-confirms the `externalTrafficPolicy: Cluster` +fix from `adda6dd`, since under the old `Local` policy the VIP would have blackholed exactly here. + +**Left alone and verified unbroken:** the oauth2-proxy sidecar still returns 302 to +`keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/auth` for an unauthenticated +request, so the Keycloak gate on the web UI is intact. Note that `crafty.home.keithjackson.dev` does +not resolve from the workstation shell — that is the pre-existing split-horizon DNS gap described in +the entry above, not a regression from this work. + +**Still open for the operator:** review the EULA acceptance; decide whether the defaults are wanted +(MOTD is still "A Minecraft Server", 20 player slots, no whitelist, online-mode on, and **the server is +open to anyone who can reach the VIP** — a whitelist or `loadBalancerSourceRanges` is a judgement call I +did not make alone); and get the world into a backup. diff --git a/platform/crafty/README.md b/platform/crafty/README.md index f01c9ac..aeca91c 100644 --- a/platform/crafty/README.md +++ b/platform/crafty/README.md @@ -117,3 +117,29 @@ Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subjec 3. **Server-create + game reach:** in the Crafty UI create a Java server bound to `25565`; start it; connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or `192.168.1.55:25565`) from the LAN and confirm the world loads. (Bedrock: a server on `19132/udp` reached at the VIP.) + +## Runtime state: the servers themselves are NOT in git + +As of 2026-09-14 one Java server exists — **"homelab"**, Paper 26.2, TCP 25565, 2 GiB heap, autostart +on, UUID `13c7b85d-00bb-4a58-8c86-2433ca47891c`. It was created through Crafty's v2 API, and **Mojang's +EULA () was accepted on the operator's behalf** to let it start; see the +DECISIONS-LOG entry of that date. + +Everything in this directory describes only the *capacity* to run Minecraft servers. The servers are +application state on the `crafty-data` Longhorn PVC, in two places, and **Flux cannot recreate either**: + +- `/crafty/app/config/db/` (`config` subPath) — Crafty's SQLite row: name, port, jar, launch command, + heap sizes, `auto_start` / `crash_detection`. +- `/crafty/servers//` (`servers` subPath) — `paper.jar`, `eula.txt`, `server.properties`, and + **the generated world**. The jar is re-downloadable; the world is not. + +So a rebuild from git alone yields a healthy but **empty** Crafty with nothing listening on 25565. +Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH +`auto_start` to true — see the next note. The world starts over. **Treat this PVC as backup-relevant.** + +### Trap: the create API silently ignores `autostart` + +`POST /api/v2/servers` accepts and validates `autostart` and `crashdetection`, but never stores them — +the persisted columns are `auto_start` and `crash_detection`. After a create that asked for `true`, +both read back `false`. Set them afterwards with `PATCH /api/v2/servers/` and **read them back to +confirm**, or the server will not come up after a pod restart. diff --git a/platform/crafty/deployment.yaml b/platform/crafty/deployment.yaml index 6c1b7d6..4fdabe9 100644 --- a/platform/crafty/deployment.yaml +++ b/platform/crafty/deployment.yaml @@ -79,8 +79,19 @@ spec: resources: requests: cpu: 250m - memory: 1Gi + # Raised 1Gi -> 3Gi when the first Minecraft server ("homelab", Paper 26.2) was created: + # that JVM runs with -Xms2000M -Xmx2000M, so the container's steady-state floor is now the + # panel (~100Mi) PLUS a permanently-committed 2 GiB heap plus JVM overhead. Measured + # cgroup memory.current with the server idle-but-running: 2.86Gi. A 1Gi request understated + # that by ~3x, which both misleads the scheduler and makes this Burstable pod an early + # eviction candidate under node memory pressure (kubelet ranks by usage-over-request). + memory: 3Gi # Minecraft JVMs are memory-hungry; cap generously for a starter set of small servers. + # Headroom check for the 4Gi cap: 2 GiB heap + ~0.5 GiB JVM non-heap (metaspace, code cache, + # GC structures, thread stacks, mmap'd region files) + ~0.1 GiB panel = ~2.9Gi observed, + # leaving ~1.2Gi. A 3 GiB heap would have landed near 4Gi and risked an OOMKill, which is + # why the server was sized at a 2 GiB heap rather than larger. Raise BOTH this limit and the + # server's own memory in Crafty together if a bigger heap is ever wanted. limits: memory: 4Gi securityContext: