diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 588a5f3..f48751f 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -5109,3 +5109,179 @@ next L2/L3 piece on top of unmerged branches would create a rebase tangle for no cannot land any faster. The next piece is specified and ready to start once these are merged: `docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its boundary landed in #13. + +## 2026-09-29 — Bedrock "cannot connect on 19132": the platform delivered the packets, the server never opened the socket + +**Branch:** `fix/crafty-bedrock-19132` → PR to `main`. **Read-only diagnosis; nothing was applied.** +**Gate:** see "Gate" below — quote it, do not round it up. + +### The report, and what actually refuses + +Reported as "users are unable to connect to Crafty Minecraft servers listening on UDP port 19132 +(Bedrock, slot 0)". No client type, no error text, no history. + +**Nothing was listening on UDP 19132.** A Bedrock server exists and its process is alive, but it is +not speaking RakNet, so it never binds a UDP port. The entire network path below it is correct and was +measured as such, including *counted packets that arrived and were allowed*. + +The chain, each line a measured field with the command that produced it: + +1. **The server exists and runs.** `kubectl -n crafty exec … -c crafty -- ps -ef`: + `crafty 46689 1 ./bedrock_server`, started 20:54, cwd + `/crafty/servers/c5837301-12b1-404e-b50e-70eafb37226b`. Crafty's DB (`select * from servers` via + `/crafty/.venv/bin/python`, read-only URI) row 2: `server_name='K3s-Bedrock'`, + `type='minecraft-bedrock'`, `server_port=19132`, `created='2026-09-29 20:51:53'`. + So hypothesis "the operator never created a Bedrock server" — the PM's prior favourite, and mine — + is **false**. It was created ~20 minutes before I looked. +2. **Nothing is bound to UDP 19132.** `cat /proc/net/udp` and `/proc/net/udp6` in the crafty + container: **no `4ABC` local port in either family**. IPv4 UDP held nothing but an ephemeral + (`BBBF`=48063); IPv6 UDP nothing but `1D7F`=7551. Meanwhile `/proc/net/tcp6` line 0 is + `…:4ABC … 0A` — port 19132 is bound, as **TCP**, LISTEN. (Java's 25565 = `63DD` and Crafty's + 8443 = `20FB` are both there in `tcp6`, so the Java server is genuinely up.) +3. **Why: `transport=nethernet`.** `grep -v ^# server.properties` on the Bedrock server dir: + `server-port=19132`, `server-portv6=19133`, **`transport=nethernet`**, `enable-lan-visibility=true`. + The file's own shipped documentation, quoted verbatim: `# Allowed values: "raknet" or "nethernet"`, + and above `server-portv6`: `# This property is ignored when transport=nethernet and a dual-stack + socket will be opened on server-port instead`. NetherNet is the WebRTC/Xbox-signalled transport — + it opens no RakNet UDP listener. Consistent with the pod's outbound `tcp` entries to + `150.171.110.177:443` / `.178:443` (Microsoft) and the churning ephemeral UDP sockets. +4. **Nobody set that; it is the vendor default.** `server.properties` mtime `Sep 29 20:52` equals the + `bedrock_server` binary's — untouched since extraction — and `grep -rn transport + /crafty/app/classes/minecraft/` returns **nothing**: Crafty has no code that reads or writes this + property. So a freshly-downloaded BDS of this build **never** listens on `19132/udp`, and no amount + of Crafty-side configuration changes that. +5. **The failure is observable from inside the pod, from localhost.** + `grep raknet /crafty/logs/commander.log`, every 5–30 s since 21:02: + `WARNING - app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping to + 127.0.0.1:19132`. Crafty's only Bedrock health probe is `raknet_ping.py` (`ls + /crafty/app/classes/remote_stats/`), so the panel is asking the right question and getting silence + **over loopback**. That single line eliminates every network hypothesis at once. +6. **Real player traffic reached the pod and was permitted.** `cilium-dbg bpf policy get ` + on `cilium-wdh82` (k3s-prod-1): + `Allow Ingress reserved:world 19132/UDP … BYTES 27660 PACKETS 26`. + **26 packets of world traffic were allowed in on 19132/UDP.** The policy did not drop them; the + application had no socket to receive them. `reserved:remote-node 19132/UDP` is also an Allow (0 + bytes — the packets landed on the node that both announces the VIP and runs the pod, so no second + hop), which re-confirms the 2026-09-14 finding that `cluster` subsumes `remote-node` here. + +**Verdict: this is not a platform defect and not a manifest defect. It is the Bedrock Dedicated +Server's default `transport=nethernet`, plus a README that told the operator creating a server on +19132 was sufficient.** The repo-side defect is documentation, and that is what this branch changes. + +### The platform path, measured sound (so the operator can rule it out for good) + +| Claim | Measured | +|---|---| +| `crafty-l4-udp` really is `Cluster` | `kubectl get svc crafty-l4-udp -o yaml` → `spec.externalTrafficPolicy: Cluster` (the 2026-09-14 fix IS live, not just in git) | +| VIP allocated | `status.loadBalancer.ingress[0].ip: 192.168.2.22`, `ipMode: VIP`, condition `cilium.io/IPAMRequestSatisfied=True reason=satisfied` | +| Backends exist | EndpointSlice `crafty-l4-udp-68kws` → `10.0.0.130`, ports `19132,19133,19134` | +| Datapath frontend present | `cilium-dbg service list` on **both** k3s-prod-1 and k3s-prod-3: `192.168.2.22:19132/UDP LoadBalancer 1 => 10.0.0.130:19132/UDP (active)` — i.e. any node can serve it, which is exactly what `Cluster` is for | +| Policy admits it | datapath policy map, `Allow Ingress reserved:world 19132/UDP` (+ `host`, `remote-node`, `unmanaged`, `init`, `health`, `kube-apiserver`) | +| LAN DNS resolves | `nslookup crafty-mc.home.keithjackson.dev 127.0.0.1` in the pihole pod → `Address: 192.168.2.22` | +| Flux applied current `main` | Kustomization `crafty` `.status.lastAppliedRevision = main@sha1:20ea70d…` == GitRepository `flux-system` `.status.artifact.revision = main@sha1:20ea70d…` | + +### A real second finding: the shared VIP's two leases sit on DIFFERENT nodes + +`kubectl -n kube-system get lease … | grep l2announce`: + + cilium-l2announce-crafty-crafty-l4-tcp k3s-prod-3 + cilium-l2announce-crafty-crafty-l4-udp k3s-prod-1 + cilium-l2announce-unifi-unifi-l4-tcp k3s-prod-2 + cilium-l2announce-unifi-unifi-l4-udp k3s-prod-2 + cilium-l2announce-pihole-pihole-dns-tcp k3s-prod-3 + cilium-l2announce-pihole-pihole-dns-udp k3s-prod-3 + +Crafty is the **only** shared-VIP pair that is split: **two nodes ARP-announce `192.168.2.22` +simultaneously.** unifi and pihole happen to be co-located. This is the per-Service, backend-blind +election described in `service-l4.yaml` doing exactly what that comment says it does — and it is a +*coincidence* that the other two pairs agree, not a property. + +**I am not calling this the bug, and I am not changing anything for it**, because under +`externalTrafficPolicy: Cluster` it is benign by construction: `cilium-dbg service list` shows the +full frontend set with an active backend on **both** announcing nodes (table above), so whichever +node's ARP reply a client caches, the packet is forwarded to the pod. The cost is ARP-cache churn on +LAN clients, not a black hole — and this is why the 2026-09-14 `Local`→`Cluster` flip was the right +fix rather than a lucky one. **No new L0 check is warranted:** +`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/check.sh` already forbids +`Local` on any announced Service tree-wide, which is precisely the invariant that makes a split +harmless. An additional "shared-VIP pairs must be co-located" check would assert a *runtime* lease +placement a static check cannot see, and Cilium offers no field to pin it. + +### Hypotheses tested and discarded, so nobody re-runs them + +- **Wrong port / IPv6-only bind (h2).** Partly right in shape, wrong in mechanism: `server-port=19132` + is correct and the process is not v6-only — it binds *no* UDP port at all. Measured, not assumed. +- **Shared-VIP ARP split (h3).** Real (above), not causal. +- **`Cluster` not actually live on the UDP Service (h4).** False; it is live, with datapath to match. +- **Loaded policy ≠ git policy (h5).** False; dumped the BPF map, 19132/UDP is an Allow and has + counted traffic. +- **SNAT collapsing RakNet sessions (h6).** **UNTESTED AND UNTESTABLE TODAY** — there is no RakNet + session to collapse. It remains a live theory for *after* the transport is fixed: Cilium is in SNAT + mode (no `loadBalancer.mode: dsr`), so every LAN player will reach the server as one address, and + RakNet keys sessions on source `ip:port`. Nothing was measured about masquerade port-reuse or UDP + conntrack here. Do not treat this paragraph as evidence of anything. +- **DNS (h8).** False; pihole answers `192.168.2.22` (table above). +- **Client-side realities (h7).** Not the *cause* of a dead socket, but they will be the next wall: + LAN/Friends discovery cannot work behind a VIP, and consoles cannot type a server address. Both are + now documented in `platform/crafty/README.md` rather than left for the next person to rediscover. + +### What changed in this repo + +`platform/crafty/README.md` only. **No manifest was touched** — the manifests are provably correct, +and adding ports or widening policy for this would have been manufacturing work to look productive. +Added: *"Bedrock on UDP 19132 — the transport trap"* (the measurement chain above, and the +`raknet_ping` warning as its signature), *"Bedrock realities that are NOT platform defects"* (no LAN +discovery, consoles, the `enable-lan-visibility` collision with slots 1–2, `server-portv6`). Corrected +three stale claims: "the operator has **not created any servers yet**" (two exist), the human test +that implied a started Bedrock server on 19132 is reachable, and the single-server *Runtime state* +section (now a two-row table). + +### What I did NOT verify — read this before acting + +- **That `transport=raknet` fixes it is REASONED, NOT OBSERVED.** It follows from the shipped property + docs (`Allowed values: "raknet" or "nethernet"`; `server-portv6` "ignored when + transport=nethernet") and from the absence of any UDP bind under `nethernet`. Editing + `server.properties` is a write to application state on the PVC, which is an operator action, so I + did not try it. **Nobody has seen a packet answered on 19132/udp on this cluster.** +- **No end-to-end LAN test.** I never sent a RakNet unconnected-ping from a LAN host to + `192.168.2.22:19132`. I did not need to: the same ping fails from `127.0.0.1` inside the pod. +- **Nothing was applied.** No `kubectl apply/edit/patch/scale`, no `flux reconcile`, no file written + inside the container. The running pod is still `crafty-64f7b6d78-pkpgp` (2/2, 15d, 0 restarts) on + the **pre-`fix/crafty-oauth2-proxy-signin`** spec — that PR is still unmerged. +- **Who those 26 packets came from is unknown.** Byte/packet counters prove arrival and permission, + not identity. I did not run a live `cilium monitor` or `tcpdump`, so "users were trying and being + ignored" is a strong inference, not a captured flow. +- **The Java server's reachability was not tested.** `25565` is bound (`tcp6 63DD LISTEN`) and its + datapath frontend is active on both nodes, but no client connected during this session. The TCP + lease being on k3s-prod-3 while the pod is on k3s-prod-1 is `Cluster` working as designed — asserted + from the service table, not from a connection. +- **The NetherNet side was not explored.** Whether this server is currently joinable *at all* through + Xbox/Friends signalling (its actual transport) is unknown and untested. +- **Bedrock slots 1 and 2 (19133/19134) have never had a server.** The `enable-lan-visibility` + collision I documented for them is read from the shipped comment, not provoked. + +### Operator actions required (I cannot do any of these) + +1. **Make the Bedrock server speak RakNet.** In Crafty: stop `K3s-Bedrock`, edit + `/crafty/servers/c5837301-12b1-404e-b50e-70eafb37226b/server.properties` → + `transport=raknet`, start it. Then confirm the socket exists before telling anyone it works: + `kubectl --context k3s-prod -n crafty exec -c crafty -- grep 4ABC /proc/net/udp /proc/net/udp6` + should now show a wildcard bind, and the `raknet_ping` timeout warnings in + `/crafty/logs/commander.log` should stop. **Both checks, not one.** +2. **Then tell the players how to connect.** Windows/mobile client → **Add Server** → + `crafty-mc.home.keithjackson.dev`, port **19132**. It will *not* appear in the LAN/Friends tab, and + a Switch/Xbox/PlayStation client cannot reach it at all. If the reported users are on consoles, + step 1 will not help them and the answer is "unsupported without a DNS redirect". +3. **Fix `auto_start` on the new server.** DB row `K3s-Bedrock` has `auto_start=0` **and** + `crash_detection=0` — the create-API trap already documented in this repo, live again. + `PATCH /api/v2/servers/c5837301-…` setting both true, **and read them back**. Until then the + Bedrock server does not survive a pod restart — and merging + `fix/crafty-oauth2-proxy-signin` restarts the pod (`strategy: Recreate`, RWO volume), stopping + *both* Minecraft servers. +4. **Optional, cosmetic:** to stop two nodes ARP-announcing `192.168.2.22`, delete + `lease/cilium-l2announce-crafty-crafty-l4-tcp` in `kube-system` and let it re-elect — a 1-in-3 + chance of landing on k3s-prod-1. Not worth doing: under `Cluster` the split costs nothing but ARP + churn, and a redraw is equally random. +5. **After step 1, watch for the SNAT/RakNet question (h6 above)** — if a *second* simultaneous player + evicts the first, that is the SNAT theory becoming real and it needs its own measured + investigation (masquerade port reuse, UDP conntrack), not a guess. diff --git a/platform/crafty/README.md b/platform/crafty/README.md index aeca91c..7fc18e5 100644 --- a/platform/crafty/README.md +++ b/platform/crafty/README.md @@ -83,8 +83,8 @@ themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → 192.16 ### Adding more game ports (when the operator creates servers) -The operator has **not created any servers yet**; the starter set covers 3 Java + 3 Bedrock slots. To -add capacity: +As of 2026-09-29 **two** servers exist (see *Runtime state* below): Java `homelab` on 25565 and +Bedrock `K3s-Bedrock` on 19132. The starter set covers 3 Java + 3 Bedrock slots. To add capacity: 1. In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP). 2. Add that port to the matching Service in `service-l4.yaml` (one @@ -93,6 +93,63 @@ add capacity: 4. (Optional) add a `containerPort` in `deployment.yaml` for documentation. 5. Commit → Flux reconciles. +### Bedrock on UDP 19132 — the transport trap (MEASURED on prod 2026-09-29) + +**A Bedrock server that Crafty created, started, and reports as running can still have nothing bound +to `19132/udp`, with every layer in this repo provably correct.** This is the first thing to check +when "Bedrock players cannot connect" — before the Service, the policy, the lease or the VIP. + +Measured on prod 2026-09-29 against server `K3s-Bedrock` (`c5837301-12b1-404e-b50e-70eafb37226b`, +DB `type=minecraft-bedrock`, `server_port=19132`, process `./bedrock_server` alive): + +- `/proc/net/udp` and `/proc/net/udp6` inside the crafty container held **no socket on 19132** + (`0x4ABC`) at all — only ephemeral UDP sockets. Port 19132 was bound as **TCP** + (`/proc/net/tcp6`, `…:4ABC`, state `0A`/LISTEN), not UDP. +- Cause, from the server's own `server.properties`: **`transport=nethernet`** — the value Microsoft's + BDS archive *ships by default*. Its own inline docs: `# Allowed values: "raknet" or "nethernet"`, + and against `server-portv6`: `# This property is ignored when transport=nethernet and a dual-stack + socket will be opened on server-port instead`. NetherNet is the WebRTC/Xbox-signalled transport; it + **never opens a RakNet UDP listener**, so nothing binds `19132/udp` however `server-port` is set. +- Crafty does not manage this property — `transport` does not appear anywhere under + `/crafty/app/classes/` — and Crafty's *only* Bedrock reachability probe is `raknet_ping.py`, so the + panel spams `WARNING app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping + to 127.0.0.1:19132` every 5–30 s against a server it started successfully. **That warning is the + signature of this trap**, and it comes from `127.0.0.1` — i.e. it indicts the server, not the LAN. +- The platform path was verified sound at the same time: `crafty-l4-udp` live as + `externalTrafficPolicy: Cluster` with `status.loadBalancer.ingress[0].ip` = the VIP, a non-empty + EndpointSlice, `cilium-dbg service list` showing `:19132/UDP → 10.0.0.130:19132/UDP (active)` + on *every* node, and the crafty endpoint's datapath policy map carrying + `Allow Ingress reserved:world 19132/UDP` **with 26 packets / 27660 bytes already counted through + it**. Real player packets reached the pod and were allowed; the application had no socket to take + them. + +**Operator fix — application state on the PVC, not git and not Flux:** stop the server in Crafty, set +`transport=raknet` in `/crafty/servers//server.properties`, start it, and **confirm a UDP +listener on 19132 exists** (`grep 4ABC /proc/net/udp /proc/net/udp6`, or the `raknet_ping` warnings +stopping) before suspecting the network. *(That `raknet` restores the UDP bind is read from the +shipped property documentation, not yet observed on this cluster — see the DECISIONS-LOG entry of +2026-09-29.)* + +### Bedrock realities that are NOT platform defects but present identically + +- **LAN / Friends-tab discovery can never work here.** Bedrock discovery is a UDP broadcast on 19132 + inside one L2 segment; this server sits behind a LoadBalancer VIP (and SNAT), not on the LAN, so it + will **never** appear in a client's Friends/LAN list. Players must use **Add Server** and type the + address and port by hand: `crafty-mc.home.keithjackson.dev` (or the VIP) with port **19132**. + `external-dns` publishes that name from the **-tcp** Service, so it resolves to the shared VIP and + is correct for Bedrock too — only the port differs from Java's 25565. +- **Consoles cannot reach this at all.** Switch/Xbox/PlayStation Bedrock clients have no + arbitrary-server-address field; only Windows/mobile/Fire do. Reaching it from a console needs a DNS + redirect of a featured-server hostname, which is out of scope for `service-l4.yaml`. +- **`enable-lan-visibility=true` (the shipped default) collides with Bedrock slots 1 and 2.** Under + `transport=raknet` it makes a server bind the **default** ports 19132/19133 *even when `server-port` + is set to something else* — the shipped comment says exactly that. So a second Bedrock server put on + 19133 or 19134 will fight the first for 19132. Set `enable-lan-visibility=false` on every Bedrock + server after the first; LAN discovery cannot work here anyway (above). +- **`server-portv6` is not exposed and is not meant to be.** `service-l4.yaml` is IPv4 single-stack, + so the default `server-portv6=19133` is unreachable from the LAN. Harmless — but do not read + `19133` in `server.properties` as "slot 1 is in use". + ## Policies `namespace.yaml` sets only `gateway-access=shared` (platform namespace, like headlamp/homepage), so @@ -114,14 +171,29 @@ Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subjec local admin. A **non-platform-admins** user → **403** (authenticated but not authorized). 2. **Pod health / fsGroup:** `kubectl -n crafty get pod` Ready 2/2; Crafty writes its DB under `/crafty/app/config` and `/crafty/servers` without EACCES. -3. **Server-create + game reach:** in the Crafty UI create a Java server bound to `25565`; start it; - connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or `192.168.1.55:25565`) - from the LAN and confirm the world loads. (Bedrock: a server on `19132/udp` reached at the VIP.) +3. **Server-create + game reach (Java):** in the Crafty UI create a Java server bound to `25565`; + start it; connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or + `192.168.1.55:25565`) from the LAN and confirm the world loads. +4. **Server-create + game reach (Bedrock):** creating and starting the server is **not sufficient** — + check `transport=raknet` and a live UDP listener on 19132 first (see *the transport trap* above), + then **Add Server** by hand in a Windows/mobile client at `crafty-mc.home.keithjackson.dev` port + `19132`. It will never show up in the LAN/Friends tab, and a console cannot reach it. ## Runtime state: the servers themselves are NOT in git -As of 2026-09-14 one Java server exists — **"homelab"**, Paper 26.2, TCP 25565, 2 GiB heap, autostart -on, UUID `13c7b85d-00bb-4a58-8c86-2433ca47891c`. It was created through Crafty's v2 API, and **Mojang's +As of 2026-09-29 **two** servers exist (both measured live, `select * from servers`): + +| Name | Type | Port | UUID | `auto_start` | +|---|---|---|---|---| +| `homelab` | `minecraft-java` (Paper 26.2, 2 GiB heap) | TCP 25565 | `13c7b85d-00bb-4a58-8c86-2433ca47891c` | **1** | +| `K3s-Bedrock` | `minecraft-bedrock` | UDP 19132 | `c5837301-12b1-404e-b50e-70eafb37226b` | **0** ⚠ | + +`K3s-Bedrock` was created 2026-09-29 20:51 and has `auto_start=0` **and** `crash_detection=0` — the +create-API trap documented at the bottom of this file, live again. **It will not come back after a pod +restart**, and merging any change to `deployment.yaml` restarts the pod (`strategy: Recreate`). PATCH +both to true and read them back. + +`homelab` was created through Crafty's v2 API, and **Mojang's EULA () was accepted on the operator's behalf** to let it start; see the DECISIONS-LOG entry of that date.