--- # ============================================================================================ # GAME-PORT L4 EXPOSURE — why the shared HTTPS Gateway is NOT enough (same shape as apps/vms/unifi). # ============================================================================================ # The Minecraft servers Crafty creates listen on RAW TCP (Java) and UDP (Bedrock) game ports inside # the crafty pod's network namespace. The Gateway API `shared` Gateway is an L7 HTTPS terminator and # CANNOT express raw TCP on arbitrary ports or UDP at all. So the game ports are exposed as # type=LoadBalancer Services whose VIP is pulled from the CiliumLoadBalancerIPPool `default-pool` # (dev 192.168.1.53-.60; see CILIUM_LB_POOL_* in cluster-vars) and ARP-announced on the flat LAN by # the existing CiliumL2AnnouncementPolicy # `default-l2` (platform-config/cilium/*) — the SAME machinery the Gateway VIP and unifi already use. # No new Cilium config is needed; these Services just consume the pool. # # THIS PATH IS NOT AUTH-GATED — and that is correct: Minecraft's Java (Mojang/Microsoft auth) and # Bedrock (Xbox Live) protocols authenticate the PLAYER themselves. Only the Crafty WEB UI (admin # control plane) is Keycloak-gated (via the oauth2-proxy sidecar); the game ports carry player traffic # and must reach the server subprocesses directly. # # VIP: CRAFTY_L4_VIP (cluster-vars, R-ENV-02; dev 192.168.1.55) — the next free address in # default-pool (dev: .53 = shared Gateway, .54 = unifi; verified free live 2026-08-28). PINNED via # lbipam.cilium.io/ips so players have a stable target. The `crafty` Flux Kustomization substitutes it. # ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so TCP and UDP split into # two Services sharing the IP via lbipam.cilium.io/sharing-key (same pattern as unifi). # # STARTER PORT SET (the operator has not created any servers yet): TCP 25565-25567 (three Java slots) # + UDP 19132-19134 (three Bedrock slots). To add capacity: create the server in Crafty bound to a NEW # port, then add that port to the matching Service below (and a containerPort in deployment.yaml for # documentation) and to the game-port ingress CiliumNetworkPolicy (networkpolicy.yaml). K8s Services # enumerate ports individually (no ranges), so add one `{ name, port, targetPort, protocol }` line each. # # external-dns (source: service) publishes the A record from the -tcp Service. # # ------------------------------------------------------------------------------------------------- # externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE. # ------------------------------------------------------------------------------------------------- # This file used to say `Local`, justified as "preserves the LAN client source IP and pins the VIP to # the node running the crafty pod (also what L2 announcement does)". THE SECOND HALF IS FALSE, and it # is what took the Minecraft port off the LAN. # # L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder per # Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which matches ALL # linux nodes — and it does NOT filter that set by which nodes hold a backend. MEASURED on prod # 2026-09-14: the crafty pod ran on k3s-prod-1, `cilium-l2announce-crafty-crafty-l4-udp` was held by # k3s-prod-1 (so Bedrock/UDP happened to work) but `-tcp` was held by k3s-prod-2 — and with `Local` an # announcing node holding no backend has nowhere to send the packet and answers ICMP port unreachable. # 192.168.2.22:25565, the Java port, was dead from the LAN for up to 8 days. That the two protocols # landed on different nodes, one working and one not, is the whole tell: the election is per-Service # and random, not backend-aware. It also does NOT self-heal — the holder keeps RENEWING after the pod # moves, so only deleting the Lease forces a (still random) re-election. See # apps/wave-0/pihole-dns.yaml, which hit this first and is the reference implementation. # # `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the pod is, which is what # makes this survive a reschedule and a rebuild. # # NO CHANGE IS NEEDED IN networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED. Under # `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's reserved # `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its # CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`. # crafty-allow-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node` (along # with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the crafty endpoint's datapath # policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state: # `reserved:remote-node` was already an Allow on 25565-25567/TCP and 19132-19134/UDP. Adding # `remote-node` to that rule would be a no-op, so it is deliberately NOT added here — do not "fix" it. # # THE CLIENT-IP COST, STATED PLAINLY AND ACCEPTED. Cilium runs in SNAT mode here (no # `loadBalancer.mode: dsr` in platform/cilium/helmrelease.yaml), so unlike UniFi — a KubeVirt VM whose # `masquerade` binding already destroyed the client address below the Service — crafty is a plain pod # and DOES lose a real client IP it was previously receiving. Every player will appear to the server # subprocesses as the ingress node's address. What that actually costs here: # * PLAYER IDENTITY IS UNAFFECTED. Minecraft Java authenticates against Mojang/Microsoft and Bedrock # against Xbox Live; a player is a UUID, not an address. Vanilla `/ban` and the whitelist are # UUID-keyed (banned-players.json, whitelist.json) and keep working exactly as before. # * `/ban-ip` (banned-ips.json) DEGRADES, and would degrade dangerously: with every player behind one # apparent address, an IP ban would hit EVERY player at once. Use `/ban` (by player), not # `/ban-ip`, on this cluster. No IP ban is configured today. # * Per-IP rate limiting / connection throttling degrades the same way — the whole LAN counts as one # client. Nothing in platform/crafty configures any, and Crafty's own web UI is a separate, # Keycloak-gated path that does not ride these Services. # This is a real cost and it is worth paying: the alternative on offer is a Java port that is dead # outright, and the only fix that keeps BOTH the client IP and a working VIP is `loadBalancer.mode: # dsr` on the Cilium HelmRelease — a platform-wide change with an Ansible [parity] contract that rolls # the CNI across all three nodes, which belongs in its own change, not in an outage fix. # # NO loadBalancerSourceRanges HERE, DELIBERATELY — this is where this file departs from # pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving up a # restriction it had and genuinely needed to be LAN-only. These Services never had a source # restriction at all (crafty-allow-ingress admits `world` on the game ports, and the block above is # explicit that player auth — not network position — is the control). `Cluster` therefore removes no # protection and there is nothing to compensate for. Pinning the game ports to ${LAN_CIDR} during an # outage fix would be a NEW restriction that would cut off any player reaching these ports from off # the LAN, trading this outage for a subtler one. apiVersion: v1 kind: Service metadata: name: crafty-l4-tcp namespace: crafty labels: app.kubernetes.io/name: crafty annotations: lbipam.cilium.io/sharing-key: "crafty" lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}" # external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target). external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}" spec: type: LoadBalancer externalTrafficPolicy: Cluster selector: app.kubernetes.io/name: crafty ports: - { name: mc-java-0, port: 25565, targetPort: 25565, protocol: TCP } - { name: mc-java-1, port: 25566, targetPort: 25566, protocol: TCP } - { name: mc-java-2, port: 25567, targetPort: 25567, protocol: TCP } --- apiVersion: v1 kind: Service metadata: name: crafty-l4-udp namespace: crafty labels: app.kubernetes.io/name: crafty annotations: lbipam.cilium.io/sharing-key: "crafty" # same key + same IP => shares the VIP with -tcp lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}" spec: type: LoadBalancer externalTrafficPolicy: Cluster selector: app.kubernetes.io/name: crafty ports: - { name: mc-bedr-0, port: 19132, targetPort: 19132, protocol: UDP } - { name: mc-bedr-1, port: 19133, targetPort: 19133, protocol: UDP } - { name: mc-bedr-2, port: 19134, targetPort: 19134, protocol: UDP }