jakman85andClaude Opus 5.5
92944486d1
Gateway target -> fc81938b-...cfargotunnel.com, --cloudflare-proxied globally, L4 service publishers (unifi, crafty-mc) pinned DNS-only, Plex web UI and Headlamp disowned (LAN/VPN-only). L0 internal-services-publish-no-public-dns gains assertion (9) tying the Gateway target to the tunnel UUID, requiring cloudflare-proxied=false on every service-source publisher, and keeping the tunnel's 404 list and the unpublished set consistent. L2 dns-exposure fixture opts out of proxying. Co-Authored-By: Claude Opus 5.5 <[email protected]>
138 lines
9.1 KiB
YAML
138 lines
9.1 KiB
YAML
---
|
|
# ============================================================================================
|
|
# GAME-PORT L4 EXPOSURE — why the shared HTTPS Gateway is NOT enough (same shape as apps/vms/unifi).
|
|
# ============================================================================================
|
|
# The Minecraft servers Crafty creates listen on RAW TCP (Java) and UDP (Bedrock) game ports inside
|
|
# the crafty pod's network namespace. The Gateway API `shared` Gateway is an L7 HTTPS terminator and
|
|
# CANNOT express raw TCP on arbitrary ports or UDP at all. So the game ports are exposed as
|
|
# type=LoadBalancer Services whose VIP is pulled from the CiliumLoadBalancerIPPool `default-pool`
|
|
# (dev 192.168.1.53-.60; see CILIUM_LB_POOL_* in cluster-vars) and ARP-announced on the flat LAN by
|
|
# the existing CiliumL2AnnouncementPolicy
|
|
# `default-l2` (platform-config/cilium/*) — the SAME machinery the Gateway VIP and unifi already use.
|
|
# No new Cilium config is needed; these Services just consume the pool.
|
|
#
|
|
# THIS PATH IS NOT AUTH-GATED — and that is correct: Minecraft's Java (Mojang/Microsoft auth) and
|
|
# Bedrock (Xbox Live) protocols authenticate the PLAYER themselves. Only the Crafty WEB UI (admin
|
|
# control plane) is Keycloak-gated (via the oauth2-proxy sidecar); the game ports carry player traffic
|
|
# and must reach the server subprocesses directly.
|
|
#
|
|
# VIP: CRAFTY_L4_VIP (cluster-vars, R-ENV-02; dev 192.168.1.55) — the next free address in
|
|
# default-pool (dev: .53 = shared Gateway, .54 = unifi; verified free live 2026-08-28). PINNED via
|
|
# lbipam.cilium.io/ips so players have a stable target. The `crafty` Flux Kustomization substitutes it.
|
|
# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so TCP and UDP split into
|
|
# two Services sharing the IP via lbipam.cilium.io/sharing-key (same pattern as unifi).
|
|
#
|
|
# STARTER PORT SET (the operator has not created any servers yet): TCP 25565-25567 (three Java slots)
|
|
# + UDP 19132-19134 (three Bedrock slots). To add capacity: create the server in Crafty bound to a NEW
|
|
# port, then add that port to the matching Service below (and a containerPort in deployment.yaml for
|
|
# documentation) and to the game-port ingress CiliumNetworkPolicy (networkpolicy.yaml). K8s Services
|
|
# enumerate ports individually (no ranges), so add one `{ name, port, targetPort, protocol }` line each.
|
|
#
|
|
# external-dns (source: service) publishes the A record from the -tcp Service.
|
|
#
|
|
# -------------------------------------------------------------------------------------------------
|
|
# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
|
|
# -------------------------------------------------------------------------------------------------
|
|
# This file used to say `Local`, justified as "preserves the LAN client source IP and pins the VIP to
|
|
# the node running the crafty pod (also what L2 announcement does)". THE SECOND HALF IS FALSE, and it
|
|
# is what took the Minecraft port off the LAN.
|
|
#
|
|
# L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder per
|
|
# Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which matches ALL
|
|
# linux nodes — and it does NOT filter that set by which nodes hold a backend. MEASURED on prod
|
|
# 2026-09-14: the crafty pod ran on k3s-prod-1, `cilium-l2announce-crafty-crafty-l4-udp` was held by
|
|
# k3s-prod-1 (so Bedrock/UDP happened to work) but `-tcp` was held by k3s-prod-2 — and with `Local` an
|
|
# announcing node holding no backend has nowhere to send the packet and answers ICMP port unreachable.
|
|
# 192.168.2.22:25565, the Java port, was dead from the LAN for up to 8 days. That the two protocols
|
|
# landed on different nodes, one working and one not, is the whole tell: the election is per-Service
|
|
# and random, not backend-aware. It also does NOT self-heal — the holder keeps RENEWING after the pod
|
|
# moves, so only deleting the Lease forces a (still random) re-election. See
|
|
# apps/wave-0/pihole-dns.yaml, which hit this first and is the reference implementation.
|
|
#
|
|
# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the pod is, which is what
|
|
# makes this survive a reschedule and a rebuild.
|
|
#
|
|
# NO CHANGE IS NEEDED IN networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED. Under
|
|
# `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's reserved
|
|
# `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
|
|
# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
|
|
# crafty-allow-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node` (along
|
|
# with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the crafty endpoint's datapath
|
|
# policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
|
|
# `reserved:remote-node` was already an Allow on 25565-25567/TCP and 19132-19134/UDP. Adding
|
|
# `remote-node` to that rule would be a no-op, so it is deliberately NOT added here — do not "fix" it.
|
|
#
|
|
# THE CLIENT-IP COST, STATED PLAINLY AND ACCEPTED. Cilium runs in SNAT mode here (no
|
|
# `loadBalancer.mode: dsr` in platform/cilium/helmrelease.yaml), so unlike UniFi — a KubeVirt VM whose
|
|
# `masquerade` binding already destroyed the client address below the Service — crafty is a plain pod
|
|
# and DOES lose a real client IP it was previously receiving. Every player will appear to the server
|
|
# subprocesses as the ingress node's address. What that actually costs here:
|
|
# * PLAYER IDENTITY IS UNAFFECTED. Minecraft Java authenticates against Mojang/Microsoft and Bedrock
|
|
# against Xbox Live; a player is a UUID, not an address. Vanilla `/ban` and the whitelist are
|
|
# UUID-keyed (banned-players.json, whitelist.json) and keep working exactly as before.
|
|
# * `/ban-ip` (banned-ips.json) DEGRADES, and would degrade dangerously: with every player behind one
|
|
# apparent address, an IP ban would hit EVERY player at once. Use `/ban` (by player), not
|
|
# `/ban-ip`, on this cluster. No IP ban is configured today.
|
|
# * Per-IP rate limiting / connection throttling degrades the same way — the whole LAN counts as one
|
|
# client. Nothing in platform/crafty configures any, and Crafty's own web UI is a separate,
|
|
# Keycloak-gated path that does not ride these Services.
|
|
# This is a real cost and it is worth paying: the alternative on offer is a Java port that is dead
|
|
# outright, and the only fix that keeps BOTH the client IP and a working VIP is `loadBalancer.mode:
|
|
# dsr` on the Cilium HelmRelease — a platform-wide change with an Ansible [parity] contract that rolls
|
|
# the CNI across all three nodes, which belongs in its own change, not in an outage fix.
|
|
#
|
|
# NO loadBalancerSourceRanges HERE, DELIBERATELY — this is where this file departs from
|
|
# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving up a
|
|
# restriction it had and genuinely needed to be LAN-only. These Services never had a source
|
|
# restriction at all (crafty-allow-ingress admits `world` on the game ports, and the block above is
|
|
# explicit that player auth — not network position — is the control). `Cluster` therefore removes no
|
|
# protection and there is nothing to compensate for. Pinning the game ports to ${LAN_CIDR} during an
|
|
# outage fix would be a NEW restriction that would cut off any player reaching these ports from off
|
|
# the LAN, trading this outage for a subtler one.
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: crafty-l4-tcp
|
|
namespace: crafty
|
|
labels:
|
|
app.kubernetes.io/name: crafty
|
|
annotations:
|
|
lbipam.cilium.io/sharing-key: "crafty"
|
|
lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
|
|
# external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target).
|
|
external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
|
|
# DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
|
|
# gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
|
|
# orange-clouded A for these L4 ports would be broken. Required by the L0 check
|
|
# internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
|
|
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
|
|
spec:
|
|
type: LoadBalancer
|
|
externalTrafficPolicy: Cluster
|
|
selector:
|
|
app.kubernetes.io/name: crafty
|
|
ports:
|
|
- { name: mc-java-0, port: 25565, targetPort: 25565, protocol: TCP }
|
|
- { name: mc-java-1, port: 25566, targetPort: 25566, protocol: TCP }
|
|
- { name: mc-java-2, port: 25567, targetPort: 25567, protocol: TCP }
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: crafty-l4-udp
|
|
namespace: crafty
|
|
labels:
|
|
app.kubernetes.io/name: crafty
|
|
annotations:
|
|
lbipam.cilium.io/sharing-key: "crafty" # same key + same IP => shares the VIP with -tcp
|
|
lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
|
|
spec:
|
|
type: LoadBalancer
|
|
externalTrafficPolicy: Cluster
|
|
selector:
|
|
app.kubernetes.io/name: crafty
|
|
ports:
|
|
- { name: mc-bedr-0, port: 19132, targetPort: 19132, protocol: UDP }
|
|
- { name: mc-bedr-1, port: 19133, targetPort: 19133, protocol: UDP }
|
|
- { name: mc-bedr-2, port: 19134, targetPort: 19134, protocol: UDP }
|