diff --git a/apps/vms/unifi/50-service-l4.yaml b/apps/vms/unifi/50-service-l4.yaml index cc68052..18c950b 100644 --- a/apps/vms/unifi/50-service-l4.yaml +++ b/apps/vms/unifi/50-service-l4.yaml @@ -21,8 +21,61 @@ # k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.) # # selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports -# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the -# VIP pinned to the node running the VM (also what L2 announcement does). +# into the guest. +# +# ------------------------------------------------------------------------------------------- +# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE. +# ------------------------------------------------------------------------------------------- +# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the +# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT +# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely. +# +# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder +# per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which +# matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend. +# MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH +# `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an +# announcing node that holds no backend has nowhere to send the packet and answers ICMP port +# unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days +# (the leases were 8 days old); the operator's controller UI and device inform path were down. +# It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only +# deleting the Lease forces a re-election — and that re-election is random, so it can pick a +# backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the +# reference implementation this file now follows. +# +# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The +# interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it +# SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi +# controller has always seen the pod-side masquerade gateway as the source, never the LAN +# client. There is a SECOND translation below the Service, and it already destroyed the very +# thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi: +# it is giving up an address the guest could not see in the first place. +# +# Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device +# MAC (and the device reports its own IP inside the inform payload), not the packet source, +# and guest-portal authorization runs on the gateway appliance, not on this controller. +# +# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which +# is what makes this survive a reschedule and a rebuild. +# +# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED. +# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's +# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its +# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`. +# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node` +# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's +# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state: +# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule +# would be a no-op, so it is deliberately NOT added here — do not "fix" it. +# +# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from +# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving +# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had +# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no +# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage +# fix would be a NEW restriction that could silently break a remote site's device informing over +# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own +# change, made deliberately and tested against the device fleet. # # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53). @@ -49,7 +102,7 @@ metadata: external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}" spec: type: LoadBalancer - externalTrafficPolicy: Local + externalTrafficPolicy: Cluster selector: kubevirt.io/vm: unifi ports: @@ -72,7 +125,7 @@ metadata: lbipam.cilium.io/ips: "${UNIFI_L4_VIP}" spec: type: LoadBalancer - externalTrafficPolicy: Local + externalTrafficPolicy: Cluster selector: kubevirt.io/vm: unifi ports: diff --git a/conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/chainsaw-test.yaml b/conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/chainsaw-test.yaml new file mode 100644 index 0000000..bba4f19 --- /dev/null +++ b/conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/chainsaw-test.yaml @@ -0,0 +1,95 @@ +# ============================================================================= +# L0 STATIC — an L2-announced LoadBalancer VIP must be servable from ANY announcing node +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# THE BUG CLASS THIS CLOSES, AND THE OUTAGE THAT PAID FOR IT. +# Cilium puts a LoadBalancer VIP on the LAN by electing ONE node per Service to answer ARP for +# it. That election is a leader lease over the nodes the CiliumL2AnnouncementPolicy's +# nodeSelector matches, and it is NOT backend-aware — Cilium does not restrict the candidates +# to nodes running a backend, and has no field that would. Combine that with +# `externalTrafficPolicy: Local`, which lets a node deliver only to a backend on ITSELF, and +# the VIP becomes a lottery: an elected announcer holding no backend answers ARP for the +# address and then has nowhere to send the packet, so the client gets ICMP port unreachable. +# On three nodes with one replica that is a 1-in-3 chance of working, drawn independently per +# Service — and it never recovers on its own, because the lease holder keeps renewing after the +# backend moves away. +# +# MEASURED on prod 2026-09-14: unifi-l4-tcp/-udp (VM on k3s-prod-1, both leases on k3s-prod-2) +# left 192.168.2.21:8443/:8080/:11443 dead from the LAN for up to 8 days, and crafty-l4-tcp +# (pod on k3s-prod-1, lease on k3s-prod-2) left the Minecraft Java port 192.168.2.22:25565 +# dead — while its twin crafty-l4-udp, same pod, happened to draw k3s-prod-1 and worked. One +# Service broken and its identical twin working is the tell. The mechanism was then confirmed +# in both directions on the announcing node with `cilium-dbg service list`: under `Local` the +# frontend is ABSENT from that node's service table; under `Cluster` it appears with an active +# backend; flipped back, it vanishes again. +# +# WHY A REVIEWER CANNOT CATCH THIS. `Local` reads as the safe, source-IP-preserving choice, and +# both shipped manifests said so in a comment that also claimed it "pins the VIP to the node +# running the pod" — the precise thing that is false here. The defect is invisible in review +# and silent at apply time; it only appears as an outage, days later, after an unrelated +# reschedule moves a pod. apps/wave-0/pihole-dns.yaml hit it first and is the reference +# implementation; this check keeps the rest of the tree on that side of the line. +# +# GIVEN the repository as git tracks it, +# WHEN every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy +# that would announce its VIP, +# THEN (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing +# node is guaranteed to hold a backend — true only for a DaemonSet-backed Service, +# and that exemption is computed from the tree rather than hand-maintained; +# (2) the premise is still live (some policy actually announces LoadBalancer VIPs), so +# the check cannot pass vacuously if L2 announcement is ever removed; +# (3) the announcer is still backend-BLIND — asserted structurally, so that if a future +# Cilium gains a backend-aware field the exemption logic is revisited deliberately +# instead of going stale in silence; +# AND (4) the check is NON-VACUOUS: a real checkout, a non-empty LoadBalancer corpus, and a +# floor on the assertion count. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library + +# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the +# SAME script. +# +# NOT proven — the runtime half, which needs a cluster: that the VIP is actually reachable, +# that a backend exists, or that an application is listening behind it. That gap is not +# academic. On 2026-09-14 all four Services above ALSO had no listening backend at all (UniFi +# OS had never been installed in the VM; no Minecraft server had ever been created in Crafty), +# so fixing this configuration defect did NOT by itself make those ports answer. A static +# check cannot see an absent application; only an L2 behavioural probe can. Nor does this +# prove `Cluster` is costless — Cilium runs in SNAT mode here, so a cross-node connection +# reaches the backend bearing the ingress node's address, and whether that is acceptable is a +# per-service judgement recorded in each manifest. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: l2-announced-vips-serve-from-any-node + labels: + conformance.home.k3s/R-DNS-01: "true" + annotations: + conformance.home.k3s/requirements: "R-DNS-01" + conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE, and necessarily so. ci/touched-requirements.sh's path map has NO arm for + # `apps/*` and none for `platform/crafty/*`, and nothing in it maps to R-DNS-01 at all. So the + # change most likely to break this invariant — authoring or editing an L4 LoadBalancer Service + # under apps/ or platform/, exactly where both halves of the 2026-09-14 outage lived — would + # attribute NO requirement and be gated by nothing. The invariant also spans three trees at + # once (the Services under apps/ and platform/, and the announcement policy under + # platform-config/), which no single requirement tag can select. + conformance.home.k3s/scope: "whole-repo" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + No LoadBalancer Service whose VIP is put on the LAN by Cilium L2 announcement may use + externalTrafficPolicy Local, because the announcing node is elected without regard to + backend placement — so an announcer holding no backend blackholes the VIP and never + self-heals. The sole exemption is a DaemonSet-backed Service, where every announcing + node is guaranteed a local backend. +spec: + steps: + - name: assert-no-l2-announced-vip-is-etp-local + description: >- + Pair every tracked type=LoadBalancer Service with the CiliumL2AnnouncementPolicy that + would announce its VIP, and assert none combines that announcement with + externalTrafficPolicy Local unless a DaemonSet guarantees a backend on every node. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/check.sh b/conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/check.sh new file mode 100755 index 0000000..e4015b8 --- /dev/null +++ b/conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/check.sh @@ -0,0 +1,242 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — l2-announced-vips-serve-from-any-node +# Serves: R-DNS-01 ("Services MUST be reachable at `.home.keithjackson.dev`." +# — spec docs/00-vision-and-scope.md, section "DNS & ingress"). +# +# THE BUG CLASS THIS CLOSES — it is not hypothetical, it cost an 8-day outage. +# ----------------------------------------------------------------------------- +# A type=LoadBalancer Service on this cluster gets its VIP from Cilium LB-IPAM and is put on +# the LAN by a CiliumL2AnnouncementPolicy, which elects ONE node per Service to answer ARP for +# that VIP. The election is a plain leader lease among the nodes the policy's nodeSelector +# matches. IT IS NOT BACKEND-AWARE: Cilium does not restrict the candidate set to nodes that +# actually run a backend, and it has no field that would let it. +# +# With `externalTrafficPolicy: Local`, a node may only deliver to a backend on ITSELF. Compose +# the two and the VIP is a lottery: if the elected announcer holds no backend, it answers ARP +# for the address and then has nowhere to send the packet, so the client gets ICMP port +# unreachable. On three nodes with one replica that is a 1-in-3 chance of working, drawn +# independently PER SERVICE — and it does not self-heal, because the lease holder keeps +# renewing after the backend moves away. Deleting the Lease only redraws the lottery. +# +# MEASURED on prod 2026-09-14, which is why this check exists: +# * unifi-l4-tcp / unifi-l4-udp — VM on k3s-prod-1, both leases on k3s-prod-2. 192.168.2.21 +# :8443, :8080 and :11443 dead from the LAN for up to 8 days (the leases were 8 days old). +# * crafty-l4-tcp — pod on k3s-prod-1, lease on k3s-prod-2: 192.168.2.22:25565 (Minecraft +# Java) dead. crafty-l4-udp — same pod, lease happened to land on k3s-prod-1, so Bedrock +# worked. One Service broken and its twin working, from the same pod, is the whole tell. +# * The causal mechanism was confirmed both directions on the announcing node with +# `cilium-dbg service list`: under `Local` the frontend 192.168.2.22:25565 is ABSENT from +# that node's service table entirely; flipped to `Cluster` it appears as +# `1 => 10.0.0.11:25565 (active)`. Flipped back, it vanishes again. +# +# `Cluster` is the fix: every node can serve the VIP by forwarding to wherever the backend is, +# so the announcement and the backend no longer have to coincide. apps/wave-0/pihole-dns.yaml +# is the reference implementation (and carries the cost analysis); this check keeps the whole +# tree on that side of the line, because the defect is INVISIBLE in review — `Local` reads as +# the safe, source-IP-preserving choice, and the manifests that shipped it said so explicitly. +# +# GIVEN the repository as git tracks it, +# WHEN every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy +# that would announce its VIP, +# THEN (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing +# node is guaranteed to hold a backend — which is true only for a DaemonSet-backed +# Service, and that exemption is computed from the tree, not hand-maintained; +# (2) the premise is still live: at least one CiliumL2AnnouncementPolicy actually +# announces LoadBalancer VIPs (`loadBalancerIPs: true`). If that ever becomes false +# this check is vacuous and must be re-derived rather than silently passing; +# (3) the announcer's candidate set really is backend-blind — asserted structurally, by +# requiring that the policy carries no field pinning it to backend-holding nodes; +# AND (4) the check is NON-VACUOUS: a real checkout is present, the LoadBalancer corpus is +# non-empty, and a floor on both the corpus and the assertion count keeps a pass +# from degenerating into a no-op when a refactor moves these files. +# +# WHY THE EXEMPTION IS NARROW, AND WHY IT IS SAFE. `Local` is correct precisely when the +# announcing node always has a local backend. Under `default-l2`'s nodeSelector +# (kubernetes.io/os=linux, i.e. every node) that holds only for a DaemonSet. A Deployment with +# replicas == node count does NOT qualify: nothing pins one replica per node, a rollout or a +# cordon breaks it, and the failure mode is this same silent outage. So the exemption keys on +# DaemonSet backing only. +# +# WHAT THIS DOES NOT PROVE (honest gaps): +# - That the VIP is actually reachable. This is a static check; it proves the configuration +# cannot produce the backend-blind-announcer failure, not that a backend exists, that an +# application is listening behind it, or that the LAN can route to it. On 2026-09-14 all +# four Services above ALSO had no listening backend (UniFi OS was never installed in the +# VM; no Minecraft server had been created in Crafty), and no static check can see that. +# The end-to-end claim belongs to an L2 behavioural test against a live cluster. +# - That `Cluster` is COSTLESS. It is not: Cilium runs in SNAT mode here, so a cross-node +# connection reaches the backend bearing the ingress node's address. Whether that cost is +# acceptable is a per-service judgement recorded in each manifest, not something this +# check can decide. +# - Anything about Services created in-cluster rather than authored here (the Gateway's own +# LoadBalancer Service is generated by Cilium from the Gateway resource and is not in the +# tree, so it is outside this static scan). +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import re, subprocess, sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def tracked() -> list[str]: + out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"], + capture_output=True, text=True, check=True).stdout + return [p for p in out.split("\0") if p] + + +def docs_of(rel: str): + try: + return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace")) + if isinstance(d, dict)] + except Exception: + return [] + + +FILES = tracked() +YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)] +check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout") + +# --------------------------------------------------------------------------- the announcers +# A CiliumL2AnnouncementPolicy with loadBalancerIPs: true puts LoadBalancer VIPs on the LAN. +# `serviceSelector` absent means EVERY LoadBalancer Service is announced (Cilium's default). +announcers: list[dict] = [] +for rel in YAML_FILES: + for d in docs_of(rel): + if d.get("kind") == "CiliumL2AnnouncementPolicy": + spec = d.get("spec") or {} + announcers.append({ + "file": rel, + "name": (d.get("metadata") or {}).get("name"), + "lb_ips": bool(spec.get("loadBalancerIPs")), + "service_selector": spec.get("serviceSelector"), + "spec": spec, + }) + +check(len(announcers) > 0, + "no CiliumL2AnnouncementPolicy is tracked — this check's premise (VIPs are put on the " + "LAN by L2 announcement) no longer holds and the invariant must be re-derived (R-DNS-01)") + +lb_announcers = [a for a in announcers if a["lb_ips"]] +# (2) the premise is live. +check(len(lb_announcers) > 0, + "no CiliumL2AnnouncementPolicy sets loadBalancerIPs: true — nothing announces LoadBalancer " + "VIPs, so this check would pass vacuously; re-derive it (R-DNS-01)") + +# (3) the candidate set is backend-blind. Cilium has no field that restricts L2 announcement to +# nodes holding a backend; assert none has appeared, so that if a future Cilium adds one the +# exemption logic here is revisited deliberately instead of silently going stale. +BACKEND_AWARE_FIELDS = {"backendSelector", "endpointSelector", "localBackendsOnly"} +for a in lb_announcers: + present = BACKEND_AWARE_FIELDS & set(a["spec"].keys()) + check(not present, + f"{a['file']}: CiliumL2AnnouncementPolicy/{a['name']} carries {sorted(present)}, which " + "may make announcement backend-aware — the `Local` exemption below must be re-derived " + "before this check can keep passing (R-DNS-01)") + +# --------------------------------------------------------------------------- DaemonSet backing +# Service.spec.selector is matched against DaemonSet pod-template labels in the same namespace. +# A DaemonSet puts a backend on every node, so `Local` cannot strand an announcing node. +daemonset_labels: list[tuple[str, dict]] = [] # (namespace, template labels) +for rel in YAML_FILES: + for d in docs_of(rel): + if d.get("kind") == "DaemonSet": + md = d.get("metadata") or {} + tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {}) + daemonset_labels.append((md.get("namespace"), tmpl.get("labels") or {})) + + +def daemonset_backed(ns, selector) -> bool: + if not selector: + return False + for ds_ns, labels in daemonset_labels: + if ds_ns == ns and all(labels.get(k) == v for k, v in selector.items()): + return True + return False + + +# --------------------------------------------------------------------------- (1) the invariant +lb_services: list[dict] = [] +for rel in YAML_FILES: + for d in docs_of(rel): + if d.get("kind") != "Service": + continue + spec = d.get("spec") or {} + if spec.get("type") != "LoadBalancer": + continue + md = d.get("metadata") or {} + lb_services.append({ + "file": rel, + "name": md.get("name"), + "ns": md.get("namespace"), + "etp": spec.get("externalTrafficPolicy"), + "selector": spec.get("selector") or {}, + "pinned": (md.get("annotations") or {}).get("lbipam.cilium.io/ips"), + }) + +# Non-vacuity floor: these Services are the reason the check exists; if the corpus empties out, +# something moved and the check must be re-pointed rather than passing on nothing. +check(len(lb_services) >= 3, + f"only {len(lb_services)} tracked type=LoadBalancer Service(s) found — the corpus this " + "check quantifies over has moved or emptied; re-point it (R-DNS-01)") + +exempt: list[str] = [] +for s in lb_services: + where = f"{s['file']}: Service/{s['ns']}/{s['name']}" + if s["etp"] != "Local": + CHECKS += 1 # counted: every Service is genuinely inspected + continue + if daemonset_backed(s["ns"], s["selector"]): + exempt.append(f"{s['ns']}/{s['name']}") + CHECKS += 1 + continue + pin = f" pinned at {s['pinned']}" if s["pinned"] else "" + check(False, + f"{where}{pin} combines type=LoadBalancer with externalTrafficPolicy: Local while " + f"CiliumL2AnnouncementPolicy/{lb_announcers[0]['name'] if lb_announcers else '?'} " + "announces its VIP from nodes chosen WITHOUT regard to backend placement. When the " + "elected announcer holds no backend the VIP answers ICMP port unreachable and does " + "not self-heal — this is the 8-day unifi/crafty outage of 2026-09-14. Use " + "externalTrafficPolicy: Cluster (see apps/wave-0/pihole-dns.yaml for the pattern and " + "the client-IP cost it trades away), or back the Service with a DaemonSet so every " + "announcing node holds a backend (R-DNS-01)") + +# --------------------------------------------------------------------------- (4) non-vacuity +check(CHECKS >= 8, + f"only {CHECKS} assertions ran — too few for this invariant to be meaningful (R-DNS-01)") + +# --------------------------------------------------------------------------------- verdict +if FAILURES: + print(f"CHECK FAIL (l2-announced-vips-serve-from-any-node): {len(FAILURES)} of {CHECKS} " + "assertions failed", file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +note = f", {len(exempt)} DaemonSet-backed exemption(s): {sorted(exempt)}" if exempt else "" +print(f"check OK: l2-announced-vips-serve-from-any-node — {CHECKS} assertions over " + f"{len(lb_services)} LoadBalancer Service(s) and {len(lb_announcers)} announcing " + f"policy(ies){note}") +PY diff --git a/platform/crafty/service-l4.yaml b/platform/crafty/service-l4.yaml index 97c9039..f99fa53 100644 --- a/platform/crafty/service-l4.yaml +++ b/platform/crafty/service-l4.yaml @@ -28,9 +28,67 @@ # documentation) and to the game-port ingress CiliumNetworkPolicy (networkpolicy.yaml). K8s Services # enumerate ports individually (no ranges), so add one `{ name, port, targetPort, protocol }` line each. # -# externalTrafficPolicy: Local preserves the LAN client source IP and pins the VIP to the node running -# the crafty pod (also what L2 announcement does). external-dns (source: service) publishes the A -# record from the -tcp Service. +# external-dns (source: service) publishes the A record from the -tcp Service. +# +# ------------------------------------------------------------------------------------------------- +# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE. +# ------------------------------------------------------------------------------------------------- +# This file used to say `Local`, justified as "preserves the LAN client source IP and pins the VIP to +# the node running the crafty pod (also what L2 announcement does)". THE SECOND HALF IS FALSE, and it +# is what took the Minecraft port off the LAN. +# +# L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder per +# Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which matches ALL +# linux nodes — and it does NOT filter that set by which nodes hold a backend. MEASURED on prod +# 2026-09-14: the crafty pod ran on k3s-prod-1, `cilium-l2announce-crafty-crafty-l4-udp` was held by +# k3s-prod-1 (so Bedrock/UDP happened to work) but `-tcp` was held by k3s-prod-2 — and with `Local` an +# announcing node holding no backend has nowhere to send the packet and answers ICMP port unreachable. +# 192.168.2.22:25565, the Java port, was dead from the LAN for up to 8 days. That the two protocols +# landed on different nodes, one working and one not, is the whole tell: the election is per-Service +# and random, not backend-aware. It also does NOT self-heal — the holder keeps RENEWING after the pod +# moves, so only deleting the Lease forces a (still random) re-election. See +# apps/wave-0/pihole-dns.yaml, which hit this first and is the reference implementation. +# +# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the pod is, which is what +# makes this survive a reschedule and a rebuild. +# +# NO CHANGE IS NEEDED IN networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED. Under +# `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's reserved +# `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its +# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`. +# crafty-allow-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node` (along +# with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the crafty endpoint's datapath +# policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state: +# `reserved:remote-node` was already an Allow on 25565-25567/TCP and 19132-19134/UDP. Adding +# `remote-node` to that rule would be a no-op, so it is deliberately NOT added here — do not "fix" it. +# +# THE CLIENT-IP COST, STATED PLAINLY AND ACCEPTED. Cilium runs in SNAT mode here (no +# `loadBalancer.mode: dsr` in platform/cilium/helmrelease.yaml), so unlike UniFi — a KubeVirt VM whose +# `masquerade` binding already destroyed the client address below the Service — crafty is a plain pod +# and DOES lose a real client IP it was previously receiving. Every player will appear to the server +# subprocesses as the ingress node's address. What that actually costs here: +# * PLAYER IDENTITY IS UNAFFECTED. Minecraft Java authenticates against Mojang/Microsoft and Bedrock +# against Xbox Live; a player is a UUID, not an address. Vanilla `/ban` and the whitelist are +# UUID-keyed (banned-players.json, whitelist.json) and keep working exactly as before. +# * `/ban-ip` (banned-ips.json) DEGRADES, and would degrade dangerously: with every player behind one +# apparent address, an IP ban would hit EVERY player at once. Use `/ban` (by player), not +# `/ban-ip`, on this cluster. No IP ban is configured today. +# * Per-IP rate limiting / connection throttling degrades the same way — the whole LAN counts as one +# client. Nothing in platform/crafty configures any, and Crafty's own web UI is a separate, +# Keycloak-gated path that does not ride these Services. +# This is a real cost and it is worth paying: the alternative on offer is a Java port that is dead +# outright, and the only fix that keeps BOTH the client IP and a working VIP is `loadBalancer.mode: +# dsr` on the Cilium HelmRelease — a platform-wide change with an Ansible [parity] contract that rolls +# the CNI across all three nodes, which belongs in its own change, not in an outage fix. +# +# NO loadBalancerSourceRanges HERE, DELIBERATELY — this is where this file departs from +# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving up a +# restriction it had and genuinely needed to be LAN-only. These Services never had a source +# restriction at all (crafty-allow-ingress admits `world` on the game ports, and the block above is +# explicit that player auth — not network position — is the control). `Cluster` therefore removes no +# protection and there is nothing to compensate for. Pinning the game ports to ${LAN_CIDR} during an +# outage fix would be a NEW restriction that would cut off any player reaching these ports from off +# the LAN, trading this outage for a subtler one. apiVersion: v1 kind: Service metadata: @@ -45,7 +103,7 @@ metadata: external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}" spec: type: LoadBalancer - externalTrafficPolicy: Local + externalTrafficPolicy: Cluster selector: app.kubernetes.io/name: crafty ports: @@ -65,7 +123,7 @@ metadata: lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}" spec: type: LoadBalancer - externalTrafficPolicy: Local + externalTrafficPolicy: Cluster selector: app.kubernetes.io/name: crafty ports: