fix(l4): externalTrafficPolicy Cluster on unifi + crafty L4 VIPs

Cilium's L2 announcer elects one node per Service to answer ARP for its
LoadBalancer VIP, and that election is NOT backend-aware. Combined with
externalTrafficPolicy: Local -- which lets a node deliver only to a backend on
itself -- an elected announcer holding no backend answers ARP for the address
and then has nowhere to send the packet, so the client gets ICMP port
unreachable. It never self-heals: the lease holder keeps renewing after the
backend moves away.

Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on
k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on
k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and
worked. Confirmed causally on the announcing node with `cilium-dbg service
list`: under Local the frontend 192.168.2.22:25565 is absent from that node's
service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565
(active)`; flipped back, it vanishes again. Durability confirmed by cordoning
k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing
node re-pointed to the new backend with no intervention.

Both files also carried a comment claiming Local "pins the VIP to the node
running the pod", which is the precise thing that is false here; both are
rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation.

The client-IP cost is stated per service rather than waved away. For unifi it
is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into
the guest, so Local was never preserving a client address the controller could
see. For crafty it is real and accepted -- player identity is a Mojang/Xbox
UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every
player at once and must not be used; no IP ban or per-IP rate limit is
configured today.

No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml:
these Services never had a source restriction (their CNPs admit `world`), so
Cluster removes no protection, and adding a LAN-only filter during an outage
fix could cut off off-LAN players or a remote site's device informing.

No CiliumNetworkPolicy change is needed, and that was verified rather than
assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by
dumping the crafty endpoint's datapath policy map with the CNP unmodified
(`reserved:remote-node` was already an Allow on the game ports). Unlike
pihole, whose policy omits `cluster` and must name remote-node explicitly.

Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with
exactly these four Services named and passes after. whole-repo scope is
necessary: ci/touched-requirements.sh has no arm for apps/* or
platform/crafty/*, so the change most likely to reintroduce this would
attribute no requirement and be gated by nothing.

HONEST LIMIT -- this does NOT end the reported outage. All four ports also
have no listening backend: UniFi OS was never installed in the VM (its README
flags this as the open blocker) and /crafty/servers is empty, so no Minecraft
server exists. Probed directly from the node, bypassing the VIP entirely,
10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two
controls answer. This commit fixes a real latent defect that would have
blackholed those VIPs once the applications exist; it cannot make an absent
application answer.

Co-Authored-By: Claude Opus 5 <[email protected]>

apps/vms/unifi/50-service-l4.yaml

@@ -21,8 +21,61 @@
 # k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
 #
 # selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
-# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
-# VIP pinned to the node running the VM (also what L2 announcement does).
+# into the guest.
+#
+# -------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
+# -------------------------------------------------------------------------------------------
+# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
+# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
+# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
+#
+# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
+#    per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
+#    matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
+#    MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
+#    `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
+#    announcing node that holds no backend has nowhere to send the packet and answers ICMP port
+#    unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
+#    (the leases were 8 days old); the operator's controller UI and device inform path were down.
+#    It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
+#    deleting the Lease forces a re-election — and that re-election is random, so it can pick a
+#    backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
+#    reference implementation this file now follows.
+#
+# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
+#    interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
+#    SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
+#    controller has always seen the pod-side masquerade gateway as the source, never the LAN
+#    client. There is a SECOND translation below the Service, and it already destroyed the very
+#    thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
+#    it is giving up an address the guest could not see in the first place.
+#
+#    Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
+#    MAC (and the device reports its own IP inside the inform payload), not the packet source,
+#    and guest-portal authorization runs on the gateway appliance, not on this controller.
+#
+# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
+# is what makes this survive a reschedule and a rebuild.
+#
+# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
+# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
+# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
+# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
+# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
+# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
+# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
+# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
+# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
+#
+# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
+# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
+# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
+# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
+# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
+# fix would be a NEW restriction that could silently break a remote site's device informing over
+# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
+# change, made deliberately and tested against the device fleet.
 #
 # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
 # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
@@ -49,7 +102,7 @@ metadata:
     external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:
@@ -72,7 +125,7 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:

conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/chainsaw-test.yaml

@@ -0,0 +1,95 @@
+# =============================================================================
+# L0 STATIC — an L2-announced LoadBalancer VIP must be servable from ANY announcing node
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE BUG CLASS THIS CLOSES, AND THE OUTAGE THAT PAID FOR IT.
+# Cilium puts a LoadBalancer VIP on the LAN by electing ONE node per Service to answer ARP for
+# it. That election is a leader lease over the nodes the CiliumL2AnnouncementPolicy's
+# nodeSelector matches, and it is NOT backend-aware — Cilium does not restrict the candidates
+# to nodes running a backend, and has no field that would. Combine that with
+# `externalTrafficPolicy: Local`, which lets a node deliver only to a backend on ITSELF, and
+# the VIP becomes a lottery: an elected announcer holding no backend answers ARP for the
+# address and then has nowhere to send the packet, so the client gets ICMP port unreachable.
+# On three nodes with one replica that is a 1-in-3 chance of working, drawn independently per
+# Service — and it never recovers on its own, because the lease holder keeps renewing after the
+# backend moves away.
+#
+# MEASURED on prod 2026-09-14: unifi-l4-tcp/-udp (VM on k3s-prod-1, both leases on k3s-prod-2)
+# left 192.168.2.21:8443/:8080/:11443 dead from the LAN for up to 8 days, and crafty-l4-tcp
+# (pod on k3s-prod-1, lease on k3s-prod-2) left the Minecraft Java port 192.168.2.22:25565
+# dead — while its twin crafty-l4-udp, same pod, happened to draw k3s-prod-1 and worked. One
+# Service broken and its identical twin working is the tell. The mechanism was then confirmed
+# in both directions on the announcing node with `cilium-dbg service list`: under `Local` the
+# frontend is ABSENT from that node's service table; under `Cluster` it appears with an active
+# backend; flipped back, it vanishes again.
+#
+# WHY A REVIEWER CANNOT CATCH THIS. `Local` reads as the safe, source-IP-preserving choice, and
+# both shipped manifests said so in a comment that also claimed it "pins the VIP to the node
+# running the pod" — the precise thing that is false here. The defect is invisible in review
+# and silent at apply time; it only appears as an outage, days later, after an unrelated
+# reschedule moves a pod. apps/wave-0/pihole-dns.yaml hit it first and is the reference
+# implementation; this check keeps the rest of the tree on that side of the line.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
+#       that would announce its VIP,
+# THEN  (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
+#           node is guaranteed to hold a backend — true only for a DaemonSet-backed Service,
+#           and that exemption is computed from the tree rather than hand-maintained;
+#       (2) the premise is still live (some policy actually announces LoadBalancer VIPs), so
+#           the check cannot pass vacuously if L2 announcement is ever removed;
+#       (3) the announcer is still backend-BLIND — asserted structurally, so that if a future
+#           Cilium gains a backend-aware field the exemption logic is revisited deliberately
+#           instead of going stale in silence;
+#   AND (4) the check is NON-VACUOUS: a real checkout, a non-empty LoadBalancer corpus, and a
+#           floor on the assertion count.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
+# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the
+# SAME script.
+#
+# NOT proven — the runtime half, which needs a cluster: that the VIP is actually reachable,
+# that a backend exists, or that an application is listening behind it. That gap is not
+# academic. On 2026-09-14 all four Services above ALSO had no listening backend at all (UniFi
+# OS had never been installed in the VM; no Minecraft server had ever been created in Crafty),
+# so fixing this configuration defect did NOT by itself make those ports answer. A static
+# check cannot see an absent application; only an L2 behavioural probe can. Nor does this
+# prove `Cluster` is costless — Cilium runs in SNAT mode here, so a cross-node connection
+# reaches the backend bearing the ingress node's address, and whether that is acceptable is a
+# per-service judgement recorded in each manifest.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: l2-announced-vips-serve-from-any-node
+  labels:
+    conformance.home.k3s/R-DNS-01: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-DNS-01"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE, and necessarily so. ci/touched-requirements.sh's path map has NO arm for
+    # `apps/*` and none for `platform/crafty/*`, and nothing in it maps to R-DNS-01 at all. So the
+    # change most likely to break this invariant — authoring or editing an L4 LoadBalancer Service
+    # under apps/ or platform/, exactly where both halves of the 2026-09-14 outage lived — would
+    # attribute NO requirement and be gated by nothing. The invariant also spans three trees at
+    # once (the Services under apps/ and platform/, and the announcement policy under
+    # platform-config/), which no single requirement tag can select.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      No LoadBalancer Service whose VIP is put on the LAN by Cilium L2 announcement may use
+      externalTrafficPolicy Local, because the announcing node is elected without regard to
+      backend placement — so an announcer holding no backend blackholes the VIP and never
+      self-heals. The sole exemption is a DaemonSet-backed Service, where every announcing
+      node is guaranteed a local backend.
+spec:
+  steps:
+    - name: assert-no-l2-announced-vip-is-etp-local
+      description: >-
+        Pair every tracked type=LoadBalancer Service with the CiliumL2AnnouncementPolicy that
+        would announce its VIP, and assert none combines that announcement with
+        externalTrafficPolicy Local unless a DaemonSet guarantees a backend on every node.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/check.sh

@@ -0,0 +1,242 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — l2-announced-vips-serve-from-any-node
+# Serves: R-DNS-01 ("Services MUST be reachable at `<service>.home.keithjackson.dev`."
+#         — spec docs/00-vision-and-scope.md, section "DNS & ingress").
+#
+# THE BUG CLASS THIS CLOSES — it is not hypothetical, it cost an 8-day outage.
+# -----------------------------------------------------------------------------
+# A type=LoadBalancer Service on this cluster gets its VIP from Cilium LB-IPAM and is put on
+# the LAN by a CiliumL2AnnouncementPolicy, which elects ONE node per Service to answer ARP for
+# that VIP. The election is a plain leader lease among the nodes the policy's nodeSelector
+# matches. IT IS NOT BACKEND-AWARE: Cilium does not restrict the candidate set to nodes that
+# actually run a backend, and it has no field that would let it.
+#
+# With `externalTrafficPolicy: Local`, a node may only deliver to a backend on ITSELF. Compose
+# the two and the VIP is a lottery: if the elected announcer holds no backend, it answers ARP
+# for the address and then has nowhere to send the packet, so the client gets ICMP port
+# unreachable. On three nodes with one replica that is a 1-in-3 chance of working, drawn
+# independently PER SERVICE — and it does not self-heal, because the lease holder keeps
+# renewing after the backend moves away. Deleting the Lease only redraws the lottery.
+#
+# MEASURED on prod 2026-09-14, which is why this check exists:
+#   * unifi-l4-tcp / unifi-l4-udp — VM on k3s-prod-1, both leases on k3s-prod-2. 192.168.2.21
+#     :8443, :8080 and :11443 dead from the LAN for up to 8 days (the leases were 8 days old).
+#   * crafty-l4-tcp — pod on k3s-prod-1, lease on k3s-prod-2: 192.168.2.22:25565 (Minecraft
+#     Java) dead. crafty-l4-udp — same pod, lease happened to land on k3s-prod-1, so Bedrock
+#     worked. One Service broken and its twin working, from the same pod, is the whole tell.
+#   * The causal mechanism was confirmed both directions on the announcing node with
+#     `cilium-dbg service list`: under `Local` the frontend 192.168.2.22:25565 is ABSENT from
+#     that node's service table entirely; flipped to `Cluster` it appears as
+#     `1 => 10.0.0.11:25565 (active)`. Flipped back, it vanishes again.
+#
+# `Cluster` is the fix: every node can serve the VIP by forwarding to wherever the backend is,
+# so the announcement and the backend no longer have to coincide. apps/wave-0/pihole-dns.yaml
+# is the reference implementation (and carries the cost analysis); this check keeps the whole
+# tree on that side of the line, because the defect is INVISIBLE in review — `Local` reads as
+# the safe, source-IP-preserving choice, and the manifests that shipped it said so explicitly.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
+#       that would announce its VIP,
+# THEN  (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
+#           node is guaranteed to hold a backend — which is true only for a DaemonSet-backed
+#           Service, and that exemption is computed from the tree, not hand-maintained;
+#       (2) the premise is still live: at least one CiliumL2AnnouncementPolicy actually
+#           announces LoadBalancer VIPs (`loadBalancerIPs: true`). If that ever becomes false
+#           this check is vacuous and must be re-derived rather than silently passing;
+#       (3) the announcer's candidate set really is backend-blind — asserted structurally, by
+#           requiring that the policy carries no field pinning it to backend-holding nodes;
+#   AND (4) the check is NON-VACUOUS: a real checkout is present, the LoadBalancer corpus is
+#           non-empty, and a floor on both the corpus and the assertion count keeps a pass
+#           from degenerating into a no-op when a refactor moves these files.
+#
+# WHY THE EXEMPTION IS NARROW, AND WHY IT IS SAFE. `Local` is correct precisely when the
+# announcing node always has a local backend. Under `default-l2`'s nodeSelector
+# (kubernetes.io/os=linux, i.e. every node) that holds only for a DaemonSet. A Deployment with
+# replicas == node count does NOT qualify: nothing pins one replica per node, a rollout or a
+# cordon breaks it, and the failure mode is this same silent outage. So the exemption keys on
+# DaemonSet backing only.
+#
+# WHAT THIS DOES NOT PROVE (honest gaps):
+#   - That the VIP is actually reachable. This is a static check; it proves the configuration
+#     cannot produce the backend-blind-announcer failure, not that a backend exists, that an
+#     application is listening behind it, or that the LAN can route to it. On 2026-09-14 all
+#     four Services above ALSO had no listening backend (UniFi OS was never installed in the
+#     VM; no Minecraft server had been created in Crafty), and no static check can see that.
+#     The end-to-end claim belongs to an L2 behavioural test against a live cluster.
+#   - That `Cluster` is COSTLESS. It is not: Cilium runs in SNAT mode here, so a cross-node
+#     connection reaches the backend bearing the ingress node's address. Whether that cost is
+#     acceptable is a per-service judgement recorded in each manifest, not something this
+#     check can decide.
+#   - Anything about Services created in-cluster rather than authored here (the Gateway's own
+#     LoadBalancer Service is generated by Cilium from the Gateway resource and is not in the
+#     tree, so it is outside this static scan).
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+import re, subprocess, sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def tracked() -> list[str]:
+    out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
+                         capture_output=True, text=True, check=True).stdout
+    return [p for p in out.split("\0") if p]
+
+
+def docs_of(rel: str):
+    try:
+        return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
+                if isinstance(d, dict)]
+    except Exception:
+        return []
+
+
+FILES = tracked()
+YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)]
+check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
+
+# --------------------------------------------------------------------------- the announcers
+# A CiliumL2AnnouncementPolicy with loadBalancerIPs: true puts LoadBalancer VIPs on the LAN.
+# `serviceSelector` absent means EVERY LoadBalancer Service is announced (Cilium's default).
+announcers: list[dict] = []
+for rel in YAML_FILES:
+    for d in docs_of(rel):
+        if d.get("kind") == "CiliumL2AnnouncementPolicy":
+            spec = d.get("spec") or {}
+            announcers.append({
+                "file": rel,
+                "name": (d.get("metadata") or {}).get("name"),
+                "lb_ips": bool(spec.get("loadBalancerIPs")),
+                "service_selector": spec.get("serviceSelector"),
+                "spec": spec,
+            })
+
+check(len(announcers) > 0,
+      "no CiliumL2AnnouncementPolicy is tracked — this check's premise (VIPs are put on the "
+      "LAN by L2 announcement) no longer holds and the invariant must be re-derived (R-DNS-01)")
+
+lb_announcers = [a for a in announcers if a["lb_ips"]]
+# (2) the premise is live.
+check(len(lb_announcers) > 0,
+      "no CiliumL2AnnouncementPolicy sets loadBalancerIPs: true — nothing announces LoadBalancer "
+      "VIPs, so this check would pass vacuously; re-derive it (R-DNS-01)")
+
+# (3) the candidate set is backend-blind. Cilium has no field that restricts L2 announcement to
+#     nodes holding a backend; assert none has appeared, so that if a future Cilium adds one the
+#     exemption logic here is revisited deliberately instead of silently going stale.
+BACKEND_AWARE_FIELDS = {"backendSelector", "endpointSelector", "localBackendsOnly"}
+for a in lb_announcers:
+    present = BACKEND_AWARE_FIELDS & set(a["spec"].keys())
+    check(not present,
+          f"{a['file']}: CiliumL2AnnouncementPolicy/{a['name']} carries {sorted(present)}, which "
+          "may make announcement backend-aware — the `Local` exemption below must be re-derived "
+          "before this check can keep passing (R-DNS-01)")
+
+# --------------------------------------------------------------------------- DaemonSet backing
+# Service.spec.selector is matched against DaemonSet pod-template labels in the same namespace.
+# A DaemonSet puts a backend on every node, so `Local` cannot strand an announcing node.
+daemonset_labels: list[tuple[str, dict]] = []   # (namespace, template labels)
+for rel in YAML_FILES:
+    for d in docs_of(rel):
+        if d.get("kind") == "DaemonSet":
+            md = d.get("metadata") or {}
+            tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {})
+            daemonset_labels.append((md.get("namespace"), tmpl.get("labels") or {}))
+
+
+def daemonset_backed(ns, selector) -> bool:
+    if not selector:
+        return False
+    for ds_ns, labels in daemonset_labels:
+        if ds_ns == ns and all(labels.get(k) == v for k, v in selector.items()):
+            return True
+    return False
+
+
+# --------------------------------------------------------------------------- (1) the invariant
+lb_services: list[dict] = []
+for rel in YAML_FILES:
+    for d in docs_of(rel):
+        if d.get("kind") != "Service":
+            continue
+        spec = d.get("spec") or {}
+        if spec.get("type") != "LoadBalancer":
+            continue
+        md = d.get("metadata") or {}
+        lb_services.append({
+            "file": rel,
+            "name": md.get("name"),
+            "ns": md.get("namespace"),
+            "etp": spec.get("externalTrafficPolicy"),
+            "selector": spec.get("selector") or {},
+            "pinned": (md.get("annotations") or {}).get("lbipam.cilium.io/ips"),
+        })
+
+# Non-vacuity floor: these Services are the reason the check exists; if the corpus empties out,
+# something moved and the check must be re-pointed rather than passing on nothing.
+check(len(lb_services) >= 3,
+      f"only {len(lb_services)} tracked type=LoadBalancer Service(s) found — the corpus this "
+      "check quantifies over has moved or emptied; re-point it (R-DNS-01)")
+
+exempt: list[str] = []
+for s in lb_services:
+    where = f"{s['file']}: Service/{s['ns']}/{s['name']}"
+    if s["etp"] != "Local":
+        CHECKS += 1                      # counted: every Service is genuinely inspected
+        continue
+    if daemonset_backed(s["ns"], s["selector"]):
+        exempt.append(f"{s['ns']}/{s['name']}")
+        CHECKS += 1
+        continue
+    pin = f" pinned at {s['pinned']}" if s["pinned"] else ""
+    check(False,
+          f"{where}{pin} combines type=LoadBalancer with externalTrafficPolicy: Local while "
+          f"CiliumL2AnnouncementPolicy/{lb_announcers[0]['name'] if lb_announcers else '?'} "
+          "announces its VIP from nodes chosen WITHOUT regard to backend placement. When the "
+          "elected announcer holds no backend the VIP answers ICMP port unreachable and does "
+          "not self-heal — this is the 8-day unifi/crafty outage of 2026-09-14. Use "
+          "externalTrafficPolicy: Cluster (see apps/wave-0/pihole-dns.yaml for the pattern and "
+          "the client-IP cost it trades away), or back the Service with a DaemonSet so every "
+          "announcing node holds a backend (R-DNS-01)")
+
+# --------------------------------------------------------------------------- (4) non-vacuity
+check(CHECKS >= 8,
+      f"only {CHECKS} assertions ran — too few for this invariant to be meaningful (R-DNS-01)")
+
+# --------------------------------------------------------------------------------- verdict
+if FAILURES:
+    print(f"CHECK FAIL (l2-announced-vips-serve-from-any-node): {len(FAILURES)} of {CHECKS} "
+          "assertions failed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+note = f", {len(exempt)} DaemonSet-backed exemption(s): {sorted(exempt)}" if exempt else ""
+print(f"check OK: l2-announced-vips-serve-from-any-node — {CHECKS} assertions over "
+      f"{len(lb_services)} LoadBalancer Service(s) and {len(lb_announcers)} announcing "
+      f"policy(ies){note}")
+PY

platform/crafty/service-l4.yaml

@@ -28,9 +28,67 @@
 # documentation) and to the game-port ingress CiliumNetworkPolicy (networkpolicy.yaml). K8s Services
 # enumerate ports individually (no ranges), so add one `{ name, port, targetPort, protocol }` line each.
 #
-# externalTrafficPolicy: Local preserves the LAN client source IP and pins the VIP to the node running
-# the crafty pod (also what L2 announcement does). external-dns (source: service) publishes the A
-# record from the -tcp Service.
+# external-dns (source: service) publishes the A record from the -tcp Service.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
+# -------------------------------------------------------------------------------------------------
+# This file used to say `Local`, justified as "preserves the LAN client source IP and pins the VIP to
+# the node running the crafty pod (also what L2 announcement does)". THE SECOND HALF IS FALSE, and it
+# is what took the Minecraft port off the LAN.
+#
+# L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder per
+# Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which matches ALL
+# linux nodes — and it does NOT filter that set by which nodes hold a backend. MEASURED on prod
+# 2026-09-14: the crafty pod ran on k3s-prod-1, `cilium-l2announce-crafty-crafty-l4-udp` was held by
+# k3s-prod-1 (so Bedrock/UDP happened to work) but `-tcp` was held by k3s-prod-2 — and with `Local` an
+# announcing node holding no backend has nowhere to send the packet and answers ICMP port unreachable.
+# 192.168.2.22:25565, the Java port, was dead from the LAN for up to 8 days. That the two protocols
+# landed on different nodes, one working and one not, is the whole tell: the election is per-Service
+# and random, not backend-aware. It also does NOT self-heal — the holder keeps RENEWING after the pod
+# moves, so only deleting the Lease forces a (still random) re-election. See
+# apps/wave-0/pihole-dns.yaml, which hit this first and is the reference implementation.
+#
+# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the pod is, which is what
+# makes this survive a reschedule and a rebuild.
+#
+# NO CHANGE IS NEEDED IN networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED. Under
+# `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's reserved
+# `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
+# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
+# crafty-allow-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node` (along
+# with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the crafty endpoint's datapath
+# policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
+# `reserved:remote-node` was already an Allow on 25565-25567/TCP and 19132-19134/UDP. Adding
+# `remote-node` to that rule would be a no-op, so it is deliberately NOT added here — do not "fix" it.
+#
+# THE CLIENT-IP COST, STATED PLAINLY AND ACCEPTED. Cilium runs in SNAT mode here (no
+# `loadBalancer.mode: dsr` in platform/cilium/helmrelease.yaml), so unlike UniFi — a KubeVirt VM whose
+# `masquerade` binding already destroyed the client address below the Service — crafty is a plain pod
+# and DOES lose a real client IP it was previously receiving. Every player will appear to the server
+# subprocesses as the ingress node's address. What that actually costs here:
+#   * PLAYER IDENTITY IS UNAFFECTED. Minecraft Java authenticates against Mojang/Microsoft and Bedrock
+#     against Xbox Live; a player is a UUID, not an address. Vanilla `/ban` and the whitelist are
+#     UUID-keyed (banned-players.json, whitelist.json) and keep working exactly as before.
+#   * `/ban-ip` (banned-ips.json) DEGRADES, and would degrade dangerously: with every player behind one
+#     apparent address, an IP ban would hit EVERY player at once. Use `/ban` (by player), not
+#     `/ban-ip`, on this cluster. No IP ban is configured today.
+#   * Per-IP rate limiting / connection throttling degrades the same way — the whole LAN counts as one
+#     client. Nothing in platform/crafty configures any, and Crafty's own web UI is a separate,
+#     Keycloak-gated path that does not ride these Services.
+# This is a real cost and it is worth paying: the alternative on offer is a Java port that is dead
+# outright, and the only fix that keeps BOTH the client IP and a working VIP is `loadBalancer.mode:
+# dsr` on the Cilium HelmRelease — a platform-wide change with an Ansible [parity] contract that rolls
+# the CNI across all three nodes, which belongs in its own change, not in an outage fix.
+#
+# NO loadBalancerSourceRanges HERE, DELIBERATELY — this is where this file departs from
+# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving up a
+# restriction it had and genuinely needed to be LAN-only. These Services never had a source
+# restriction at all (crafty-allow-ingress admits `world` on the game ports, and the block above is
+# explicit that player auth — not network position — is the control). `Cluster` therefore removes no
+# protection and there is nothing to compensate for. Pinning the game ports to ${LAN_CIDR} during an
+# outage fix would be a NEW restriction that would cut off any player reaching these ports from off
+# the LAN, trading this outage for a subtler one.
 apiVersion: v1
 kind: Service
 metadata:
@@ -45,7 +103,7 @@ metadata:
     external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     app.kubernetes.io/name: crafty
   ports:
@@ -65,7 +123,7 @@ metadata:
     lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     app.kubernetes.io/name: crafty
   ports: