fix(l4): externalTrafficPolicy Cluster on unifi + crafty L4 VIPs

Cilium's L2 announcer elects one node per Service to answer ARP for its
LoadBalancer VIP, and that election is NOT backend-aware. Combined with
externalTrafficPolicy: Local -- which lets a node deliver only to a backend on
itself -- an elected announcer holding no backend answers ARP for the address
and then has nowhere to send the packet, so the client gets ICMP port
unreachable. It never self-heals: the lease holder keeps renewing after the
backend moves away.

Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on
k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on
k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and
worked. Confirmed causally on the announcing node with `cilium-dbg service
list`: under Local the frontend 192.168.2.22:25565 is absent from that node's
service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565
(active)`; flipped back, it vanishes again. Durability confirmed by cordoning
k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing
node re-pointed to the new backend with no intervention.

Both files also carried a comment claiming Local "pins the VIP to the node
running the pod", which is the precise thing that is false here; both are
rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation.

The client-IP cost is stated per service rather than waved away. For unifi it
is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into
the guest, so Local was never preserving a client address the controller could
see. For crafty it is real and accepted -- player identity is a Mojang/Xbox
UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every
player at once and must not be used; no IP ban or per-IP rate limit is
configured today.

No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml:
these Services never had a source restriction (their CNPs admit `world`), so
Cluster removes no protection, and adding a LAN-only filter during an outage
fix could cut off off-LAN players or a remote site's device informing.

No CiliumNetworkPolicy change is needed, and that was verified rather than
assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by
dumping the crafty endpoint's datapath policy map with the CNP unmodified
(`reserved:remote-node` was already an Allow on the game ports). Unlike
pihole, whose policy omits `cluster` and must name remote-node explicitly.

Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with
exactly these four Services named and passes after. whole-repo scope is
necessary: ci/touched-requirements.sh has no arm for apps/* or
platform/crafty/*, so the change most likely to reintroduce this would
attribute no requirement and be gated by nothing.

HONEST LIMIT -- this does NOT end the reported outage. All four ports also
have no listening backend: UniFi OS was never installed in the VM (its README
flags this as the open blocker) and /crafty/servers is empty, so no Minecraft
server exists. Probed directly from the node, bypassing the VIP entirely,
10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two
controls answer. This commit fixes a real latent defect that would have
blackholed those VIPs once the applications exist; it cannot make an absent
application answer.

Co-Authored-By: Claude Opus 5 <[email protected]>

apps/vms/unifi/50-service-l4.yaml

@@ -21,8 +21,61 @@
 # k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
 #
 # selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
-# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
-# VIP pinned to the node running the VM (also what L2 announcement does).
+# into the guest.
+#
+# -------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
+# -------------------------------------------------------------------------------------------
+# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
+# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
+# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
+#
+# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
+#    per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
+#    matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
+#    MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
+#    `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
+#    announcing node that holds no backend has nowhere to send the packet and answers ICMP port
+#    unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
+#    (the leases were 8 days old); the operator's controller UI and device inform path were down.
+#    It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
+#    deleting the Lease forces a re-election — and that re-election is random, so it can pick a
+#    backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
+#    reference implementation this file now follows.
+#
+# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
+#    interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
+#    SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
+#    controller has always seen the pod-side masquerade gateway as the source, never the LAN
+#    client. There is a SECOND translation below the Service, and it already destroyed the very
+#    thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
+#    it is giving up an address the guest could not see in the first place.
+#
+#    Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
+#    MAC (and the device reports its own IP inside the inform payload), not the packet source,
+#    and guest-portal authorization runs on the gateway appliance, not on this controller.
+#
+# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
+# is what makes this survive a reschedule and a rebuild.
+#
+# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
+# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
+# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
+# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
+# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
+# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
+# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
+# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
+# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
+#
+# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
+# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
+# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
+# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
+# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
+# fix would be a NEW restriction that could silently break a remote site's device informing over
+# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
+# change, made deliberately and tested against the device fleet.
 #
 # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
 # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
@@ -49,7 +102,7 @@ metadata:
     external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:
@@ -72,7 +125,7 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports: