jakman85andClaude Opus 5
4664b41db3
Cilium's L2 announcer elects one node per Service to answer ARP for its LoadBalancer VIP, and that election is NOT backend-aware. Combined with externalTrafficPolicy: Local -- which lets a node deliver only to a backend on itself -- an elected announcer holding no backend answers ARP for the address and then has nowhere to send the packet, so the client gets ICMP port unreachable. It never self-heals: the lease holder keeps renewing after the backend moves away. Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and worked. Confirmed causally on the announcing node with `cilium-dbg service list`: under Local the frontend 192.168.2.22:25565 is absent from that node's service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565 (active)`; flipped back, it vanishes again. Durability confirmed by cordoning k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing node re-pointed to the new backend with no intervention. Both files also carried a comment claiming Local "pins the VIP to the node running the pod", which is the precise thing that is false here; both are rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation. The client-IP cost is stated per service rather than waved away. For unifi it is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into the guest, so Local was never preserving a client address the controller could see. For crafty it is real and accepted -- player identity is a Mojang/Xbox UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every player at once and must not be used; no IP ban or per-IP rate limit is configured today. No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml: these Services never had a source restriction (their CNPs admit `world`), so Cluster removes no protection, and adding a LAN-only filter during an outage fix could cut off off-LAN players or a remote site's device informing. No CiliumNetworkPolicy change is needed, and that was verified rather than assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by dumping the crafty endpoint's datapath policy map with the CNP unmodified (`reserved:remote-node` was already an Allow on the game ports). Unlike pihole, whose policy omits `cluster` and must name remote-node explicitly. Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with exactly these four Services named and passes after. whole-repo scope is necessary: ci/touched-requirements.sh has no arm for apps/* or platform/crafty/*, so the change most likely to reintroduce this would attribute no requirement and be gated by nothing. HONEST LIMIT -- this does NOT end the reported outage. All four ports also have no listening backend: UniFi OS was never installed in the VM (its README flags this as the open blocker) and /crafty/servers is empty, so no Minecraft server exists. Probed directly from the node, bypassing the VIP entirely, 10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two controls answer. This commit fixes a real latent defect that would have blackholed those VIPs once the applications exist; it cannot make an absent application answer. Co-Authored-By: Claude Opus 5 <[email protected]>
96 lines
6.2 KiB
YAML
96 lines
6.2 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — an L2-announced LoadBalancer VIP must be servable from ANY announcing node
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
#
|
|
# THE BUG CLASS THIS CLOSES, AND THE OUTAGE THAT PAID FOR IT.
|
|
# Cilium puts a LoadBalancer VIP on the LAN by electing ONE node per Service to answer ARP for
|
|
# it. That election is a leader lease over the nodes the CiliumL2AnnouncementPolicy's
|
|
# nodeSelector matches, and it is NOT backend-aware — Cilium does not restrict the candidates
|
|
# to nodes running a backend, and has no field that would. Combine that with
|
|
# `externalTrafficPolicy: Local`, which lets a node deliver only to a backend on ITSELF, and
|
|
# the VIP becomes a lottery: an elected announcer holding no backend answers ARP for the
|
|
# address and then has nowhere to send the packet, so the client gets ICMP port unreachable.
|
|
# On three nodes with one replica that is a 1-in-3 chance of working, drawn independently per
|
|
# Service — and it never recovers on its own, because the lease holder keeps renewing after the
|
|
# backend moves away.
|
|
#
|
|
# MEASURED on prod 2026-09-14: unifi-l4-tcp/-udp (VM on k3s-prod-1, both leases on k3s-prod-2)
|
|
# left 192.168.2.21:8443/:8080/:11443 dead from the LAN for up to 8 days, and crafty-l4-tcp
|
|
# (pod on k3s-prod-1, lease on k3s-prod-2) left the Minecraft Java port 192.168.2.22:25565
|
|
# dead — while its twin crafty-l4-udp, same pod, happened to draw k3s-prod-1 and worked. One
|
|
# Service broken and its identical twin working is the tell. The mechanism was then confirmed
|
|
# in both directions on the announcing node with `cilium-dbg service list`: under `Local` the
|
|
# frontend is ABSENT from that node's service table; under `Cluster` it appears with an active
|
|
# backend; flipped back, it vanishes again.
|
|
#
|
|
# WHY A REVIEWER CANNOT CATCH THIS. `Local` reads as the safe, source-IP-preserving choice, and
|
|
# both shipped manifests said so in a comment that also claimed it "pins the VIP to the node
|
|
# running the pod" — the precise thing that is false here. The defect is invisible in review
|
|
# and silent at apply time; it only appears as an outage, days later, after an unrelated
|
|
# reschedule moves a pod. apps/wave-0/pihole-dns.yaml hit it first and is the reference
|
|
# implementation; this check keeps the rest of the tree on that side of the line.
|
|
#
|
|
# GIVEN the repository as git tracks it,
|
|
# WHEN every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
|
|
# that would announce its VIP,
|
|
# THEN (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
|
|
# node is guaranteed to hold a backend — true only for a DaemonSet-backed Service,
|
|
# and that exemption is computed from the tree rather than hand-maintained;
|
|
# (2) the premise is still live (some policy actually announces LoadBalancer VIPs), so
|
|
# the check cannot pass vacuously if L2 announcement is ever removed;
|
|
# (3) the announcer is still backend-BLIND — asserted structurally, so that if a future
|
|
# Cilium gains a backend-aware field the exemption logic is revisited deliberately
|
|
# instead of going stale in silence;
|
|
# AND (4) the check is NON-VACUOUS: a real checkout, a non-empty LoadBalancer corpus, and a
|
|
# floor on the assertion count.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
|
|
# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the
|
|
# SAME script.
|
|
#
|
|
# NOT proven — the runtime half, which needs a cluster: that the VIP is actually reachable,
|
|
# that a backend exists, or that an application is listening behind it. That gap is not
|
|
# academic. On 2026-09-14 all four Services above ALSO had no listening backend at all (UniFi
|
|
# OS had never been installed in the VM; no Minecraft server had ever been created in Crafty),
|
|
# so fixing this configuration defect did NOT by itself make those ports answer. A static
|
|
# check cannot see an absent application; only an L2 behavioural probe can. Nor does this
|
|
# prove `Cluster` is costless — Cilium runs in SNAT mode here, so a cross-node connection
|
|
# reaches the backend bearing the ingress node's address, and whether that is acceptable is a
|
|
# per-service judgement recorded in each manifest.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: l2-announced-vips-serve-from-any-node
|
|
labels:
|
|
conformance.home.k3s/R-DNS-01: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-DNS-01"
|
|
conformance.home.k3s/layer: "L0"
|
|
# WHOLE-REPO SCOPE, and necessarily so. ci/touched-requirements.sh's path map has NO arm for
|
|
# `apps/*` and none for `platform/crafty/*`, and nothing in it maps to R-DNS-01 at all. So the
|
|
# change most likely to break this invariant — authoring or editing an L4 LoadBalancer Service
|
|
# under apps/ or platform/, exactly where both halves of the 2026-09-14 outage lived — would
|
|
# attribute NO requirement and be gated by nothing. The invariant also spans three trees at
|
|
# once (the Services under apps/ and platform/, and the announcement policy under
|
|
# platform-config/), which no single requirement tag can select.
|
|
conformance.home.k3s/scope: "whole-repo"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
No LoadBalancer Service whose VIP is put on the LAN by Cilium L2 announcement may use
|
|
externalTrafficPolicy Local, because the announcing node is elected without regard to
|
|
backend placement — so an announcer holding no backend blackholes the VIP and never
|
|
self-heals. The sole exemption is a DaemonSet-backed Service, where every announcing
|
|
node is guaranteed a local backend.
|
|
spec:
|
|
steps:
|
|
- name: assert-no-l2-announced-vip-is-etp-local
|
|
description: >-
|
|
Pair every tracked type=LoadBalancer Service with the CiliumL2AnnouncementPolicy that
|
|
would announce its VIP, and assert none combines that announcement with
|
|
externalTrafficPolicy Local unless a DaemonSet guarantees a backend on every node.
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|