home-network-k3s-impl/conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/check.sh

#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — l2-announced-vips-serve-from-any-node
# Serves: R-DNS-01 ("Services MUST be reachable at `<service>.home.keithjackson.dev`."
#         — spec docs/00-vision-and-scope.md, section "DNS & ingress").
#
# THE BUG CLASS THIS CLOSES — it is not hypothetical, it cost an 8-day outage.
# -----------------------------------------------------------------------------
# A type=LoadBalancer Service on this cluster gets its VIP from Cilium LB-IPAM and is put on
# the LAN by a CiliumL2AnnouncementPolicy, which elects ONE node per Service to answer ARP for
# that VIP. The election is a plain leader lease among the nodes the policy's nodeSelector
# matches. IT IS NOT BACKEND-AWARE: Cilium does not restrict the candidate set to nodes that
# actually run a backend, and it has no field that would let it.
#
# With `externalTrafficPolicy: Local`, a node may only deliver to a backend on ITSELF. Compose
# the two and the VIP is a lottery: if the elected announcer holds no backend, it answers ARP
# for the address and then has nowhere to send the packet, so the client gets ICMP port
# unreachable. On three nodes with one replica that is a 1-in-3 chance of working, drawn
# independently PER SERVICE — and it does not self-heal, because the lease holder keeps
# renewing after the backend moves away. Deleting the Lease only redraws the lottery.
#
# MEASURED on prod 2026-09-14, which is why this check exists:
#   * unifi-l4-tcp / unifi-l4-udp — VM on k3s-prod-1, both leases on k3s-prod-2. 192.168.2.21
#     :8443, :8080 and :11443 dead from the LAN for up to 8 days (the leases were 8 days old).
#   * crafty-l4-tcp — pod on k3s-prod-1, lease on k3s-prod-2: 192.168.2.22:25565 (Minecraft
#     Java) dead. crafty-l4-udp — same pod, lease happened to land on k3s-prod-1, so Bedrock
#     worked. One Service broken and its twin working, from the same pod, is the whole tell.
#   * The causal mechanism was confirmed both directions on the announcing node with
#     `cilium-dbg service list`: under `Local` the frontend 192.168.2.22:25565 is ABSENT from
#     that node's service table entirely; flipped to `Cluster` it appears as
#     `1 => 10.0.0.11:25565 (active)`. Flipped back, it vanishes again.
#
# `Cluster` is the fix: every node can serve the VIP by forwarding to wherever the backend is,
# so the announcement and the backend no longer have to coincide. apps/wave-0/pihole-dns.yaml
# is the reference implementation (and carries the cost analysis); this check keeps the whole
# tree on that side of the line, because the defect is INVISIBLE in review — `Local` reads as
# the safe, source-IP-preserving choice, and the manifests that shipped it said so explicitly.
#
# GIVEN the repository as git tracks it,
# WHEN  every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
#       that would announce its VIP,
# THEN  (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
#           node is guaranteed to hold a backend — which is true only for a DaemonSet-backed
#           Service, and that exemption is computed from the tree, not hand-maintained;
#       (2) the premise is still live: at least one CiliumL2AnnouncementPolicy actually
#           announces LoadBalancer VIPs (`loadBalancerIPs: true`). If that ever becomes false
#           this check is vacuous and must be re-derived rather than silently passing;
#       (3) the announcer's candidate set really is backend-blind — asserted structurally, by
#           requiring that the policy carries no field pinning it to backend-holding nodes;
#   AND (4) the check is NON-VACUOUS: a real checkout is present, the LoadBalancer corpus is
#           non-empty, and a floor on both the corpus and the assertion count keeps a pass
#           from degenerating into a no-op when a refactor moves these files.
#
# WHY THE EXEMPTION IS NARROW, AND WHY IT IS SAFE. `Local` is correct precisely when the
# announcing node always has a local backend. Under `default-l2`'s nodeSelector
# (kubernetes.io/os=linux, i.e. every node) that holds only for a DaemonSet. A Deployment with
# replicas == node count does NOT qualify: nothing pins one replica per node, a rollout or a
# cordon breaks it, and the failure mode is this same silent outage. So the exemption keys on
# DaemonSet backing only.
#
# WHAT THIS DOES NOT PROVE (honest gaps):
#   - That the VIP is actually reachable. This is a static check; it proves the configuration
#     cannot produce the backend-blind-announcer failure, not that a backend exists, that an
#     application is listening behind it, or that the LAN can route to it. On 2026-09-14 all
#     four Services above ALSO had no listening backend (UniFi OS was never installed in the
#     VM; no Minecraft server had been created in Crafty), and no static check can see that.
#     The end-to-end claim belongs to an L2 behavioural test against a live cluster.
#   - That `Cluster` is COSTLESS. It is not: Cilium runs in SNAT mode here, so a cross-node
#     connection reaches the backend bearing the ingress node's address. Whether that cost is
#     acceptable is a per-service judgement recorded in each manifest, not something this
#     check can decide.
#   - Anything about Services created in-cluster rather than authored here (the Gateway's own
#     LoadBalancer Service is generated by Cilium from the Gateway resource and is not in the
#     tree, so it is outside this static scan).
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
PYTHON="${PYTHON:-python3}"

cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import re, subprocess, sys
from pathlib import Path

try:
    import yaml
except ImportError:                                  # pragma: no cover
    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
          file=sys.stderr)
    raise SystemExit(1)

ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
CHECKS = 0


def check(cond, msg):
    global CHECKS
    CHECKS += 1
    if not cond:
        FAILURES.append(msg)


def tracked() -> list[str]:
    out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
                         capture_output=True, text=True, check=True).stdout
    return [p for p in out.split("\0") if p]


def docs_of(rel: str):
    try:
        return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
                if isinstance(d, dict)]
    except Exception:
        return []


FILES = tracked()
YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)]
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")

# --------------------------------------------------------------------------- the announcers
# A CiliumL2AnnouncementPolicy with loadBalancerIPs: true puts LoadBalancer VIPs on the LAN.
# `serviceSelector` absent means EVERY LoadBalancer Service is announced (Cilium's default).
announcers: list[dict] = []
for rel in YAML_FILES:
    for d in docs_of(rel):
        if d.get("kind") == "CiliumL2AnnouncementPolicy":
            spec = d.get("spec") or {}
            announcers.append({
                "file": rel,
                "name": (d.get("metadata") or {}).get("name"),
                "lb_ips": bool(spec.get("loadBalancerIPs")),
                "service_selector": spec.get("serviceSelector"),
                "spec": spec,
            })

check(len(announcers) > 0,
      "no CiliumL2AnnouncementPolicy is tracked — this check's premise (VIPs are put on the "
      "LAN by L2 announcement) no longer holds and the invariant must be re-derived (R-DNS-01)")

lb_announcers = [a for a in announcers if a["lb_ips"]]
# (2) the premise is live.
check(len(lb_announcers) > 0,
      "no CiliumL2AnnouncementPolicy sets loadBalancerIPs: true — nothing announces LoadBalancer "
      "VIPs, so this check would pass vacuously; re-derive it (R-DNS-01)")

# (3) the candidate set is backend-blind. Cilium has no field that restricts L2 announcement to
#     nodes holding a backend; assert none has appeared, so that if a future Cilium adds one the
#     exemption logic here is revisited deliberately instead of silently going stale.
BACKEND_AWARE_FIELDS = {"backendSelector", "endpointSelector", "localBackendsOnly"}
for a in lb_announcers:
    present = BACKEND_AWARE_FIELDS & set(a["spec"].keys())
    check(not present,
          f"{a['file']}: CiliumL2AnnouncementPolicy/{a['name']} carries {sorted(present)}, which "
          "may make announcement backend-aware — the `Local` exemption below must be re-derived "
          "before this check can keep passing (R-DNS-01)")

# --------------------------------------------------------------------------- DaemonSet backing
# Service.spec.selector is matched against DaemonSet pod-template labels in the same namespace.
# A DaemonSet puts a backend on every node, so `Local` cannot strand an announcing node.
daemonset_labels: list[tuple[str, dict]] = []   # (namespace, template labels)
for rel in YAML_FILES:
    for d in docs_of(rel):
        if d.get("kind") == "DaemonSet":
            md = d.get("metadata") or {}
            tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {})
            daemonset_labels.append((md.get("namespace"), tmpl.get("labels") or {}))


def daemonset_backed(ns, selector) -> bool:
    if not selector:
        return False
    for ds_ns, labels in daemonset_labels:
        if ds_ns == ns and all(labels.get(k) == v for k, v in selector.items()):
            return True
    return False


# --------------------------------------------------------------------------- (1) the invariant
lb_services: list[dict] = []
for rel in YAML_FILES:
    for d in docs_of(rel):
        if d.get("kind") != "Service":
            continue
        spec = d.get("spec") or {}
        if spec.get("type") != "LoadBalancer":
            continue
        md = d.get("metadata") or {}
        lb_services.append({
            "file": rel,
            "name": md.get("name"),
            "ns": md.get("namespace"),
            "etp": spec.get("externalTrafficPolicy"),
            "selector": spec.get("selector") or {},
            "pinned": (md.get("annotations") or {}).get("lbipam.cilium.io/ips"),
        })

# Non-vacuity floor: these Services are the reason the check exists; if the corpus empties out,
# something moved and the check must be re-pointed rather than passing on nothing.
check(len(lb_services) >= 3,
      f"only {len(lb_services)} tracked type=LoadBalancer Service(s) found — the corpus this "
      "check quantifies over has moved or emptied; re-point it (R-DNS-01)")

exempt: list[str] = []
for s in lb_services:
    where = f"{s['file']}: Service/{s['ns']}/{s['name']}"
    if s["etp"] != "Local":
        CHECKS += 1                      # counted: every Service is genuinely inspected
        continue
    if daemonset_backed(s["ns"], s["selector"]):
        exempt.append(f"{s['ns']}/{s['name']}")
        CHECKS += 1
        continue
    pin = f" pinned at {s['pinned']}" if s["pinned"] else ""
    check(False,
          f"{where}{pin} combines type=LoadBalancer with externalTrafficPolicy: Local while "
          f"CiliumL2AnnouncementPolicy/{lb_announcers[0]['name'] if lb_announcers else '?'} "
          "announces its VIP from nodes chosen WITHOUT regard to backend placement. When the "
          "elected announcer holds no backend the VIP answers ICMP port unreachable and does "
          "not self-heal — this is the 8-day unifi/crafty outage of 2026-09-14. Use "
          "externalTrafficPolicy: Cluster (see apps/wave-0/pihole-dns.yaml for the pattern and "
          "the client-IP cost it trades away), or back the Service with a DaemonSet so every "
          "announcing node holds a backend (R-DNS-01)")

# --------------------------------------------------------------------------- (4) non-vacuity
check(CHECKS >= 8,
      f"only {CHECKS} assertions ran — too few for this invariant to be meaningful (R-DNS-01)")

# --------------------------------------------------------------------------------- verdict
if FAILURES:
    print(f"CHECK FAIL (l2-announced-vips-serve-from-any-node): {len(FAILURES)} of {CHECKS} "
          "assertions failed", file=sys.stderr)
    for f in FAILURES:
        print(f"  - {f}", file=sys.stderr)
    raise SystemExit(1)
note = f", {len(exempt)} DaemonSet-backed exemption(s): {sorted(exempt)}" if exempt else ""
print(f"check OK: l2-announced-vips-serve-from-any-node — {CHECKS} assertions over "
      f"{len(lb_services)} LoadBalancer Service(s) and {len(lb_announcers)} announcing "
      f"policy(ies){note}")
PY