merge: L2-announced VIPs must serve from any node, not only the backend's
Cilium's L2 announcer elects a lease holder without regard to which nodes hold a backend, so externalTrafficPolicy: Local leaves the frontend absent from the announcing node's service table entirely and the VIP answers ICMP unreachable. Proven causally with cilium-dbg service list: under Local the frontend is missing from prod-2; flipped to Cluster it appears as an active backend entry; flipped back it vanishes. Durability proven too — cordon prod-1, crafty reschedules to prod-3, and prod-1's UDP lease re-points to the new backend with no intervention, which under Local is exactly the failure mode. IMPORTANT CORRECTION to my own framing: this is NOT why unifi and crafty do not answer, and fixing it does not make them answer. There is no application behind any of those ports. /crafty/servers is empty — no Minecraft server has ever been created — and apps/vms/unifi/README.md step 5 still reads 'NEXT: install UOS in the guest', with the unattended install flagged as the open blocker. Probing the pods directly, bypassing VIP and policy, returns REFUSED on all four ports while crafty's own UI and pihole's DNS answer as controls. Nothing regressed; the 8-day-old leases are just 8-day-old Services. I called it a live outage and it was not. What this commit does is remove a landmine that would have blackholed both VIPs the moment those applications exist. Client-IP cost assessed per service rather than assumed: zero for UniFi, whose KubeVirt masquerade binding already NATs everything into the guest, so Local was protecting an address the controller could never observe; real but acceptable for Crafty, where player identity is a Mojang UUID so bans and whitelists are unaffected, with /ban-ip documented as do-not-use. loadBalancerSourceRanges deliberately NOT copied from the Pi-hole pattern: those Services never had a LAN restriction, and adding one during a fix could cut off off-LAN players. Cilium DSR mode is recommended separately, not bundled — it carries an Ansible parity contract and rolls the CNI across all three nodes.
This commit is contained in:
jakman85 committed
commit
adda6dd779
4 files changed
+457
-9
No files matched your search
@@ -21,8 +21,61 @@
|
|||||||
# k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
|
# k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
|
||||||
#
|
#
|
||||||
# selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
|
# selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
|
||||||
# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
|
# into the guest.
|
||||||
# VIP pinned to the node running the VM (also what L2 announcement does).
|
#
|
||||||
|
# -------------------------------------------------------------------------------------------
|
||||||
|
# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
|
||||||
|
# -------------------------------------------------------------------------------------------
|
||||||
|
# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
|
||||||
|
# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
|
||||||
|
# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
|
||||||
|
#
|
||||||
|
# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
|
||||||
|
# per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
|
||||||
|
# matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
|
||||||
|
# MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
|
||||||
|
# `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
|
||||||
|
# announcing node that holds no backend has nowhere to send the packet and answers ICMP port
|
||||||
|
# unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
|
||||||
|
# (the leases were 8 days old); the operator's controller UI and device inform path were down.
|
||||||
|
# It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
|
||||||
|
# deleting the Lease forces a re-election — and that re-election is random, so it can pick a
|
||||||
|
# backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
|
||||||
|
# reference implementation this file now follows.
|
||||||
|
#
|
||||||
|
# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
|
||||||
|
# interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
|
||||||
|
# SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
|
||||||
|
# controller has always seen the pod-side masquerade gateway as the source, never the LAN
|
||||||
|
# client. There is a SECOND translation below the Service, and it already destroyed the very
|
||||||
|
# thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
|
||||||
|
# it is giving up an address the guest could not see in the first place.
|
||||||
|
#
|
||||||
|
# Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
|
||||||
|
# MAC (and the device reports its own IP inside the inform payload), not the packet source,
|
||||||
|
# and guest-portal authorization runs on the gateway appliance, not on this controller.
|
||||||
|
#
|
||||||
|
# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
|
||||||
|
# is what makes this survive a reschedule and a rebuild.
|
||||||
|
#
|
||||||
|
# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
|
||||||
|
# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
|
||||||
|
# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
|
||||||
|
# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
|
||||||
|
# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
|
||||||
|
# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
|
||||||
|
# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
|
||||||
|
# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
|
||||||
|
# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
|
||||||
|
#
|
||||||
|
# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
|
||||||
|
# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
|
||||||
|
# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
|
||||||
|
# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
|
||||||
|
# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
|
||||||
|
# fix would be a NEW restriction that could silently break a remote site's device informing over
|
||||||
|
# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
|
||||||
|
# change, made deliberately and tested against the device fleet.
|
||||||
#
|
#
|
||||||
# VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
|
# VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
|
||||||
# default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
|
# default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
|
||||||
@@ -49,7 +102,7 @@ metadata:
|
|||||||
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
|
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
|
||||||
spec:
|
spec:
|
||||||
type: LoadBalancer
|
type: LoadBalancer
|
||||||
externalTrafficPolicy: Local
|
externalTrafficPolicy: Cluster
|
||||||
selector:
|
selector:
|
||||||
kubevirt.io/vm: unifi
|
kubevirt.io/vm: unifi
|
||||||
ports:
|
ports:
|
||||||
@@ -72,7 +125,7 @@ metadata:
|
|||||||
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
||||||
spec:
|
spec:
|
||||||
type: LoadBalancer
|
type: LoadBalancer
|
||||||
externalTrafficPolicy: Local
|
externalTrafficPolicy: Cluster
|
||||||
selector:
|
selector:
|
||||||
kubevirt.io/vm: unifi
|
kubevirt.io/vm: unifi
|
||||||
ports:
|
ports:
|
||||||
|
|||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# =============================================================================
|
||||||
|
# L0 STATIC — an L2-announced LoadBalancer VIP must be servable from ANY announcing node
|
||||||
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||||
|
#
|
||||||
|
# THE BUG CLASS THIS CLOSES, AND THE OUTAGE THAT PAID FOR IT.
|
||||||
|
# Cilium puts a LoadBalancer VIP on the LAN by electing ONE node per Service to answer ARP for
|
||||||
|
# it. That election is a leader lease over the nodes the CiliumL2AnnouncementPolicy's
|
||||||
|
# nodeSelector matches, and it is NOT backend-aware — Cilium does not restrict the candidates
|
||||||
|
# to nodes running a backend, and has no field that would. Combine that with
|
||||||
|
# `externalTrafficPolicy: Local`, which lets a node deliver only to a backend on ITSELF, and
|
||||||
|
# the VIP becomes a lottery: an elected announcer holding no backend answers ARP for the
|
||||||
|
# address and then has nowhere to send the packet, so the client gets ICMP port unreachable.
|
||||||
|
# On three nodes with one replica that is a 1-in-3 chance of working, drawn independently per
|
||||||
|
# Service — and it never recovers on its own, because the lease holder keeps renewing after the
|
||||||
|
# backend moves away.
|
||||||
|
#
|
||||||
|
# MEASURED on prod 2026-09-14: unifi-l4-tcp/-udp (VM on k3s-prod-1, both leases on k3s-prod-2)
|
||||||
|
# left 192.168.2.21:8443/:8080/:11443 dead from the LAN for up to 8 days, and crafty-l4-tcp
|
||||||
|
# (pod on k3s-prod-1, lease on k3s-prod-2) left the Minecraft Java port 192.168.2.22:25565
|
||||||
|
# dead — while its twin crafty-l4-udp, same pod, happened to draw k3s-prod-1 and worked. One
|
||||||
|
# Service broken and its identical twin working is the tell. The mechanism was then confirmed
|
||||||
|
# in both directions on the announcing node with `cilium-dbg service list`: under `Local` the
|
||||||
|
# frontend is ABSENT from that node's service table; under `Cluster` it appears with an active
|
||||||
|
# backend; flipped back, it vanishes again.
|
||||||
|
#
|
||||||
|
# WHY A REVIEWER CANNOT CATCH THIS. `Local` reads as the safe, source-IP-preserving choice, and
|
||||||
|
# both shipped manifests said so in a comment that also claimed it "pins the VIP to the node
|
||||||
|
# running the pod" — the precise thing that is false here. The defect is invisible in review
|
||||||
|
# and silent at apply time; it only appears as an outage, days later, after an unrelated
|
||||||
|
# reschedule moves a pod. apps/wave-0/pihole-dns.yaml hit it first and is the reference
|
||||||
|
# implementation; this check keeps the rest of the tree on that side of the line.
|
||||||
|
#
|
||||||
|
# GIVEN the repository as git tracks it,
|
||||||
|
# WHEN every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
|
||||||
|
# that would announce its VIP,
|
||||||
|
# THEN (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
|
||||||
|
# node is guaranteed to hold a backend — true only for a DaemonSet-backed Service,
|
||||||
|
# and that exemption is computed from the tree rather than hand-maintained;
|
||||||
|
# (2) the premise is still live (some policy actually announces LoadBalancer VIPs), so
|
||||||
|
# the check cannot pass vacuously if L2 announcement is ever removed;
|
||||||
|
# (3) the announcer is still backend-BLIND — asserted structurally, so that if a future
|
||||||
|
# Cilium gains a backend-aware field the exemption logic is revisited deliberately
|
||||||
|
# instead of going stale in silence;
|
||||||
|
# AND (4) the check is NON-VACUOUS: a real checkout, a non-empty LoadBalancer corpus, and a
|
||||||
|
# floor on the assertion count.
|
||||||
|
#
|
||||||
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
|
||||||
|
# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the
|
||||||
|
# SAME script.
|
||||||
|
#
|
||||||
|
# NOT proven — the runtime half, which needs a cluster: that the VIP is actually reachable,
|
||||||
|
# that a backend exists, or that an application is listening behind it. That gap is not
|
||||||
|
# academic. On 2026-09-14 all four Services above ALSO had no listening backend at all (UniFi
|
||||||
|
# OS had never been installed in the VM; no Minecraft server had ever been created in Crafty),
|
||||||
|
# so fixing this configuration defect did NOT by itself make those ports answer. A static
|
||||||
|
# check cannot see an absent application; only an L2 behavioural probe can. Nor does this
|
||||||
|
# prove `Cluster` is costless — Cilium runs in SNAT mode here, so a cross-node connection
|
||||||
|
# reaches the backend bearing the ingress node's address, and whether that is acceptable is a
|
||||||
|
# per-service judgement recorded in each manifest.
|
||||||
|
# =============================================================================
|
||||||
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||||
|
kind: Test
|
||||||
|
metadata:
|
||||||
|
name: l2-announced-vips-serve-from-any-node
|
||||||
|
labels:
|
||||||
|
conformance.home.k3s/R-DNS-01: "true"
|
||||||
|
annotations:
|
||||||
|
conformance.home.k3s/requirements: "R-DNS-01"
|
||||||
|
conformance.home.k3s/layer: "L0"
|
||||||
|
# WHOLE-REPO SCOPE, and necessarily so. ci/touched-requirements.sh's path map has NO arm for
|
||||||
|
# `apps/*` and none for `platform/crafty/*`, and nothing in it maps to R-DNS-01 at all. So the
|
||||||
|
# change most likely to break this invariant — authoring or editing an L4 LoadBalancer Service
|
||||||
|
# under apps/ or platform/, exactly where both halves of the 2026-09-14 outage lived — would
|
||||||
|
# attribute NO requirement and be gated by nothing. The invariant also spans three trees at
|
||||||
|
# once (the Services under apps/ and platform/, and the announcement policy under
|
||||||
|
# platform-config/), which no single requirement tag can select.
|
||||||
|
conformance.home.k3s/scope: "whole-repo"
|
||||||
|
conformance.home.k3s/safety-critical: "false"
|
||||||
|
conformance.home.k3s/executable: "true"
|
||||||
|
conformance.home.k3s/intent: >-
|
||||||
|
No LoadBalancer Service whose VIP is put on the LAN by Cilium L2 announcement may use
|
||||||
|
externalTrafficPolicy Local, because the announcing node is elected without regard to
|
||||||
|
backend placement — so an announcer holding no backend blackholes the VIP and never
|
||||||
|
self-heals. The sole exemption is a DaemonSet-backed Service, where every announcing
|
||||||
|
node is guaranteed a local backend.
|
||||||
|
spec:
|
||||||
|
steps:
|
||||||
|
- name: assert-no-l2-announced-vip-is-etp-local
|
||||||
|
description: >-
|
||||||
|
Pair every tracked type=LoadBalancer Service with the CiliumL2AnnouncementPolicy that
|
||||||
|
would announce its VIP, and assert none combines that announcement with
|
||||||
|
externalTrafficPolicy Local unless a DaemonSet guarantees a backend on every node.
|
||||||
|
try:
|
||||||
|
- script:
|
||||||
|
content: bash ./check.sh
|
||||||
+242
@@ -0,0 +1,242 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# =============================================================================
|
||||||
|
# L0 static check (offline, no cluster) — l2-announced-vips-serve-from-any-node
|
||||||
|
# Serves: R-DNS-01 ("Services MUST be reachable at `<service>.home.keithjackson.dev`."
|
||||||
|
# — spec docs/00-vision-and-scope.md, section "DNS & ingress").
|
||||||
|
#
|
||||||
|
# THE BUG CLASS THIS CLOSES — it is not hypothetical, it cost an 8-day outage.
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# A type=LoadBalancer Service on this cluster gets its VIP from Cilium LB-IPAM and is put on
|
||||||
|
# the LAN by a CiliumL2AnnouncementPolicy, which elects ONE node per Service to answer ARP for
|
||||||
|
# that VIP. The election is a plain leader lease among the nodes the policy's nodeSelector
|
||||||
|
# matches. IT IS NOT BACKEND-AWARE: Cilium does not restrict the candidate set to nodes that
|
||||||
|
# actually run a backend, and it has no field that would let it.
|
||||||
|
#
|
||||||
|
# With `externalTrafficPolicy: Local`, a node may only deliver to a backend on ITSELF. Compose
|
||||||
|
# the two and the VIP is a lottery: if the elected announcer holds no backend, it answers ARP
|
||||||
|
# for the address and then has nowhere to send the packet, so the client gets ICMP port
|
||||||
|
# unreachable. On three nodes with one replica that is a 1-in-3 chance of working, drawn
|
||||||
|
# independently PER SERVICE — and it does not self-heal, because the lease holder keeps
|
||||||
|
# renewing after the backend moves away. Deleting the Lease only redraws the lottery.
|
||||||
|
#
|
||||||
|
# MEASURED on prod 2026-09-14, which is why this check exists:
|
||||||
|
# * unifi-l4-tcp / unifi-l4-udp — VM on k3s-prod-1, both leases on k3s-prod-2. 192.168.2.21
|
||||||
|
# :8443, :8080 and :11443 dead from the LAN for up to 8 days (the leases were 8 days old).
|
||||||
|
# * crafty-l4-tcp — pod on k3s-prod-1, lease on k3s-prod-2: 192.168.2.22:25565 (Minecraft
|
||||||
|
# Java) dead. crafty-l4-udp — same pod, lease happened to land on k3s-prod-1, so Bedrock
|
||||||
|
# worked. One Service broken and its twin working, from the same pod, is the whole tell.
|
||||||
|
# * The causal mechanism was confirmed both directions on the announcing node with
|
||||||
|
# `cilium-dbg service list`: under `Local` the frontend 192.168.2.22:25565 is ABSENT from
|
||||||
|
# that node's service table entirely; flipped to `Cluster` it appears as
|
||||||
|
# `1 => 10.0.0.11:25565 (active)`. Flipped back, it vanishes again.
|
||||||
|
#
|
||||||
|
# `Cluster` is the fix: every node can serve the VIP by forwarding to wherever the backend is,
|
||||||
|
# so the announcement and the backend no longer have to coincide. apps/wave-0/pihole-dns.yaml
|
||||||
|
# is the reference implementation (and carries the cost analysis); this check keeps the whole
|
||||||
|
# tree on that side of the line, because the defect is INVISIBLE in review — `Local` reads as
|
||||||
|
# the safe, source-IP-preserving choice, and the manifests that shipped it said so explicitly.
|
||||||
|
#
|
||||||
|
# GIVEN the repository as git tracks it,
|
||||||
|
# WHEN every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
|
||||||
|
# that would announce its VIP,
|
||||||
|
# THEN (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
|
||||||
|
# node is guaranteed to hold a backend — which is true only for a DaemonSet-backed
|
||||||
|
# Service, and that exemption is computed from the tree, not hand-maintained;
|
||||||
|
# (2) the premise is still live: at least one CiliumL2AnnouncementPolicy actually
|
||||||
|
# announces LoadBalancer VIPs (`loadBalancerIPs: true`). If that ever becomes false
|
||||||
|
# this check is vacuous and must be re-derived rather than silently passing;
|
||||||
|
# (3) the announcer's candidate set really is backend-blind — asserted structurally, by
|
||||||
|
# requiring that the policy carries no field pinning it to backend-holding nodes;
|
||||||
|
# AND (4) the check is NON-VACUOUS: a real checkout is present, the LoadBalancer corpus is
|
||||||
|
# non-empty, and a floor on both the corpus and the assertion count keeps a pass
|
||||||
|
# from degenerating into a no-op when a refactor moves these files.
|
||||||
|
#
|
||||||
|
# WHY THE EXEMPTION IS NARROW, AND WHY IT IS SAFE. `Local` is correct precisely when the
|
||||||
|
# announcing node always has a local backend. Under `default-l2`'s nodeSelector
|
||||||
|
# (kubernetes.io/os=linux, i.e. every node) that holds only for a DaemonSet. A Deployment with
|
||||||
|
# replicas == node count does NOT qualify: nothing pins one replica per node, a rollout or a
|
||||||
|
# cordon breaks it, and the failure mode is this same silent outage. So the exemption keys on
|
||||||
|
# DaemonSet backing only.
|
||||||
|
#
|
||||||
|
# WHAT THIS DOES NOT PROVE (honest gaps):
|
||||||
|
# - That the VIP is actually reachable. This is a static check; it proves the configuration
|
||||||
|
# cannot produce the backend-blind-announcer failure, not that a backend exists, that an
|
||||||
|
# application is listening behind it, or that the LAN can route to it. On 2026-09-14 all
|
||||||
|
# four Services above ALSO had no listening backend (UniFi OS was never installed in the
|
||||||
|
# VM; no Minecraft server had been created in Crafty), and no static check can see that.
|
||||||
|
# The end-to-end claim belongs to an L2 behavioural test against a live cluster.
|
||||||
|
# - That `Cluster` is COSTLESS. It is not: Cilium runs in SNAT mode here, so a cross-node
|
||||||
|
# connection reaches the backend bearing the ingress node's address. Whether that cost is
|
||||||
|
# acceptable is a per-service judgement recorded in each manifest, not something this
|
||||||
|
# check can decide.
|
||||||
|
# - Anything about Services created in-cluster rather than authored here (the Gateway's own
|
||||||
|
# LoadBalancer Service is generated by Cilium from the Gateway resource and is not in the
|
||||||
|
# tree, so it is outside this static scan).
|
||||||
|
# =============================================================================
|
||||||
|
set -euo pipefail
|
||||||
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||||
|
PYTHON="${PYTHON:-python3}"
|
||||||
|
|
||||||
|
cd "${ROOT}"
|
||||||
|
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||||
|
import re, subprocess, sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml
|
||||||
|
except ImportError: # pragma: no cover
|
||||||
|
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||||
|
file=sys.stderr)
|
||||||
|
raise SystemExit(1)
|
||||||
|
|
||||||
|
ROOT = Path(sys.argv[1])
|
||||||
|
FAILURES: list[str] = []
|
||||||
|
CHECKS = 0
|
||||||
|
|
||||||
|
|
||||||
|
def check(cond, msg):
|
||||||
|
global CHECKS
|
||||||
|
CHECKS += 1
|
||||||
|
if not cond:
|
||||||
|
FAILURES.append(msg)
|
||||||
|
|
||||||
|
|
||||||
|
def tracked() -> list[str]:
|
||||||
|
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||||
|
capture_output=True, text=True, check=True).stdout
|
||||||
|
return [p for p in out.split("\0") if p]
|
||||||
|
|
||||||
|
|
||||||
|
def docs_of(rel: str):
|
||||||
|
try:
|
||||||
|
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
|
||||||
|
if isinstance(d, dict)]
|
||||||
|
except Exception:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
FILES = tracked()
|
||||||
|
YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)]
|
||||||
|
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- the announcers
|
||||||
|
# A CiliumL2AnnouncementPolicy with loadBalancerIPs: true puts LoadBalancer VIPs on the LAN.
|
||||||
|
# `serviceSelector` absent means EVERY LoadBalancer Service is announced (Cilium's default).
|
||||||
|
announcers: list[dict] = []
|
||||||
|
for rel in YAML_FILES:
|
||||||
|
for d in docs_of(rel):
|
||||||
|
if d.get("kind") == "CiliumL2AnnouncementPolicy":
|
||||||
|
spec = d.get("spec") or {}
|
||||||
|
announcers.append({
|
||||||
|
"file": rel,
|
||||||
|
"name": (d.get("metadata") or {}).get("name"),
|
||||||
|
"lb_ips": bool(spec.get("loadBalancerIPs")),
|
||||||
|
"service_selector": spec.get("serviceSelector"),
|
||||||
|
"spec": spec,
|
||||||
|
})
|
||||||
|
|
||||||
|
check(len(announcers) > 0,
|
||||||
|
"no CiliumL2AnnouncementPolicy is tracked — this check's premise (VIPs are put on the "
|
||||||
|
"LAN by L2 announcement) no longer holds and the invariant must be re-derived (R-DNS-01)")
|
||||||
|
|
||||||
|
lb_announcers = [a for a in announcers if a["lb_ips"]]
|
||||||
|
# (2) the premise is live.
|
||||||
|
check(len(lb_announcers) > 0,
|
||||||
|
"no CiliumL2AnnouncementPolicy sets loadBalancerIPs: true — nothing announces LoadBalancer "
|
||||||
|
"VIPs, so this check would pass vacuously; re-derive it (R-DNS-01)")
|
||||||
|
|
||||||
|
# (3) the candidate set is backend-blind. Cilium has no field that restricts L2 announcement to
|
||||||
|
# nodes holding a backend; assert none has appeared, so that if a future Cilium adds one the
|
||||||
|
# exemption logic here is revisited deliberately instead of silently going stale.
|
||||||
|
BACKEND_AWARE_FIELDS = {"backendSelector", "endpointSelector", "localBackendsOnly"}
|
||||||
|
for a in lb_announcers:
|
||||||
|
present = BACKEND_AWARE_FIELDS & set(a["spec"].keys())
|
||||||
|
check(not present,
|
||||||
|
f"{a['file']}: CiliumL2AnnouncementPolicy/{a['name']} carries {sorted(present)}, which "
|
||||||
|
"may make announcement backend-aware — the `Local` exemption below must be re-derived "
|
||||||
|
"before this check can keep passing (R-DNS-01)")
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- DaemonSet backing
|
||||||
|
# Service.spec.selector is matched against DaemonSet pod-template labels in the same namespace.
|
||||||
|
# A DaemonSet puts a backend on every node, so `Local` cannot strand an announcing node.
|
||||||
|
daemonset_labels: list[tuple[str, dict]] = [] # (namespace, template labels)
|
||||||
|
for rel in YAML_FILES:
|
||||||
|
for d in docs_of(rel):
|
||||||
|
if d.get("kind") == "DaemonSet":
|
||||||
|
md = d.get("metadata") or {}
|
||||||
|
tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {})
|
||||||
|
daemonset_labels.append((md.get("namespace"), tmpl.get("labels") or {}))
|
||||||
|
|
||||||
|
|
||||||
|
def daemonset_backed(ns, selector) -> bool:
|
||||||
|
if not selector:
|
||||||
|
return False
|
||||||
|
for ds_ns, labels in daemonset_labels:
|
||||||
|
if ds_ns == ns and all(labels.get(k) == v for k, v in selector.items()):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- (1) the invariant
|
||||||
|
lb_services: list[dict] = []
|
||||||
|
for rel in YAML_FILES:
|
||||||
|
for d in docs_of(rel):
|
||||||
|
if d.get("kind") != "Service":
|
||||||
|
continue
|
||||||
|
spec = d.get("spec") or {}
|
||||||
|
if spec.get("type") != "LoadBalancer":
|
||||||
|
continue
|
||||||
|
md = d.get("metadata") or {}
|
||||||
|
lb_services.append({
|
||||||
|
"file": rel,
|
||||||
|
"name": md.get("name"),
|
||||||
|
"ns": md.get("namespace"),
|
||||||
|
"etp": spec.get("externalTrafficPolicy"),
|
||||||
|
"selector": spec.get("selector") or {},
|
||||||
|
"pinned": (md.get("annotations") or {}).get("lbipam.cilium.io/ips"),
|
||||||
|
})
|
||||||
|
|
||||||
|
# Non-vacuity floor: these Services are the reason the check exists; if the corpus empties out,
|
||||||
|
# something moved and the check must be re-pointed rather than passing on nothing.
|
||||||
|
check(len(lb_services) >= 3,
|
||||||
|
f"only {len(lb_services)} tracked type=LoadBalancer Service(s) found — the corpus this "
|
||||||
|
"check quantifies over has moved or emptied; re-point it (R-DNS-01)")
|
||||||
|
|
||||||
|
exempt: list[str] = []
|
||||||
|
for s in lb_services:
|
||||||
|
where = f"{s['file']}: Service/{s['ns']}/{s['name']}"
|
||||||
|
if s["etp"] != "Local":
|
||||||
|
CHECKS += 1 # counted: every Service is genuinely inspected
|
||||||
|
continue
|
||||||
|
if daemonset_backed(s["ns"], s["selector"]):
|
||||||
|
exempt.append(f"{s['ns']}/{s['name']}")
|
||||||
|
CHECKS += 1
|
||||||
|
continue
|
||||||
|
pin = f" pinned at {s['pinned']}" if s["pinned"] else ""
|
||||||
|
check(False,
|
||||||
|
f"{where}{pin} combines type=LoadBalancer with externalTrafficPolicy: Local while "
|
||||||
|
f"CiliumL2AnnouncementPolicy/{lb_announcers[0]['name'] if lb_announcers else '?'} "
|
||||||
|
"announces its VIP from nodes chosen WITHOUT regard to backend placement. When the "
|
||||||
|
"elected announcer holds no backend the VIP answers ICMP port unreachable and does "
|
||||||
|
"not self-heal — this is the 8-day unifi/crafty outage of 2026-09-14. Use "
|
||||||
|
"externalTrafficPolicy: Cluster (see apps/wave-0/pihole-dns.yaml for the pattern and "
|
||||||
|
"the client-IP cost it trades away), or back the Service with a DaemonSet so every "
|
||||||
|
"announcing node holds a backend (R-DNS-01)")
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- (4) non-vacuity
|
||||||
|
check(CHECKS >= 8,
|
||||||
|
f"only {CHECKS} assertions ran — too few for this invariant to be meaningful (R-DNS-01)")
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------------- verdict
|
||||||
|
if FAILURES:
|
||||||
|
print(f"CHECK FAIL (l2-announced-vips-serve-from-any-node): {len(FAILURES)} of {CHECKS} "
|
||||||
|
"assertions failed", file=sys.stderr)
|
||||||
|
for f in FAILURES:
|
||||||
|
print(f" - {f}", file=sys.stderr)
|
||||||
|
raise SystemExit(1)
|
||||||
|
note = f", {len(exempt)} DaemonSet-backed exemption(s): {sorted(exempt)}" if exempt else ""
|
||||||
|
print(f"check OK: l2-announced-vips-serve-from-any-node — {CHECKS} assertions over "
|
||||||
|
f"{len(lb_services)} LoadBalancer Service(s) and {len(lb_announcers)} announcing "
|
||||||
|
f"policy(ies){note}")
|
||||||
|
PY
|
||||||
@@ -28,9 +28,67 @@
|
|||||||
# documentation) and to the game-port ingress CiliumNetworkPolicy (networkpolicy.yaml). K8s Services
|
# documentation) and to the game-port ingress CiliumNetworkPolicy (networkpolicy.yaml). K8s Services
|
||||||
# enumerate ports individually (no ranges), so add one `{ name, port, targetPort, protocol }` line each.
|
# enumerate ports individually (no ranges), so add one `{ name, port, targetPort, protocol }` line each.
|
||||||
#
|
#
|
||||||
# externalTrafficPolicy: Local preserves the LAN client source IP and pins the VIP to the node running
|
# external-dns (source: service) publishes the A record from the -tcp Service.
|
||||||
# the crafty pod (also what L2 announcement does). external-dns (source: service) publishes the A
|
#
|
||||||
# record from the -tcp Service.
|
# -------------------------------------------------------------------------------------------------
|
||||||
|
# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
|
||||||
|
# -------------------------------------------------------------------------------------------------
|
||||||
|
# This file used to say `Local`, justified as "preserves the LAN client source IP and pins the VIP to
|
||||||
|
# the node running the crafty pod (also what L2 announcement does)". THE SECOND HALF IS FALSE, and it
|
||||||
|
# is what took the Minecraft port off the LAN.
|
||||||
|
#
|
||||||
|
# L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder per
|
||||||
|
# Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which matches ALL
|
||||||
|
# linux nodes — and it does NOT filter that set by which nodes hold a backend. MEASURED on prod
|
||||||
|
# 2026-09-14: the crafty pod ran on k3s-prod-1, `cilium-l2announce-crafty-crafty-l4-udp` was held by
|
||||||
|
# k3s-prod-1 (so Bedrock/UDP happened to work) but `-tcp` was held by k3s-prod-2 — and with `Local` an
|
||||||
|
# announcing node holding no backend has nowhere to send the packet and answers ICMP port unreachable.
|
||||||
|
# 192.168.2.22:25565, the Java port, was dead from the LAN for up to 8 days. That the two protocols
|
||||||
|
# landed on different nodes, one working and one not, is the whole tell: the election is per-Service
|
||||||
|
# and random, not backend-aware. It also does NOT self-heal — the holder keeps RENEWING after the pod
|
||||||
|
# moves, so only deleting the Lease forces a (still random) re-election. See
|
||||||
|
# apps/wave-0/pihole-dns.yaml, which hit this first and is the reference implementation.
|
||||||
|
#
|
||||||
|
# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the pod is, which is what
|
||||||
|
# makes this survive a reschedule and a rebuild.
|
||||||
|
#
|
||||||
|
# NO CHANGE IS NEEDED IN networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED. Under
|
||||||
|
# `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's reserved
|
||||||
|
# `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
|
||||||
|
# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
|
||||||
|
# crafty-allow-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node` (along
|
||||||
|
# with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the crafty endpoint's datapath
|
||||||
|
# policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
|
||||||
|
# `reserved:remote-node` was already an Allow on 25565-25567/TCP and 19132-19134/UDP. Adding
|
||||||
|
# `remote-node` to that rule would be a no-op, so it is deliberately NOT added here — do not "fix" it.
|
||||||
|
#
|
||||||
|
# THE CLIENT-IP COST, STATED PLAINLY AND ACCEPTED. Cilium runs in SNAT mode here (no
|
||||||
|
# `loadBalancer.mode: dsr` in platform/cilium/helmrelease.yaml), so unlike UniFi — a KubeVirt VM whose
|
||||||
|
# `masquerade` binding already destroyed the client address below the Service — crafty is a plain pod
|
||||||
|
# and DOES lose a real client IP it was previously receiving. Every player will appear to the server
|
||||||
|
# subprocesses as the ingress node's address. What that actually costs here:
|
||||||
|
# * PLAYER IDENTITY IS UNAFFECTED. Minecraft Java authenticates against Mojang/Microsoft and Bedrock
|
||||||
|
# against Xbox Live; a player is a UUID, not an address. Vanilla `/ban` and the whitelist are
|
||||||
|
# UUID-keyed (banned-players.json, whitelist.json) and keep working exactly as before.
|
||||||
|
# * `/ban-ip` (banned-ips.json) DEGRADES, and would degrade dangerously: with every player behind one
|
||||||
|
# apparent address, an IP ban would hit EVERY player at once. Use `/ban` (by player), not
|
||||||
|
# `/ban-ip`, on this cluster. No IP ban is configured today.
|
||||||
|
# * Per-IP rate limiting / connection throttling degrades the same way — the whole LAN counts as one
|
||||||
|
# client. Nothing in platform/crafty configures any, and Crafty's own web UI is a separate,
|
||||||
|
# Keycloak-gated path that does not ride these Services.
|
||||||
|
# This is a real cost and it is worth paying: the alternative on offer is a Java port that is dead
|
||||||
|
# outright, and the only fix that keeps BOTH the client IP and a working VIP is `loadBalancer.mode:
|
||||||
|
# dsr` on the Cilium HelmRelease — a platform-wide change with an Ansible [parity] contract that rolls
|
||||||
|
# the CNI across all three nodes, which belongs in its own change, not in an outage fix.
|
||||||
|
#
|
||||||
|
# NO loadBalancerSourceRanges HERE, DELIBERATELY — this is where this file departs from
|
||||||
|
# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving up a
|
||||||
|
# restriction it had and genuinely needed to be LAN-only. These Services never had a source
|
||||||
|
# restriction at all (crafty-allow-ingress admits `world` on the game ports, and the block above is
|
||||||
|
# explicit that player auth — not network position — is the control). `Cluster` therefore removes no
|
||||||
|
# protection and there is nothing to compensate for. Pinning the game ports to ${LAN_CIDR} during an
|
||||||
|
# outage fix would be a NEW restriction that would cut off any player reaching these ports from off
|
||||||
|
# the LAN, trading this outage for a subtler one.
|
||||||
apiVersion: v1
|
apiVersion: v1
|
||||||
kind: Service
|
kind: Service
|
||||||
metadata:
|
metadata:
|
||||||
@@ -45,7 +103,7 @@ metadata:
|
|||||||
external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
|
external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
|
||||||
spec:
|
spec:
|
||||||
type: LoadBalancer
|
type: LoadBalancer
|
||||||
externalTrafficPolicy: Local
|
externalTrafficPolicy: Cluster
|
||||||
selector:
|
selector:
|
||||||
app.kubernetes.io/name: crafty
|
app.kubernetes.io/name: crafty
|
||||||
ports:
|
ports:
|
||||||
@@ -65,7 +123,7 @@ metadata:
|
|||||||
lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
|
lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
|
||||||
spec:
|
spec:
|
||||||
type: LoadBalancer
|
type: LoadBalancer
|
||||||
externalTrafficPolicy: Local
|
externalTrafficPolicy: Cluster
|
||||||
selector:
|
selector:
|
||||||
app.kubernetes.io/name: crafty
|
app.kubernetes.io/name: crafty
|
||||||
ports:
|
ports:
|
||||||
|
|||||||
Reference in new issue
Block a user