jakman85andClaude Opus 5
4664b41db3
Cilium's L2 announcer elects one node per Service to answer ARP for its LoadBalancer VIP, and that election is NOT backend-aware. Combined with externalTrafficPolicy: Local -- which lets a node deliver only to a backend on itself -- an elected announcer holding no backend answers ARP for the address and then has nowhere to send the packet, so the client gets ICMP port unreachable. It never self-heals: the lease holder keeps renewing after the backend moves away. Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and worked. Confirmed causally on the announcing node with `cilium-dbg service list`: under Local the frontend 192.168.2.22:25565 is absent from that node's service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565 (active)`; flipped back, it vanishes again. Durability confirmed by cordoning k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing node re-pointed to the new backend with no intervention. Both files also carried a comment claiming Local "pins the VIP to the node running the pod", which is the precise thing that is false here; both are rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation. The client-IP cost is stated per service rather than waved away. For unifi it is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into the guest, so Local was never preserving a client address the controller could see. For crafty it is real and accepted -- player identity is a Mojang/Xbox UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every player at once and must not be used; no IP ban or per-IP rate limit is configured today. No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml: these Services never had a source restriction (their CNPs admit `world`), so Cluster removes no protection, and adding a LAN-only filter during an outage fix could cut off off-LAN players or a remote site's device informing. No CiliumNetworkPolicy change is needed, and that was verified rather than assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by dumping the crafty endpoint's datapath policy map with the CNP unmodified (`reserved:remote-node` was already an Allow on the game ports). Unlike pihole, whose policy omits `cluster` and must name remote-node explicitly. Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with exactly these four Services named and passes after. whole-repo scope is necessary: ci/touched-requirements.sh has no arm for apps/* or platform/crafty/*, so the change most likely to reintroduce this would attribute no requirement and be gated by nothing. HONEST LIMIT -- this does NOT end the reported outage. All four ports also have no listening backend: UniFi OS was never installed in the VM (its README flags this as the open blocker) and /crafty/servers is empty, so no Minecraft server exists. Probed directly from the node, bypassing the VIP entirely, 10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two controls answer. This commit fixes a real latent defect that would have blackholed those VIPs once the applications exist; it cannot make an absent application answer. Co-Authored-By: Claude Opus 5 <[email protected]>
134 lines
8.2 KiB
YAML
134 lines
8.2 KiB
YAML
---
|
|
# LIVE — reconciled by the non-gating `apps-vms` Flux Kustomization (see namespace.yaml header).
|
|
#
|
|
# ============================================================================================
|
|
# L4 EXPOSURE (the hard part) — why the shared HTTPS Gateway is NOT enough for UniFi.
|
|
# ============================================================================================
|
|
# The Gateway API `shared` Gateway is an L7 HTTPS terminator (host-routed :443 -> backend). It
|
|
# CANNOT express: raw TCP on non-HTTP ports (8080 inform, 6789), UDP at all (3478 STUN, 10001
|
|
# discovery), or the device-facing protocols that never speak HTTP. UniFi device ADOPTION rides
|
|
# exactly those L4 ports. So the controller is exposed as a type=LoadBalancer Service whose VIP
|
|
# is pulled from the CiliumLoadBalancerIPPool `default-pool` (dev 192.168.1.53-.60 — the
|
|
# CILIUM_LB_POOL_START/STOP cluster-vars knobs) and ARP-announced on the flat LAN by the existing
|
|
# CiliumL2AnnouncementPolicy `default-l2` — the SAME L2/LB-IPAM
|
|
# machinery the Gateway ingress VIP already uses (platform-config/cilium/*). No new Cilium config
|
|
# is required; these Services just consume the pool.
|
|
#
|
|
# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so TCP and UDP are
|
|
# split into two Services that SHARE ONE IP via Cilium's `lbipam.cilium.io/sharing-key`. The IP is
|
|
# PINNED (lbipam.cilium.io/ips) so devices have a STABLE inform/STUN target that never moves.
|
|
# (Alternative: one mixed-protocol Service — needs the MixedProtocolLBService feature, GA on
|
|
# k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
|
|
#
|
|
# selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
|
|
# into the guest.
|
|
#
|
|
# -------------------------------------------------------------------------------------------
|
|
# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
|
|
# -------------------------------------------------------------------------------------------
|
|
# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
|
|
# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
|
|
# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
|
|
#
|
|
# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
|
|
# per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
|
|
# matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
|
|
# MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
|
|
# `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
|
|
# announcing node that holds no backend has nowhere to send the packet and answers ICMP port
|
|
# unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
|
|
# (the leases were 8 days old); the operator's controller UI and device inform path were down.
|
|
# It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
|
|
# deleting the Lease forces a re-election — and that re-election is random, so it can pick a
|
|
# backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
|
|
# reference implementation this file now follows.
|
|
#
|
|
# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
|
|
# interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
|
|
# SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
|
|
# controller has always seen the pod-side masquerade gateway as the source, never the LAN
|
|
# client. There is a SECOND translation below the Service, and it already destroyed the very
|
|
# thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
|
|
# it is giving up an address the guest could not see in the first place.
|
|
#
|
|
# Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
|
|
# MAC (and the device reports its own IP inside the inform payload), not the packet source,
|
|
# and guest-portal authorization runs on the gateway appliance, not on this controller.
|
|
#
|
|
# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
|
|
# is what makes this survive a reschedule and a rebuild.
|
|
#
|
|
# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
|
|
# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
|
|
# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
|
|
# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
|
|
# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
|
|
# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
|
|
# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
|
|
# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
|
|
# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
|
|
#
|
|
# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
|
|
# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
|
|
# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
|
|
# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
|
|
# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
|
|
# fix would be a NEW restriction that could silently break a remote site's device informing over
|
|
# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
|
|
# change, made deliberately and tested against the device fleet.
|
|
#
|
|
# VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
|
|
# default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
|
|
# Substituted by the `apps-vms` Flux Kustomization, which carries postBuild.substituteFrom for
|
|
# exactly this token; every shell $${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...}
|
|
# accordingly. A new environment MUST set UNIFI_L4_VIP to a free address in ITS pool.
|
|
#
|
|
# REUSABLE PATTERN (the deferred `minecraft` :25565/L4 problem is the same shape): expose an
|
|
# L4-only workload as a Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing
|
|
# key; publish its A record from the Service via external-dns. minecraft = one TCP port; unifi is
|
|
# the multi-port TCP+UDP generalization.
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: unifi-l4-tcp
|
|
namespace: unifi
|
|
labels:
|
|
app.kubernetes.io/name: unifi
|
|
annotations:
|
|
lbipam.cilium.io/sharing-key: "unifi"
|
|
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
|
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
|
|
# This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
|
|
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
|
|
spec:
|
|
type: LoadBalancer
|
|
externalTrafficPolicy: Cluster
|
|
selector:
|
|
kubevirt.io/vm: unifi
|
|
ports:
|
|
- { name: inform, port: 8080, targetPort: 8080, protocol: TCP }
|
|
- { name: ui, port: 11443, targetPort: 11443, protocol: TCP }
|
|
- { name: ui-legacy, port: 8443, targetPort: 8443, protocol: TCP }
|
|
- { name: portal-http, port: 8880, targetPort: 8880, protocol: TCP }
|
|
- { name: portal-https, port: 8843, targetPort: 8843, protocol: TCP }
|
|
- { name: throughput, port: 6789, targetPort: 6789, protocol: TCP }
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: unifi-l4-udp
|
|
namespace: unifi
|
|
labels:
|
|
app.kubernetes.io/name: unifi
|
|
annotations:
|
|
lbipam.cilium.io/sharing-key: "unifi" # same key + same IP => shares the VIP with -tcp
|
|
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
|
spec:
|
|
type: LoadBalancer
|
|
externalTrafficPolicy: Cluster
|
|
selector:
|
|
kubevirt.io/vm: unifi
|
|
ports:
|
|
- { name: stun, port: 3478, targetPort: 3478, protocol: UDP }
|
|
- { name: discovery, port: 10001, targetPort: 10001, protocol: UDP }
|