--- # CiliumL2AnnouncementPolicy — ARP-announce LoadBalancer VIPs on the node LAN so LAN hosts (dev # 192.168.1.0/24, prod 192.168.2.0/24) can reach Gateway listeners at the pool IP. The RANGE itself # is env-scoped in l2-ip-pool.yaml via CILIUM_LB_POOL_START/STOP; this policy is env-agnostic and # needs no cluster-vars knob (l2announcements.enabled=true in the # Cilium HelmRelease turns on the controller; this policy scopes WHAT/WHERE it announces). # Announces on the primary ethernet interface. On Proxmox VMs the NIC is typically ens18/eth0; # `^e(n|th)` matches ens*/enp*/eth* so the exact name need not be pinned. VERIFY-AT-BRINGUP: if # announcement fails, confirm the node's primary NIC name (`ip -o link`) matches this selector. apiVersion: cilium.io/v2alpha1 kind: CiliumL2AnnouncementPolicy metadata: name: default-l2 spec: loadBalancerIPs: true # announce LoadBalancer Service VIPs (Gateway listeners) externalIPs: false interfaces: - ^e(n|th) # =============================================================================================== # WHY THIS IS A matchExpressions LIST AND NOT `matchLabels: {kubernetes.io/os: linux}` # =============================================================================================== # The second expression excludes every node labelled `homelab.keithjackson.dev/vpn-ingress=true` # from announcing ANY LoadBalancer VIP. That is the node the SOHO router forwards WAN udp/51820 to. # It is not tuning; it is the fix for a measured, silent, total failure of remote access. # # MEASURED ON PROD 2026-10-08: A NODE CANNOT REACH A VIP IT ANNOUNCES ITSELF. Cilium's L2 # announcer answers ARP for the VIP on behalf of OTHER LAN hosts, but not for the announcing node # itself. On the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP # is dropped before it leaves the node: # # k3s-prod-1, holding cilium-l2announce-gateway-cilium-gateway-shared: # ip neigh show 192.168.2.20 -> 192.168.2.20 dev eno1 FAILED (its own VIP) # nc -z 192.168.2.20 443 -> FAIL nc -z 192.168.2.20 80 -> FAIL # k3s-prod-2 and k3s-prod-3, holding no lease for it: # ip neigh show 192.168.2.20 -> lladdr bc:e9:2f:7e:c4:a6 (prod-1) nc 443/80 -> OK # # Symmetric across all 3 nodes x 5 VIPs: every node reached every VIP EXCEPT the ones it alone # announced (prod-2 could not reach crafty 192.168.2.22 or unifi 192.168.2.21; prod-3 could not # reach plex 192.168.2.24). So this is a property of L2 announcement, not of one Service. # # WHY IT ONLY HURTS THE VPN NODE. For an ordinary workload this is harmless: pods reach Services # through the cluster datapath and never ARP for a VIP. platform/vpn-ingress is the exception -- # it runs WireGuard with `hostNetwork: true` and MASQUERADEs remote clients to the node's own # address, so a VPN client's packets take the node's LAN path and inherit exactly this ARP # failure. On 2026-10-08 the forward pointed at k3s-prod-1, which held the Gateway lease, and all # 16 Gateway hostnames were unreachable over the VPN while PiHole DNS still answered -- so names # resolved and every app hung. Nothing in the cluster reported it; the alerts below now do. # # WHY A NODE LABEL AND NOT A cluster-vars KNOB. A knob would put the node's identity in git, which # is more reviewable -- but it CANNOT BE USED BY THE ALERT that guards this, and an unguarded # invariant here is worth less than a reviewable one. The alert harness # (platform/monitoring/tests/extract_rules.py) REFUSES any alert whose `expr` contains a # dollar-brace postBuild placeholder (spelled without braces here ON PURPOSE: this file is itself # envsubst'd, and a literal dollar-brace-dot-dot-dot is UNPARSEABLE to envsubst, which fails the # WHOLE Kustomization -- ci: flux-postbuild-substitution caught exactly this comment), because the # rule it would unit-test is not the rule that gets # deployed. A label is readable from BOTH sides: this selector matches on it, and # `kube_node_labels` carries it into Prometheus so `VpnIngressNodeAnnouncesL2VIP` can join lease # holders against it. One source of truth, no git-vs-cluster skew, and the policy stays # env-agnostic as its header above claims. The label is set by # infra/ansible/roles/node_baseline (node_baseline_vpn_ingress_node), the same shape # roles/vpn_egress_gateway already uses for `egress-gateway`. # # THE COST, STATED: VIP announcement candidacy drops from 3 nodes to 2. That is still redundant, # and the trade is deliberate -- the excluded node is the one whose *outbound* reach to every VIP # matters most, because it carries every remote client. If BOTH remaining nodes are down the VIPs # go dark where previously one node might have served them, but at that point the cluster has one # node left and far worse problems. # # `NotIn` with an ABSENT key MATCHES (k8s apimachinery: a missing label is "not in the set"), so # the other two nodes need no label and nothing has to be applied to them. If the label is never # applied anywhere, this excludes nothing and the policy degrades to its pre-2026-10-08 behaviour # -- the status quo, not an outage: it fails back, not down. That silent-degradation path is # exactly why `VpnIngressNodeLabelMissing` exists as a watchdog: without it, the alert below # would be structurally INERT (its join would match nothing) and would look identical to healthy. nodeSelector: matchExpressions: - key: kubernetes.io/os operator: In values: - linux # The node terminating remote-access WireGuard must announce nothing -- see above. - key: homelab.keithjackson.dev/vpn-ingress operator: NotIn values: - "true"