home-network-k3s-impl/runbooks/vpn-ingress-l2-collision.md

vpn-ingress-l2-collision — the VPN ingress node is announcing a LoadBalancer VIP

Alerts: VpnIngressNodeAnnouncesL2VIP (critical, 15m) and VpnIngressNodeLabelMissing (warning, 30m) File: platform/monitoring/alerts.yaml Related: /runbooks/vpn-remote-access is the setup and enrolment runbook for this component. /runbooks/vpn-egress-stale and /runbooks/vpn-down are the other VPN — the outbound PIA tunnel — and have nothing to do with this.

What this means here

A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself. The announcer answers ARP for the VIP on behalf of other LAN hosts, but not for the announcing node's own request. On the lease holder the neighbour entry settles at FAILED, and every packet to that VIP is dropped before it leaves the node. Measured on prod 2026-10-08, symmetric across all three nodes and five VIPs — each node reached every VIP except the ones it alone announced.

For an ordinary workload this is harmless: pods reach Services through the cluster datapath and never ARP for a VIP. platform/vpn-ingress is the exception. It runs WireGuard with hostNetwork: true and MASQUERADEs remote clients to the node's own address, so a remote client's packets take that node's LAN path and inherit its ARP failures. If the node the router forwards udp/51820 to is also an L2 lease holder, every remote VPN client silently loses all access to the Services behind the VIPs it announces.

The node terminating WireGuard is marked with homelab.keithjackson.dev/vpn-ingress=true, and platform-config/cilium/l2-announcement-policy.yaml excludes that label from announcing anything, so this should be unreachable.

The symptom your users report will not sound like a network fault. On 2026-10-08 the forward sat on k3s-prod-1, which held the shared Gateway lease, so all 16 Gateway hostnames were dead over the VPN — while PiHole DNS kept answering, because 192.168.2.23 is announced by two Services and k3s-prod-2 held the other lease. So the tunnel handshakes, wg show looks perfect, names resolve, and every app hangs on connect. Expect "the VPN connects but nothing loads", not "the VPN is down".

Which Services are affected depends on which lease moved — read {{ $labels.lease }} from the alert. cilium-l2announce-gateway-cilium-gateway-shared is the worst case: it is every web app at once.

If it is VpnIngressNodeLabelMissing that fired

That is the watchdog, and it means the protection is switched off, not that something is merely untidy. The exclusion is a NotIn on the label, and NotIn with an absent key matches — so with the label on no node, the policy excludes nothing and every node is an announcement candidate again. VpnIngressNodeAnnouncesL2VIP is simultaneously inert: its join matches nothing and returns no series, which is indistinguishable from healthy. Treat a quiet critical alert as meaningless until this warning is cleared. Usual cause: a node was rebuilt and node_baseline has not been re-run on it.

First commands

Run cluster commands through the pinned toolbox, which supplies the kube context:

# 1. WHO HOLDS WHAT, and WHO IS THE VPN NODE — the whole question, in two lines.
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kube-system get lease | grep l2announce'
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/vpn-ingress'
# Exactly ONE node should carry vpn-ingress=true, and NO l2announce lease should name it.

# 2. IS THE EXCLUSION STILL IN THE LIVE POLICY? Expect a matchExpressions list with a
#    homelab.keithjackson.dev/vpn-ingress NotIn [true] entry — not a bare matchLabels.
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get ciliuml2announcementpolicy default-l2 -o jsonpath="{.spec.nodeSelector}"; echo'

# 3. IS THE LABEL REACHING PROMETHEUS? If this is empty the alerts are inert even when
#    the label is on the node (kube-state-metrics metricLabelsAllowlist).
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring exec deploy/kube-prometheus-stack-kube-state-metrics -- \
     wget -qO- localhost:8080/metrics | grep vpn_ingress | head'

Confirm the actual damage, which is the one measurement that is unambiguous. From the WireGuard pod on the affected node (it shares the host netns, so this is the path a VPN client takes):

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress get pods -o wide'   # pick the pod on that node

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- \
     sh -c "ping -c1 -W1 192.168.2.20 >/dev/null 2>&1; ip neigh show 192.168.2.20"'
# BROKEN:  192.168.2.20 dev eno1 FAILED
# HEALTHY: 192.168.2.20 dev eno1 lladdr bc:e9:2f:7e:c4:a6 STALE|REACHABLE|DELAY

The ping is only there to trigger ARP — the VIP does not answer ICMP even when healthy, so ignore its 100% packet loss; the neighbour state is the answer. Substitute the VIP that belongs to {{ $labels.lease }} if the Gateway is not the one that moved.

Which node actually holds the forward (the router cannot be queried, but the cluster knows): only the forwarded node's wg show has a peer with an endpoint and a recent latest handshake.

./infra/devcontainer/dev.sh run bash -c \
  'for p in $(kubectl --context k3s-prod -n vpn-ingress get pods -o name); do
     echo "== $p"; kubectl --context k3s-prod -n vpn-ingress exec ${p#pod/} -c wireguard -- wg show; done'

Likely causes, most likely first

1. The label is on the wrong node — someone re-pointed the router's forward during a node failover and did not move node_baseline_vpn_ingress_node. This is the most likely cause by a wide margin, because the router is off-repo and nothing can check it. Confirm with the wg show loop above: the node with the live peer endpoint is the real forward target. Fix: merge a PR setting node_baseline_vpn_ingress_node (infra/ansible/inventory/group_vars/k3s_prod/vars.yml) to that node, then re-run the role — ansible-playbook -i inventory/hosts.ini <playbook> -e k3s_hosts=k3s_prod --tags vpn-ingress. The role labels the new node and removes the label from the old one. Then confirm Cilium re-elected the lease off the newly-labelled node.

2. The exclusion was edited out of platform-config/cilium/l2-announcement-policy.yaml, or the file was reverted to a bare matchLabels: {kubernetes.io/os: linux}. Command 2 above shows this directly. Fix: restore the matchExpressions form; the file's own header explains why it is shaped that way.

3. The label is missing entirely (VpnIngressNodeLabelMissing firing) — a node rebuild without a node_baseline re-run. Fix: re-run the role as above.

4. The label is on the node but not in Prometheus — metricLabelsAllowlist in platform/monitoring/kube-prometheus-stack.yaml lost the vpn-ingress entry. Then the critical alert is inert and the watchdog fires. Command 3 above distinguishes this from cause 3.

5. Cilium kept renewing a lease its holder is no longer eligible for. The exclusion is correct and in place, but the lease never re-elected. This was never verified when the fix was authored (see decisions/2026-10-17-vpn-ingress-l2-lease-collision.md). Fix: delete that one Lease object to force an election — kubectl -n kube-system delete lease <name> — and watch it come back on a different node.

6. A Cilium upgrade changed NotIn semantics or the nodeSelector field. Least likely, but it is the reason this is a runtime alert rather than a static check on the policy text.

When to escalate or stop

Act now, don't wait for a window. Remote access is fully broken for the affected VIPs while this is firing, and it does not self-heal: a lease holder keeps renewing, so this condition persists indefinitely once entered. That is also why the 15m for: is safe — a transient move during a node reboot or a Cilium roll resolves on its own well inside it.

The fastest honest mitigation is to move the forward, not the leases. Re-pointing the router's udp/51820 at a node that holds no lease for the affected VIP restores service in seconds, with no cluster change and no new client profiles — all three nodes answer the same keys. Follow it with the node_baseline_vpn_ingress_node PR and role re-run so the two agree again; until you do, this alert is watching the wrong node. Note each node's own announcements when choosing: as of 2026-10-08, prod-2 announced crafty 192.168.2.22 and unifi 192.168.2.21, prod-3 announced plex 192.168.2.24, so neither is a cost-free target.

Do not kubectl edit the CiliumL2AnnouncementPolicy to fix this — Flux owns it and will revert you, and the drift is the kind that outlives the memory of making it. The two acceptable hand actions are deleting a single Lease to force re-election (cause 5), and kubectl label node as a temporary stopgap while the PR lands — note that a node rebuild or a node_baseline run will undo a hand-applied label, so it is not a fix.

Stop and escalate to the operator if the leases will not leave the excluded node after a forced re-election, or if the intended forward target is unclear. Both need a decision about which node carries remote access, which is an operator call, not a runbook step.

Known gap, so you are not misled: if someone re-points the router without moving the label, VpnIngressNodeAnnouncesL2VIP watches the wrong node and stays silent while the VPN breaks in exactly the way described above. There is no in-cluster signal for that today; closing it needs a WireGuard exporter publishing which node is actually terminating peer traffic (platform/vpn-ingress/README.md, "Not built") — the wg show loop above is the manual substitute. Until then, if remote access is broken and both alerts are quiet, check the forward against the label by hand before believing the alerts.