# vpn-ingress-l2-collision — the VPN ingress node is announcing a LoadBalancer VIP **Alerts:** `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) and `VpnIngressNodeLabelMissing` (warning, 30m) **File:** `platform/monitoring/alerts.yaml` **Related:** `/runbooks/vpn-remote-access` is the setup and enrolment runbook for this component. `/runbooks/vpn-egress-stale` and `/runbooks/vpn-down` are the **other** VPN — the outbound PIA tunnel — and have nothing to do with this. ## What this means here **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The announcer answers ARP for the VIP on behalf of other LAN hosts, but not for the announcing node's own request. On the lease holder the neighbour entry settles at `FAILED`, and every packet to that VIP is dropped before it leaves the node. Measured on prod 2026-10-08, symmetric across all three nodes and five VIPs — each node reached every VIP **except** the ones it alone announced. For an ordinary workload this is harmless: pods reach Services through the cluster datapath and never ARP for a VIP. `platform/vpn-ingress` is the exception. It runs WireGuard with `hostNetwork: true` and MASQUERADEs remote clients to the node's own address, so a remote client's packets take that node's LAN path and inherit its ARP failures. **If the node the router forwards `udp/51820` to is also an L2 lease holder, every remote VPN client silently loses all access to the Services behind the VIPs it announces.** The node terminating WireGuard is marked with `homelab.keithjackson.dev/vpn-ingress=true`, and `platform-config/cilium/l2-announcement-policy.yaml` excludes that label from announcing anything, so this should be unreachable. **The symptom your users report will not sound like a network fault.** On 2026-10-08 the forward sat on `k3s-prod-1`, which held the `shared` Gateway lease, so all 16 Gateway hostnames were dead over the VPN — while PiHole DNS kept answering, because `192.168.2.23` is announced by *two* Services and `k3s-prod-2` held the other lease. So the tunnel handshakes, `wg show` looks perfect, names resolve, and **every app hangs on connect**. Expect "the VPN connects but nothing loads", not "the VPN is down". Which Services are affected depends on which lease moved — read `{{ $labels.lease }}` from the alert. `cilium-l2announce-gateway-cilium-gateway-shared` is the worst case: it is every web app at once. ### If it is `VpnIngressNodeLabelMissing` that fired That is the watchdog, and it means the protection is **switched off**, not that something is merely untidy. The exclusion is a `NotIn` on the label, and `NotIn` with an absent key **matches** — so with the label on no node, the policy excludes nothing and every node is an announcement candidate again. `VpnIngressNodeAnnouncesL2VIP` is simultaneously **inert**: its join matches nothing and returns no series, which is indistinguishable from healthy. Treat a quiet critical alert as meaningless until this warning is cleared. Usual cause: a node was rebuilt and `node_baseline` has not been re-run on it. ## First commands Run cluster commands through the pinned toolbox, which supplies the kube context: ```bash # 1. WHO HOLDS WHAT, and WHO IS THE VPN NODE — the whole question, in two lines. ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n kube-system get lease | grep l2announce' ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/vpn-ingress' # Exactly ONE node should carry vpn-ingress=true, and NO l2announce lease should name it. # 2. IS THE EXCLUSION STILL IN THE LIVE POLICY? Expect a matchExpressions list with a # homelab.keithjackson.dev/vpn-ingress NotIn [true] entry — not a bare matchLabels. ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get ciliuml2announcementpolicy default-l2 -o jsonpath="{.spec.nodeSelector}"; echo' # 3. IS THE LABEL REACHING PROMETHEUS? If this is empty the alerts are inert even when # the label is on the node (kube-state-metrics metricLabelsAllowlist). ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n monitoring exec deploy/kube-prometheus-stack-kube-state-metrics -- \ wget -qO- localhost:8080/metrics | grep vpn_ingress | head' ``` **Confirm the actual damage**, which is the one measurement that is unambiguous. From the WireGuard pod on the affected node (it shares the host netns, so this is the path a VPN client takes): ```bash ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress get pods -o wide' # pick the pod on that node ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress exec -c wireguard -- \ sh -c "ping -c1 -W1 192.168.2.20 >/dev/null 2>&1; ip neigh show 192.168.2.20"' # BROKEN: 192.168.2.20 dev eno1 FAILED # HEALTHY: 192.168.2.20 dev eno1 lladdr bc:e9:2f:7e:c4:a6 STALE|REACHABLE|DELAY ``` The `ping` is only there to trigger ARP — the VIP does not answer ICMP even when healthy, so **ignore its 100% packet loss**; the neighbour state is the answer. Substitute the VIP that belongs to `{{ $labels.lease }}` if the Gateway is not the one that moved. **Which node actually holds the forward** (the router cannot be queried, but the cluster knows): only the forwarded node's `wg show` has a peer with an `endpoint` and a recent `latest handshake`. ```bash ./infra/devcontainer/dev.sh run bash -c \ 'for p in $(kubectl --context k3s-prod -n vpn-ingress get pods -o name); do echo "== $p"; kubectl --context k3s-prod -n vpn-ingress exec ${p#pod/} -c wireguard -- wg show; done' ``` ## Likely causes, most likely first 1. **The label is on the wrong node** — someone re-pointed the router's forward during a node failover and did not move `node_baseline_vpn_ingress_node`. This is the most likely cause by a wide margin, because the router is off-repo and nothing can check it. Confirm with the `wg show` loop above: the node with the live peer endpoint is the real forward target. *Fix:* merge a PR setting `node_baseline_vpn_ingress_node` (`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`) to that node, then re-run the role — `ansible-playbook -i inventory/hosts.ini -e k3s_hosts=k3s_prod --tags vpn-ingress`. The role labels the new node and **removes** the label from the old one. Then confirm Cilium re-elected the lease off the newly-labelled node. 2. **The exclusion was edited out of `platform-config/cilium/l2-announcement-policy.yaml`**, or the file was reverted to a bare `matchLabels: {kubernetes.io/os: linux}`. Command 2 above shows this directly. *Fix:* restore the `matchExpressions` form; the file's own header explains why it is shaped that way. 3. **The label is missing entirely** (`VpnIngressNodeLabelMissing` firing) — a node rebuild without a `node_baseline` re-run. *Fix:* re-run the role as above. 4. **The label is on the node but not in Prometheus** — `metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` lost the `vpn-ingress` entry. Then the critical alert is inert *and* the watchdog fires. Command 3 above distinguishes this from cause 3. 5. **Cilium kept renewing a lease its holder is no longer eligible for.** The exclusion is correct and in place, but the lease never re-elected. This was never verified when the fix was authored (see `decisions/2026-10-17-vpn-ingress-l2-lease-collision.md`). *Fix:* delete that one Lease object to force an election — `kubectl -n kube-system delete lease ` — and watch it come back on a different node. 6. **A Cilium upgrade changed `NotIn` semantics or the `nodeSelector` field.** Least likely, but it is the reason this is a runtime alert rather than a static check on the policy text. ## When to escalate or stop **Act now, don't wait for a window.** Remote access is fully broken for the affected VIPs while this is firing, and it does **not** self-heal: a lease holder keeps renewing, so this condition persists indefinitely once entered. That is also why the 15m `for:` is safe — a transient move during a node reboot or a Cilium roll resolves on its own well inside it. **The fastest honest mitigation is to move the forward, not the leases.** Re-pointing the router's `udp/51820` at a node that holds no lease for the affected VIP restores service in seconds, with no cluster change and no new client profiles — all three nodes answer the same keys. Follow it with the `node_baseline_vpn_ingress_node` PR and role re-run so the two agree again; until you do, this alert is watching the wrong node. Note each node's own announcements when choosing: as of 2026-10-08, prod-2 announced crafty `192.168.2.22` and unifi `192.168.2.21`, prod-3 announced plex `192.168.2.24`, so neither is a cost-free target. **Do not** `kubectl edit` the `CiliumL2AnnouncementPolicy` to fix this — Flux owns it and will revert you, and the drift is the kind that outlives the memory of making it. The two acceptable hand actions are deleting a single Lease to force re-election (cause 5), and `kubectl label node` as a *temporary* stopgap while the PR lands — note that a node rebuild or a `node_baseline` run will undo a hand-applied label, so it is not a fix. **Stop and escalate to the operator if** the leases will not leave the excluded node after a forced re-election, or if the intended forward target is unclear. Both need a decision about which node carries remote access, which is an operator call, not a runbook step. **Known gap, so you are not misled:** if someone re-points the router without moving the label, `VpnIngressNodeAnnouncesL2VIP` watches the wrong node and **stays silent** while the VPN breaks in exactly the way described above. There is no in-cluster signal for that today; closing it needs a WireGuard exporter publishing which node is actually terminating peer traffic (`platform/vpn-ingress/README.md`, "Not built") — the `wg show` loop above is the manual substitute. Until then, if remote access is broken and both alerts are quiet, **check the forward against the label by hand before believing the alerts.**