fix(vpn-ingress): the VPN ingress node must announce no L2 VIPs, or clients route nothing

A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
holds itself: the announcer does not answer its own ARP request, so on the
lease holder the neighbour entry settles at FAILED and packets to that VIP die
before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes
x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced.

platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and
MASQUERADEs remote clients to the node's own address, so a VPN client takes
the node's LAN path and inherits its ARP failures. The router's udp/51820
forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16
Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering
(its VIP has two Services and k3s-prod-2 held the other lease). The tunnel
handshook, names resolved, every app hung, and nothing reported it.

  on k3s-prod-1:  ip neigh show 192.168.2.20 -> FAILED      (its own VIP)
  on prod-2/3:    ip neigh show 192.168.2.20 -> lladdr ...  (prod-1), 443 OK

Mark the node and exclude it from announcing:

* node_baseline labels the forwarded node (node_baseline_vpn_ingress_node,
  k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes
  the label from any node that no longer matches so moving it MOVES the label.
* l2-announcement-policy.yaml excludes that label, so the node ARP-resolves
  every VIP. Fixes all VIPs for VPN clients, not just the one that collided.
  Cost: announcement candidacy drops from 3 nodes to 2.
* VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels
  (label added to the kube-state-metrics allowlist), with
  VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches,
  so a missing label would silently re-arm the whole failure AND leave the
  critical alert inert. Four promtool cases, including that inertness.

A node LABEL rather than a cluster-vars knob because the alert must read it
too, and extract_rules.py rightly refuses a ${...} placeholder in an alert
expr -- the rule unit-tested would not be the rule deployed. One source of
truth, and it touches no sensitive path.

Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and
"failover is this one line". Both false; the first caused this.

UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the
handoff entry's third section.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

runbooks/vpn-remote-access.md

@@ -171,14 +171,57 @@ server without a `wg0` "ready".
 On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
 `192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.
 
-Pick any node — all three run an identical listener. Note which one you chose here:
+> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node.
+>
+> This step used to say "pick any node — all three run an identical listener". **That sentence
+> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a
+> cluster-vars knob instead of a free choice here.
+>
+> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
+> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease
+> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they
+> leave the node. Measured on prod 2026-10-08:
+>
+> ```text
+> on k3s-prod-1 (held the shared Gateway lease):   ip neigh show 192.168.2.20 -> FAILED
+> on k3s-prod-2 / k3s-prod-3:                      ip neigh show 192.168.2.20 -> lladdr … (prod-1)
+> ```
+>
+> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN
+> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway
+> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so
+> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported
+> it.
+>
+`infra/ansible/roles/node_baseline` now labels the forwarded node
+> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in
+> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes
+> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI
+> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a
+> labelled node taking a lease, but it CANNOT catch the forward being moved without the label.
+
+Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`).
+Note it here:
 
 ```
 # WAN udp/51820 -> 192.168.2.__  (k3s-prod-_)   set <date>
+# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml
 ```
 
-**Node loss = failover is this one line.** Re-point the forward at another node address; every
-device keeps working with the same profile, because all three nodes answer the same keys.
+**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that
+was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and
+reintroduces the failure above against a different set of addresses (prod-2 announces crafty and
+unifi; prod-3 announces plex). Do both, in this order:
+
+1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role
+   (`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium
+   then moves the leases off the newly-labelled node and onto its peers.
+2. **Re-point the router** forward at the new node's address.
+
+Every device keeps working with the same profile either way; all three nodes answer the same keys,
+and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that
+handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because
+that alert watches the node the knob names, not the node your router actually points at.
 
 Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
 already silent to unauthenticated probes (README, "Why this is safe to expose").