fix(vpn-ingress): the VPN ingress node must announce no L2 VIPs, or clients route nothing

A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
holds itself: the announcer does not answer its own ARP request, so on the
lease holder the neighbour entry settles at FAILED and packets to that VIP die
before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes
x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced.

platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and
MASQUERADEs remote clients to the node's own address, so a VPN client takes
the node's LAN path and inherits its ARP failures. The router's udp/51820
forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16
Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering
(its VIP has two Services and k3s-prod-2 held the other lease). The tunnel
handshook, names resolved, every app hung, and nothing reported it.

  on k3s-prod-1:  ip neigh show 192.168.2.20 -> FAILED      (its own VIP)
  on prod-2/3:    ip neigh show 192.168.2.20 -> lladdr ...  (prod-1), 443 OK

Mark the node and exclude it from announcing:

* node_baseline labels the forwarded node (node_baseline_vpn_ingress_node,
  k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes
  the label from any node that no longer matches so moving it MOVES the label.
* l2-announcement-policy.yaml excludes that label, so the node ARP-resolves
  every VIP. Fixes all VIPs for VPN clients, not just the one that collided.
  Cost: announcement candidacy drops from 3 nodes to 2.
* VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels
  (label added to the kube-state-metrics allowlist), with
  VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches,
  so a missing label would silently re-arm the whole failure AND leave the
  critical alert inert. Four promtool cases, including that inertness.

A node LABEL rather than a cluster-vars knob because the alert must read it
too, and extract_rules.py rightly refuses a ${...} placeholder in an alert
expr -- the rule unit-tested would not be the rule deployed. One source of
truth, and it touches no sensitive path.

Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and
"failover is this one line". Both false; the first caused this.

UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the
handoff entry's third section.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -420,3 +420,22 @@ flux_bootstrap_env_path: "clusters/prod"
 # rather than leaving an inert drop-in behind, so this group-wide `true` is still correct
 # for it.
 etcd_io_priority_enabled: true
+
+# --- The node terminating remote-access WireGuard — DECIDED (measured 2026-10-08) ---------------
+# k3s-prod-1 (192.168.2.10) is what the SOHO router forwards WAN udp/51820 to, confirmed from the
+# cluster rather than assumed: only prod-1's `wg show` had a peer with an `endpoint` and a recent
+# handshake, the other two listeners were idle.
+#
+# node_baseline labels it `homelab.keithjackson.dev/vpn-ingress=true`, and
+# platform-config/cilium/l2-announcement-policy.yaml excludes that label from announcing any
+# LoadBalancer VIP -- because a node CANNOT reach a VIP whose Cilium L2 announcement lease it holds
+# itself (the announcer does not answer its own ARP request; the neighbour entry goes to `FAILED`).
+# prod-1 held the `shared` Gateway lease, and since vpn-ingress MASQUERADEs clients to the node's
+# own address, all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved.
+#
+# [parity] MUST equal the node named by the router's forward (runbooks/vpn-remote-access.md step 3).
+# Nothing in CI can read that forward, so re-pointing it means changing BOTH: this value plus a
+# node_baseline re-run, and the router line. `VpnIngressNodeAnnouncesL2VIP` catches the case where
+# the label lands on a node that then takes a lease; it CANNOT catch the forward being moved
+# without this value following, which is why the runbook spells the two-step out.
+node_baseline_vpn_ingress_node: k3s-prod-1

infra/ansible/roles/node_baseline/defaults/main.yml

@@ -259,3 +259,30 @@ node_baseline_dns_options:
 # Written as ONE marked block, so removing an entry here removes it from the node, and
 # an empty list removes the block entirely.
 node_baseline_hosts_pins: []
+
+# --- The node that terminates remote-access WireGuard (tags: vpn-ingress) ----------------------
+# Hostname of the node the SOHO router forwards WAN udp/51820 to, i.e. the one platform/vpn-ingress
+# actually serves remote clients from. Set it in group_vars for the environment; EMPTY by default,
+# because dev has no such forward and because a wrong value is worse than none.
+#
+# WHAT IT IS FOR, AND WHY IT IS NOT COSMETIC. The matching node gets
+# `homelab.keithjackson.dev/vpn-ingress=true`, and
+# platform-config/cilium/l2-announcement-policy.yaml EXCLUDES that label from announcing any
+# LoadBalancer VIP. MEASURED on prod 2026-10-08: a node cannot reach a VIP whose Cilium L2
+# announcement lease it holds itself (the announcer does not answer its own ARP request -- the
+# neighbour entry goes to `FAILED`). Because vpn-ingress runs on `hostNetwork` and MASQUERADEs
+# clients to the node's own address, a VPN client inherits that: with the forward on the node
+# holding the `shared` Gateway lease, all 16 Gateway hostnames were dead over the VPN while PiHole
+# DNS still resolved. Names worked, every app hung, nothing alerted.
+#
+# [parity] MUST name the node the router's forward actually points at. The forward is off-repo and
+# no check can read it (runbooks/vpn-remote-access.md step 3). Changing the forward is therefore a
+# two-part change: this value (then re-run the role), and the router line.
+#
+# Only ONE node should ever carry the label: it removes that node from VIP announcement candidacy,
+# so labelling all three would leave nothing announcing any VIP at all. The task below removes the
+# label from a node that no longer matches, so moving the value moves the label.
+node_baseline_vpn_ingress_node: ""
+node_baseline_vpn_ingress_label_key: homelab.keithjackson.dev/vpn-ingress
+# k3s ships its kubeconfig here; every prod node is a k3s server, so `k3s kubectl` works locally.
+node_baseline_vpn_ingress_kubeconfig: /etc/rancher/k3s/k3s.yaml

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -1109,3 +1109,45 @@
     state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
     backup: true
   tags: [dns, hosts]
+
+# ── vpn-ingress node label — tag `vpn-ingress` ────────────────────────────────────────────────
+# Marks the node terminating remote-access WireGuard so Cilium's L2 announcement policy can EXCLUDE
+# it from announcing LoadBalancer VIPs (a node cannot reach a VIP it announces itself -- see
+# defaults/main.yml). Same shape as roles/vpn_egress_gateway's egress-gateway label, and for the
+# same reason: the label is the one place BOTH a nodeSelector and a PromQL alert can read it from.
+#
+# Skipped entirely when node_baseline_vpn_ingress_node is empty (dev, or before the forward exists),
+# so this adds nothing to an environment that has no WAN forward.
+- name: Label this node as the vpn-ingress node
+  ansible.builtin.command:
+    cmd: >-
+      k3s kubectl label node {{ inventory_hostname }}
+      {{ node_baseline_vpn_ingress_label_key }}=true --overwrite
+  environment:
+    KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}"
+  register: node_baseline_vpn_ingress_label_apply
+  changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_apply.stdout"
+  when:
+    - node_baseline_vpn_ingress_node | length > 0
+    - inventory_hostname == node_baseline_vpn_ingress_node
+  tags: [vpn-ingress]
+
+# The other half: moving the knob must MOVE the label, not add a second one. Two labelled nodes
+# would remove two of three nodes from announcement candidacy; three would leave no announcer at
+# all and take every VIP off the LAN.
+- name: Remove the vpn-ingress label from a node that no longer holds the forward
+  ansible.builtin.command:
+    cmd: >-
+      k3s kubectl label node {{ inventory_hostname }}
+      {{ node_baseline_vpn_ingress_label_key }}-
+  environment:
+    KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}"
+  register: node_baseline_vpn_ingress_label_remove
+  changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_remove.stdout"
+  failed_when:
+    - node_baseline_vpn_ingress_label_remove.rc != 0
+    - "'not found' not in node_baseline_vpn_ingress_label_remove.stderr"
+  when:
+    - node_baseline_vpn_ingress_node | length > 0
+    - inventory_hostname != node_baseline_vpn_ingress_node
+  tags: [vpn-ingress]