fix(vpn-ingress): the VPN ingress node must announce no L2 VIPs, or clients route nothing

A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
holds itself: the announcer does not answer its own ARP request, so on the
lease holder the neighbour entry settles at FAILED and packets to that VIP die
before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes
x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced.

platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and
MASQUERADEs remote clients to the node's own address, so a VPN client takes
the node's LAN path and inherits its ARP failures. The router's udp/51820
forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16
Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering
(its VIP has two Services and k3s-prod-2 held the other lease). The tunnel
handshook, names resolved, every app hung, and nothing reported it.

  on k3s-prod-1:  ip neigh show 192.168.2.20 -> FAILED      (its own VIP)
  on prod-2/3:    ip neigh show 192.168.2.20 -> lladdr ...  (prod-1), 443 OK

Mark the node and exclude it from announcing:

* node_baseline labels the forwarded node (node_baseline_vpn_ingress_node,
  k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes
  the label from any node that no longer matches so moving it MOVES the label.
* l2-announcement-policy.yaml excludes that label, so the node ARP-resolves
  every VIP. Fixes all VIPs for VPN clients, not just the one that collided.
  Cost: announcement candidacy drops from 3 nodes to 2.
* VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels
  (label added to the kube-state-metrics allowlist), with
  VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches,
  so a missing label would silently re-arm the whole failure AND leave the
  critical alert inert. Four promtool cases, including that inertness.

A node LABEL rather than a cluster-vars knob because the alert must read it
too, and extract_rules.py rightly refuses a ${...} placeholder in an alert
expr -- the rule unit-tested would not be the rule deployed. One source of
truth, and it touches no sensitive path.

Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and
"failover is this one line". Both false; the first caused this.

UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the
handoff entry's third section.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

platform-config/cilium/l2-announcement-policy.yaml

@@ -16,6 +16,71 @@ spec:
   externalIPs: false
   interfaces:
     - ^e(n|th)
+  # ===============================================================================================
+  # WHY THIS IS A matchExpressions LIST AND NOT `matchLabels: {kubernetes.io/os: linux}`
+  # ===============================================================================================
+  # The second expression excludes every node labelled `homelab.keithjackson.dev/vpn-ingress=true`
+  # from announcing ANY LoadBalancer VIP. That is the node the SOHO router forwards WAN udp/51820 to.
+  # It is not tuning; it is the fix for a measured, silent, total failure of remote access.
+  #
+  # MEASURED ON PROD 2026-10-08: A NODE CANNOT REACH A VIP IT ANNOUNCES ITSELF. Cilium's L2
+  # announcer answers ARP for the VIP on behalf of OTHER LAN hosts, but not for the announcing node
+  # itself. On the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP
+  # is dropped before it leaves the node:
+  #
+  #   k3s-prod-1, holding cilium-l2announce-gateway-cilium-gateway-shared:
+  #     ip neigh show 192.168.2.20  ->  192.168.2.20 dev eno1 FAILED       (its own VIP)
+  #     nc -z 192.168.2.20 443      ->  FAIL          nc -z 192.168.2.20 80 -> FAIL
+  #   k3s-prod-2 and k3s-prod-3, holding no lease for it:
+  #     ip neigh show 192.168.2.20  ->  lladdr bc:e9:2f:7e:c4:a6 (prod-1)   nc 443/80 -> OK
+  #
+  # Symmetric across all 3 nodes x 5 VIPs: every node reached every VIP EXCEPT the ones it alone
+  # announced (prod-2 could not reach crafty 192.168.2.22 or unifi 192.168.2.21; prod-3 could not
+  # reach plex 192.168.2.24). So this is a property of L2 announcement, not of one Service.
+  #
+  # WHY IT ONLY HURTS THE VPN NODE. For an ordinary workload this is harmless: pods reach Services
+  # through the cluster datapath and never ARP for a VIP. platform/vpn-ingress is the exception --
+  # it runs WireGuard with `hostNetwork: true` and MASQUERADEs remote clients to the node's own
+  # address, so a VPN client's packets take the node's LAN path and inherit exactly this ARP
+  # failure. On 2026-10-08 the forward pointed at k3s-prod-1, which held the Gateway lease, and all
+  # 16 Gateway hostnames were unreachable over the VPN while PiHole DNS still answered -- so names
+  # resolved and every app hung. Nothing in the cluster reported it; the alerts below now do.
+  #
+  # WHY A NODE LABEL AND NOT A cluster-vars KNOB. A knob would put the node's identity in git, which
+  # is more reviewable -- but it CANNOT BE USED BY THE ALERT that guards this, and an unguarded
+  # invariant here is worth less than a reviewable one. The alert harness
+  # (platform/monitoring/tests/extract_rules.py) REFUSES any alert whose `expr` contains a
+  # dollar-brace postBuild placeholder (spelled without braces here ON PURPOSE: this file is itself
+  # envsubst'd, and a literal dollar-brace-dot-dot-dot is UNPARSEABLE to envsubst, which fails the
+  # WHOLE Kustomization -- ci: flux-postbuild-substitution caught exactly this comment), because the
+  # rule it would unit-test is not the rule that gets
+  # deployed. A label is readable from BOTH sides: this selector matches on it, and
+  # `kube_node_labels` carries it into Prometheus so `VpnIngressNodeAnnouncesL2VIP` can join lease
+  # holders against it. One source of truth, no git-vs-cluster skew, and the policy stays
+  # env-agnostic as its header above claims. The label is set by
+  # infra/ansible/roles/node_baseline (node_baseline_vpn_ingress_node), the same shape
+  # roles/vpn_egress_gateway already uses for `egress-gateway`.
+  #
+  # THE COST, STATED: VIP announcement candidacy drops from 3 nodes to 2. That is still redundant,
+  # and the trade is deliberate -- the excluded node is the one whose *outbound* reach to every VIP
+  # matters most, because it carries every remote client. If BOTH remaining nodes are down the VIPs
+  # go dark where previously one node might have served them, but at that point the cluster has one
+  # node left and far worse problems.
+  #
+  # `NotIn` with an ABSENT key MATCHES (k8s apimachinery: a missing label is "not in the set"), so
+  # the other two nodes need no label and nothing has to be applied to them. If the label is never
+  # applied anywhere, this excludes nothing and the policy degrades to its pre-2026-10-08 behaviour
+  # -- the status quo, not an outage: it fails back, not down. That silent-degradation path is
+  # exactly why `VpnIngressNodeLabelMissing` exists as a watchdog: without it, the alert below
+  # would be structurally INERT (its join would match nothing) and would look identical to healthy.
   nodeSelector:
-    matchLabels:
-      kubernetes.io/os: linux
+    matchExpressions:
+      - key: kubernetes.io/os
+        operator: In
+        values:
+          - linux
+      # The node terminating remote-access WireGuard must announce nothing -- see above.
+      - key: homelab.keithjackson.dev/vpn-ingress
+        operator: NotIn
+        values:
+          - "true"