fix(vpn-ingress): the VPN ingress node must announce no L2 VIPs, or clients route nothing

A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
holds itself: the announcer does not answer its own ARP request, so on the
lease holder the neighbour entry settles at FAILED and packets to that VIP die
before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes
x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced.

platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and
MASQUERADEs remote clients to the node's own address, so a VPN client takes
the node's LAN path and inherits its ARP failures. The router's udp/51820
forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16
Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering
(its VIP has two Services and k3s-prod-2 held the other lease). The tunnel
handshook, names resolved, every app hung, and nothing reported it.

  on k3s-prod-1:  ip neigh show 192.168.2.20 -> FAILED      (its own VIP)
  on prod-2/3:    ip neigh show 192.168.2.20 -> lladdr ...  (prod-1), 443 OK

Mark the node and exclude it from announcing:

* node_baseline labels the forwarded node (node_baseline_vpn_ingress_node,
  k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes
  the label from any node that no longer matches so moving it MOVES the label.
* l2-announcement-policy.yaml excludes that label, so the node ARP-resolves
  every VIP. Fixes all VIPs for VPN clients, not just the one that collided.
  Cost: announcement candidacy drops from 3 nodes to 2.
* VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels
  (label added to the kube-state-metrics allowlist), with
  VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches,
  so a missing label would silently re-arm the whole failure AND leave the
  critical alert inert. Four promtool cases, including that inertness.

A node LABEL rather than a cluster-vars knob because the alert must read it
too, and extract_rules.py rightly refuses a ${...} placeholder in an alert
expr -- the rule unit-tested would not be the rule deployed. One source of
truth, and it touches no sensitive path.

Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and
"failover is this one line". Both false; the first caused this.

UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the
handoff entry's third section.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

decisions/2026-10-17-vpn-ingress-l2-lease-collision.md

@@ -0,0 +1,137 @@
+---
+date: 2026-10-17
+branch: fix/vpn-ingress-l2-lease-collision
+title: "A node cannot reach a VIP it announces — so the VPN ingress node must announce nothing"
+---
+
+## What changed
+
+The first device enrolled on the remote-access VPN handshook, resolved names, and could not open a
+single connection to any cluster web app. The tunnel was not at fault. The fault is a property of
+Cilium L2 announcement that nothing in this repo knew about:
+
+**A node cannot reach a LoadBalancer VIP whose `CiliumL2AnnouncementPolicy` lease it holds itself.**
+The announcer answers ARP for the VIP on behalf of other LAN hosts but not for its own request, so
+on the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP is dropped
+before it leaves the node.
+
+`platform/vpn-ingress` is uniquely exposed because it runs WireGuard with `hostNetwork: true` and
+MASQUERADEs remote clients to the node's own address — a VPN client's packets take the node's LAN
+path and inherit its ARP failures. Ordinary pods never do. The router's `udp/51820` forward pointed
+at `k3s-prod-1`, which held `cilium-l2announce-gateway-cilium-gateway-shared`, so **all 16 `shared`
+Gateway hostnames were dead over the VPN** — while PiHole DNS kept answering, because
+`192.168.2.23` is announced by *two* Services and `k3s-prod-2` held the other lease. Names resolved
+and every app hung on connect.
+
+Seven files:
+
+* **`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`** — new
+  `node_baseline_vpn_ingress_node`; the matching node gets
+  `homelab.keithjackson.dev/vpn-ingress=true`, and a second task **removes** the label from any node
+  that no longer matches, so moving the value moves the label rather than adding a second one. Same
+  shape as `roles/vpn_egress_gateway`'s `egress-gateway` label. Tagged `vpn-ingress`; skipped
+  entirely when the value is empty.
+* **`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`** — set to `k3s-prod-1`, with a
+  `[parity]` note that it must equal the router's forward.
+* **`platform-config/cilium/l2-announcement-policy.yaml`** — `nodeSelector` becomes a
+  `matchExpressions` list keeping `kubernetes.io/os: linux` and adding
+  `homelab.keithjackson.dev/vpn-ingress NotIn ["true"]`. The VPN node then holds no lease, so it
+  ARP-resolves every VIP. This fixes **all** VIPs for VPN clients (Gateway, crafty, plex, pihole,
+  unifi), not just the one that happened to collide.
+* **`platform/monitoring/kube-prometheus-stack.yaml`** — the label added to kube-state-metrics'
+  `metricLabelsAllowlist`, which is what lets the invariant be watched at all.
+* **`platform/monitoring/alerts.yaml`** — `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) joins
+  `kube_lease_owner` against `kube_node_labels` via `label_replace`, plus
+  `VpnIngressNodeLabelMissing` (warning, 30m) as its watchdog.
+* **`platform/monitoring/tests/vpn-ingress-l2.test.yaml`** — four promtool cases: fires on the real
+  collision, quiet in the fixed state, quiet across a transient 11-minute lease move (pinning the
+  `for: 15m`), and the label-absent case asserting the critical alert is inert *and* the watchdog
+  fires.
+* **`runbooks/vpn-ingress-l2-collision.md`** (new, registered in `runbooks/kustomization.yaml`),
+  **`runbooks/vpn-remote-access.md`**, **`platform/vpn-ingress/{daemonset.yaml,README.md}`** — step 3
+  of the setup runbook said *"pick any node — all three run an identical listener"* and *"node loss =
+  failover is this one line"*. Both were false and the first caused this. Failover is now documented
+  as two ordered changes.
+
+**Design note — why a node label and not a `cluster-vars` knob.** The first attempt added
+`WG_INGRESS_NODE` to `clusters/base/cluster-vars.yaml` and substituted it into both the policy and
+the alert. The alert half is **impossible**: `platform/monitoring/tests/extract_rules.py` rejects any
+alert whose `expr` contains a `${...}` placeholder, because the rule it would unit-test is not the
+rule deployed. Rather than duplicate the node name into the alert as a literal (two sources that can
+skew), the label is readable from both sides — the selector matches on it and `kube_node_labels`
+carries it into Prometheus. One source of truth, and as a side effect the change touches no
+sensitive path, so it merges on a green gate rather than needing a human review.
+
+The DaemonSet deliberately gained **no** probe for this. A readinessProbe on VIP reachability would
+wedge a rollout behind an unrelated Gateway outage (`maxUnavailable: 1`); a liveness probe would tear
+down a tunnel carrying LAN traffic correctly.
+
+## What I verified
+
+Measured live against prod, `--context k3s-prod`, on 2026-10-08:
+
+* **The whole tunnel is healthy**, which is what made this hard: `wg show` on `k3s-prod-1` shows the
+  peer with a real endpoint and bidirectional transfer (346 KiB in / 200 KiB out);
+  `net.ipv4.ip_forward=1`; `wg0` up at `10.90.0.1/24` MTU 1420; `iptables -t nat -S POSTROUTING`
+  carries the `MASQUERADE` rule with 26 packets; `FORWARD -i/-o wg0` ACCEPT with 108 / 171 packets;
+  DaemonSet generation 2, 3/3 ready on the content-hashed `wg-peers-m9kcckcb5t`.
+* **The ARP failure and its symmetry.** On `k3s-prod-1`: `ip neigh show 192.168.2.20` → `FAILED`,
+  while `.21`/`.23` resolve to prod-2's MAC `04:0e:3c:96:b0:94`. On prod-2/prod-3, `.20` resolves to
+  prod-1's `bc:e9:2f:7e:c4:a6`. TCP agrees: `.20:443` and `.20:80` FAIL from prod-1, OK from the
+  other two. Repeated across 3 nodes × 5 VIPs — each node reached every VIP except the ones it alone
+  announced (prod-2 ✗ `.22`/`.21`; prod-3 ✗ `.24`).
+* **Not an artefact of probing from the host netns.** `curl --interface wg0` from prod-1 fails the
+  same way, and a neighbour entry is node-global state, so the forwarded path cannot differ in ARP
+  resolution.
+* **The Gateway itself was never broken:** from the workstation (192.168.2.133)
+  `home.keithjackson.dev` → `200`, `plex.home.keithjackson.dev` → `401`, both at `192.168.2.20`.
+  From prod-1 the router (`.1`) and TrueNAS (`.30`) are `REACHABLE`.
+* **`NotIn` semantics, empirically** — the basis for the fail-back claim in the policy's comment:
+  `kubectl get nodes -l 'kubernetes.io/hostname notin (k3s-prod-1)'` returns prod-2 and prod-3, and
+  `NotIn` on a key no node carries returns **all three**. So an unapplied label excludes nothing and
+  degrades to the old behaviour rather than taking every VIP off the LAN.
+* **The alert metric is really scraped:** `kube_lease_owner` is served by kube-state-metrics (55
+  series, every `cilium-l2announce-*` lease with its holder in `lease_holder`), and `kube_node_labels`
+  already carries the two `egress-*` labels through the same allowlist (`k3s-prod-3` shows
+  `egress-gateway=true`), so the third entry follows a proven path.
+* **The four promtool cases pass** (`platform/monitoring/tests/run.sh` → `SUCCESS`, 55 rules
+  checked), including that `VpnIngressNodeAnnouncesL2VIP` is inert when the label is absent.
+* All edited YAML parses; the rendered `nodeSelector` is the intended two-expression list.
+
+## What I did NOT verify
+
+* **The fix is UNAPPLIED and therefore UNPROVEN end to end.** Applying it needs a mutating
+  `kubectl`/`flux reconcile` plus an Ansible run, which are operator actions (CLAUDE.md §5). I have
+  **not** observed `k3s-prod-1` resolve `192.168.2.20` after the exclusion lands, nor a VPN client
+  reaching an app. The prediction rests on the measured symmetry (non-announcing nodes resolve every
+  VIP, 15/15 data points), not on a test of the changed state. **Verify after merge, in this order:**
+  re-run `node_baseline --tags vpn-ingress`; `kubectl get nodes -L
+  homelab.keithjackson.dev/vpn-ingress` (exactly one node); `kubectl -n kube-system get lease | grep
+  l2announce` (no holder is that node); `ip neigh show 192.168.2.20` from the prod-1 wireguard pod
+  (expect a `lladdr`, not `FAILED`); then a client.
+* **That Cilium actually releases a lease when its holder stops matching the `nodeSelector`.** I
+  assumed re-election and did not test it. If Cilium keeps renewing, the affected Leases must be
+  deleted once by hand to force an election. Check this first if the fix appears not to work.
+* **The brief interruption on apply.** Moving the Gateway and `pihole-dns-udp` leases off prod-1 will
+  blip `192.168.2.20` and `192.168.2.23` for as long as re-election plus gratuitous ARP takes. Not
+  measured, and I did not test whether in-flight LAN sessions survive it.
+* **The Ansible tasks have never been executed.** They parse as YAML and follow
+  `roles/vpn_egress_gateway`'s shape, but neither the label task nor the label-removal task has run,
+  so the `changed_when`/`failed_when` conditions and the `k3s kubectl label ... -` removal path are
+  unproven. The removal task's `failed_when` tolerates `'not found'` in stderr; that string was
+  assumed from kubectl's behaviour, not observed.
+* **The residual off-repo coupling is NOT closed and cannot be, as built.** The router's forward is
+  unreadable from the cluster. If someone re-points it without moving the label, the alert watches
+  the wrong node and stays silent while the VPN breaks identically against a different set of VIPs.
+  Only the runbook guards it. Closing it needs a WireGuard exporter publishing which node is
+  actually terminating peer traffic (`wg show` knows — only the forwarded node has a peer
+  `endpoint`); listed as "Not built" in the component README and **recommended follow-up, not done
+  here**.
+* **Why Cilium behaves this way** — intended, a known upstream issue, or fixed in a later version. I
+  measured the behaviour and did not read Cilium's source or issue tracker, so I cannot say whether
+  an upgrade would remove the need for this exclusion.
+* **`192.168.2.21:8443` (unifi) fails from all three nodes**, including ones where ARP resolves, so
+  it is a different, pre-existing fault. Not investigated, not addressed.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 offline
+  subset only; none of the above is covered by a test that runs in the gate.
+* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).