revert(vpn-ingress): withdraw the L2 lease-collision fix — a node CAN reach a VIP it announces

PR #69 claimed a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
lease it holds, and excluded the VPN ingress node from announcing on that basis.
The claim makes a sharp prediction; today's power-cycle put ALL EIGHT
cilium-l2announce-* leases on k3s-prod-1, and the prediction failed.

Measured 2026-10-17 from the wireguard pod's host netns on each node: the holder
reaches 192.168.2.20/.22/.23/.24 normally (curl 200 in 18ms on .20) while its
ip neigh for every one of them sits at FAILED. The asserted cause is present
without the asserted effect, which falsifies it. .21 (unifi) fails from all three
nodes and is a separate pre-existing fault.

It is inert because no path resolves a VIP by ARP: cilium-dbg reports
KubeProxyReplacement on [eno1, wg0] and programs 192.168.2.20:443 =>
127.0.0.1:13606, so VIPs are DNAT'd in BPF before any neighbour lookup, on the
forwarded path as well as the host one.

Both probes behind the original diagnosis were bad. `ip neigh` FAILED is
permanent and harmless, and a ping artefact. `curl --interface wg0` — the control
used to rule out a host-netns artefact — fails on all three nodes, including ones
whose ARP resolves, so it discriminates nothing. The deeper error: a MASQUERADEd
client is FORWARDED while every pod-runnable probe is socket-originated. Sharing
a netns is not sharing a hook.

Reverted the policy nodeSelector, the Ansible node label, both alerts, their
promtool tests and the kube-state-metrics allowlist entry. Kept documentation
only: the measurements, which probes lie, and the fact that the forwarded path
remains unmeasured without a connected client.

Also recorded, for the reported "LAN and cluster work, internet does not":
new-peer.sh has emitted a FULL tunnel (0.0.0.0/0) by default since f92707f, so
there is no full-tunnel option to add and no code change is made. The server side
is capable — ip_forward=1, the 10.90.0.0/24 MASQUERADE present with an empty
CILIUM_POST_nat ahead of it, FORWARD accepts, default route intact, nodes reach
1.1.1.1. A full tunnel egresses the HOUSE WAN, not PIA: the egress policy selects
pods by egress-class=vpn, this pod is labelled internet, and tun0 exists on
k3s-prod-3 only. Root cause NOT found; no client has connected since the
power-cycle (0 forwarded packets, no handshake), so it is not reproducible here.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

platform-config/cilium/l2-announcement-policy.yaml

@@ -16,71 +16,6 @@ spec:
   externalIPs: false
   interfaces:
     - ^e(n|th)
-  # ===============================================================================================
-  # WHY THIS IS A matchExpressions LIST AND NOT `matchLabels: {kubernetes.io/os: linux}`
-  # ===============================================================================================
-  # The second expression excludes every node labelled `homelab.keithjackson.dev/vpn-ingress=true`
-  # from announcing ANY LoadBalancer VIP. That is the node the SOHO router forwards WAN udp/51820 to.
-  # It is not tuning; it is the fix for a measured, silent, total failure of remote access.
-  #
-  # MEASURED ON PROD 2026-10-08: A NODE CANNOT REACH A VIP IT ANNOUNCES ITSELF. Cilium's L2
-  # announcer answers ARP for the VIP on behalf of OTHER LAN hosts, but not for the announcing node
-  # itself. On the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP
-  # is dropped before it leaves the node:
-  #
-  #   k3s-prod-1, holding cilium-l2announce-gateway-cilium-gateway-shared:
-  #     ip neigh show 192.168.2.20  ->  192.168.2.20 dev eno1 FAILED       (its own VIP)
-  #     nc -z 192.168.2.20 443      ->  FAIL          nc -z 192.168.2.20 80 -> FAIL
-  #   k3s-prod-2 and k3s-prod-3, holding no lease for it:
-  #     ip neigh show 192.168.2.20  ->  lladdr bc:e9:2f:7e:c4:a6 (prod-1)   nc 443/80 -> OK
-  #
-  # Symmetric across all 3 nodes x 5 VIPs: every node reached every VIP EXCEPT the ones it alone
-  # announced (prod-2 could not reach crafty 192.168.2.22 or unifi 192.168.2.21; prod-3 could not
-  # reach plex 192.168.2.24). So this is a property of L2 announcement, not of one Service.
-  #
-  # WHY IT ONLY HURTS THE VPN NODE. For an ordinary workload this is harmless: pods reach Services
-  # through the cluster datapath and never ARP for a VIP. platform/vpn-ingress is the exception --
-  # it runs WireGuard with `hostNetwork: true` and MASQUERADEs remote clients to the node's own
-  # address, so a VPN client's packets take the node's LAN path and inherit exactly this ARP
-  # failure. On 2026-10-08 the forward pointed at k3s-prod-1, which held the Gateway lease, and all
-  # 16 Gateway hostnames were unreachable over the VPN while PiHole DNS still answered -- so names
-  # resolved and every app hung. Nothing in the cluster reported it; the alerts below now do.
-  #
-  # WHY A NODE LABEL AND NOT A cluster-vars KNOB. A knob would put the node's identity in git, which
-  # is more reviewable -- but it CANNOT BE USED BY THE ALERT that guards this, and an unguarded
-  # invariant here is worth less than a reviewable one. The alert harness
-  # (platform/monitoring/tests/extract_rules.py) REFUSES any alert whose `expr` contains a
-  # dollar-brace postBuild placeholder (spelled without braces here ON PURPOSE: this file is itself
-  # envsubst'd, and a literal dollar-brace-dot-dot-dot is UNPARSEABLE to envsubst, which fails the
-  # WHOLE Kustomization -- ci: flux-postbuild-substitution caught exactly this comment), because the
-  # rule it would unit-test is not the rule that gets
-  # deployed. A label is readable from BOTH sides: this selector matches on it, and
-  # `kube_node_labels` carries it into Prometheus so `VpnIngressNodeAnnouncesL2VIP` can join lease
-  # holders against it. One source of truth, no git-vs-cluster skew, and the policy stays
-  # env-agnostic as its header above claims. The label is set by
-  # infra/ansible/roles/node_baseline (node_baseline_vpn_ingress_node), the same shape
-  # roles/vpn_egress_gateway already uses for `egress-gateway`.
-  #
-  # THE COST, STATED: VIP announcement candidacy drops from 3 nodes to 2. That is still redundant,
-  # and the trade is deliberate -- the excluded node is the one whose *outbound* reach to every VIP
-  # matters most, because it carries every remote client. If BOTH remaining nodes are down the VIPs
-  # go dark where previously one node might have served them, but at that point the cluster has one
-  # node left and far worse problems.
-  #
-  # `NotIn` with an ABSENT key MATCHES (k8s apimachinery: a missing label is "not in the set"), so
-  # the other two nodes need no label and nothing has to be applied to them. If the label is never
-  # applied anywhere, this excludes nothing and the policy degrades to its pre-2026-10-08 behaviour
-  # -- the status quo, not an outage: it fails back, not down. That silent-degradation path is
-  # exactly why `VpnIngressNodeLabelMissing` exists as a watchdog: without it, the alert below
-  # would be structurally INERT (its join would match nothing) and would look identical to healthy.
   nodeSelector:
-    matchExpressions:
-      - key: kubernetes.io/os
-        operator: In
-        values:
-          - linux
-      # The node terminating remote-access WireGuard must announce nothing -- see above.
-      - key: homelab.keithjackson.dev/vpn-ingress
-        operator: NotIn
-        values:
-          - "true"
+    matchLabels:
+      kubernetes.io/os: linux