fix(vpn-ingress): the VPN ingress node must announce no L2 VIPs, or clients route nothing

A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself: the announcer does not answer its own ARP request, so on the lease holder the neighbour entry settles at FAILED and packets to that VIP die before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced. platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and MASQUERADEs remote clients to the node's own address, so a VPN client takes the node's LAN path and inherits its ARP failures. The router's udp/51820 forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16 Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering (its VIP has two Services and k3s-prod-2 held the other lease). The tunnel handshook, names resolved, every app hung, and nothing reported it. on k3s-prod-1: ip neigh show 192.168.2.20 -> FAILED (its own VIP) on prod-2/3: ip neigh show 192.168.2.20 -> lladdr ... (prod-1), 443 OK Mark the node and exclude it from announcing: * node_baseline labels the forwarded node (node_baseline_vpn_ingress_node, k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes the label from any node that no longer matches so moving it MOVES the label. * l2-announcement-policy.yaml excludes that label, so the node ARP-resolves every VIP. Fixes all VIPs for VPN clients, not just the one that collided. Cost: announcement candidacy drops from 3 nodes to 2. * VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels (label added to the kube-state-metrics allowlist), with VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches, so a missing label would silently re-arm the whole failure AND leave the critical alert inert. Four promtool cases, including that inertness. A node LABEL rather than a cluster-vars knob because the alert must read it too, and extract_rules.py rightly refuses a ${...} placeholder in an alert expr -- the rule unit-tested would not be the rule deployed. One source of truth, and it touches no sensitive path. Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and "failover is this one line". Both false; the first caused this. UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the handoff entry's third section. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-17 21:58:12 -05:00
1 parent 9a41a8e1e5
commit f5353cbd30
@@ -1147,6 +1147,97 @@ spec:
              metricLabelsAllowlist.
            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
        # -----------------------------------------------------------------------------------------
+        # REMOTE-ACCESS VPN (platform/vpn-ingress) -- NOT the egress tunnel above. Different
+        # direction, different workload: this is inbound WireGuard for remote devices.
+        # -----------------------------------------------------------------------------------------
+        - alert: VpnIngressNodeAnnouncesL2VIP
+          # THE FAILURE THIS CATCHES. MEASURED on prod 2026-10-08: a node CANNOT reach a
+          # LoadBalancer VIP whose Cilium L2 announcement lease it holds itself -- the announcer
+          # does not answer its own ARP request, so the neighbour entry settles at `FAILED` and
+          # packets die before leaving the node (`ip neigh show 192.168.2.20` -> FAILED on the
+          # lease holder, resolved on both others). platform/vpn-ingress runs WireGuard on
+          # `hostNetwork` and MASQUERADEs remote clients to the node's own address, so if the node
+          # the router forwards udp/51820 to is also a lease holder, VPN clients cannot reach the
+          # VIPs it announces. On 2026-10-08 that was the `shared` Gateway: all 16 app hostnames
+          # dead over the VPN, while PiHole DNS kept answering (its VIP has two Services and the
+          # other lease sat on prod-2) -- so the tunnel handshook, names resolved, and every app
+          # hung. Nothing reported it for ~2 days.
+          #
+          # platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled
+          # `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so this should be
+          # permanently unsatisfiable. The alert exists because the exclusion has ways to stop
+          # working that no static check on that file's TEXT can see: the label being removed or
+          # moved to the wrong node, a Cilium upgrade changing `NotIn` semantics or the
+          # nodeSelector field, and the selector silently failing to match. This watches the
+          # OUTCOME -- who actually holds the lease -- so all of them surface the same way.
+          #
+          # WHY THE LABEL JOIN RATHER THAN A NODE NAME. The node's identity is NOT written into
+          # this expr, because the harness (tests/extract_rules.py) refuses any alert whose expr
+          # contains a dollar-brace placeholder -- spelled in words because a literal one here is
+          # unparseable to envsubst and would fail this whole Kustomization. Rightly so: the rule it
+          # unit-tests would not be the
+          # rule deployed. Joining `kube_lease_owner` against `kube_node_labels` keeps ONE source
+          # of truth (the label itself, which is also what the policy selects on), so this alert
+          # cannot drift out of agreement with the exclusion it guards. `label_replace` is needed
+          # because kube_lease_owner names the holder `lease_holder` while kube_node_labels names
+          # it `node`; the join is then on a shared `node` label. The label is in
+          # kube-state-metrics' metricLabelsAllowlist (kube-prometheus-stack.yaml) -- without that
+          # line this rule is inert, which `VpnIngressNodeLabelMissing` below exists to catch.
+          #
+          # `for: 15m` because leases legitimately move during a node reboot or a Cilium roll and
+          # Cilium re-elects within seconds; a real collision persists, since the lease holder
+          # keeps renewing and it never self-heals.
+          expr: |
+            max by (lease) (
+              label_replace(
+                kube_lease_owner{namespace="kube-system", lease=~"cilium-l2announce-.*"},
+                "node", "$1", "lease_holder", "(.+)"
+              )
+              * on (node) group_left ()
+              kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"}
+            ) == 1
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "VPN ingress node is announcing a LoadBalancer VIP ({{ $labels.lease }})"
+            description: >-
+              The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access
+              WireGuard, and it now holds the Cilium L2 announcement lease `{{ $labels.lease }}`. A
+              node cannot reach a VIP it announces itself, so every remote VPN client has just lost
+              all access to that VIP's Service -- silently, with DNS still resolving. Check that
+              platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that
+              the label is still on the node the router forwards udp/51820 to.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
+        - alert: VpnIngressNodeLabelMissing
+          # THE WATCHDOG for the rule above, and the reason it is not optional. The exclusion in
+          # l2-announcement-policy.yaml is a `NotIn` on this label, and `NotIn` with an ABSENT key
+          # MATCHES -- so if the label is on NO node, the policy excludes nothing, every node is an
+          # announcement candidate again, and the 2026-10-08 failure is fully re-armed. The alert
+          # above would not fire either: its join would match nothing and return no series. Present,
+          # evaluating, and incapable of firing -- the exact "inert critical alert" shape that
+          # VPNEgressUplinkLabelMissing exists for on the egress side, and that this repo has been
+          # burned by before.
+          #
+          # The label is applied by infra/ansible/roles/node_baseline
+          # (node_baseline_vpn_ingress_node); the most likely cause of it going absent is a node
+          # rebuild where that role has not been re-run.
+          expr: absent(kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"})
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF"
+            description: >-
+              No node publishes homelab.keithjackson.dev/vpn-ingress=true, so
+              l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none:
+              any node may now announce a LoadBalancer VIP, including the one terminating
+              remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so
+              the warning matters more than its severity suggests. Re-run the node_baseline role on
+              the forwarded node, and confirm the label is in kube-state-metrics'
+              metricLabelsAllowlist.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
+        # -----------------------------------------------------------------------------------------
        # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
        # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
        # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`