fix(vpn-ingress): the VPN ingress node must announce no L2 VIPs, or clients route nothing
gate / gate (pull_request) Successful in 2m22s
gate / conformance (pull_request) Has been skipped

A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
holds itself: the announcer does not answer its own ARP request, so on the
lease holder the neighbour entry settles at FAILED and packets to that VIP die
before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes
x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced.

platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and
MASQUERADEs remote clients to the node's own address, so a VPN client takes
the node's LAN path and inherits its ARP failures. The router's udp/51820
forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16
Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering
(its VIP has two Services and k3s-prod-2 held the other lease). The tunnel
handshook, names resolved, every app hung, and nothing reported it.

  on k3s-prod-1:  ip neigh show 192.168.2.20 -> FAILED      (its own VIP)
  on prod-2/3:    ip neigh show 192.168.2.20 -> lladdr ...  (prod-1), 443 OK

Mark the node and exclude it from announcing:

* node_baseline labels the forwarded node (node_baseline_vpn_ingress_node,
  k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes
  the label from any node that no longer matches so moving it MOVES the label.
* l2-announcement-policy.yaml excludes that label, so the node ARP-resolves
  every VIP. Fixes all VIPs for VPN clients, not just the one that collided.
  Cost: announcement candidacy drops from 3 nodes to 2.
* VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels
  (label added to the kube-state-metrics allowlist), with
  VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches,
  so a missing label would silently re-arm the whole failure AND leave the
  critical alert inert. Four promtool cases, including that inertness.

A node LABEL rather than a cluster-vars knob because the alert must read it
too, and extract_rules.py rightly refuses a ${...} placeholder in an alert
expr -- the rule unit-tested would not be the rule deployed. One source of
truth, and it touches no sensitive path.

Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and
"failover is this one line". Both false; the first caused this.

UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the
handoff entry's third section.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-17 21:58:12 -05:00
1 parent 9a41a8e1e5
commit f5353cbd30
13 files changed
+900 -11

No files matched your search

+33 -3
View File
@@ -62,9 +62,39 @@
# as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to
# whichever node the scheduler picked and silently invalidate that forward on every reschedule. A
# DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of
# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no
# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own
# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each.
# them, and if that node is lost every other node is already warm and answers the same keys, so no
# client profile ever changes. Each node binds udp/51820 in its own netns, so there is no conflict,
# and the two idle listeners cost ~15 MiB each.
#
# BUT THE NODES ARE NOT INTERCHANGEABLE, AND THIS COMMENT USED TO CLAIM THEY WERE.
# It said "FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work". That was wrong, and the
# wording cost two days of silently broken remote access on 2026-10-08. MEASURED that day: a node
# cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds ITSELF — the announcer
# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and packets to
# that VIP die before leaving the node:
#
# k3s-prod-1, holding the shared Gateway lease: ip neigh show 192.168.2.20 -> FAILED
# k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr …(prod-1)
#
# Because this pod MASQUERADEs clients to the node's own address (see PostUp below), a remote client
# inherits exactly that failure. With the forward on the Gateway's announcing node, all 16 Gateway
# hostnames were unreachable over the VPN while PiHole DNS still resolved — a tunnel that
# handshakes, resolves names, and then hangs on every single app.
#
# THE FIX IS DECLARATIVE AND LIVES OUTSIDE THIS FILE: the forwarded node is labelled
# homelab.keithjackson.dev/vpn-ingress=true by infra/ansible/roles/node_baseline (from
# node_baseline_vpn_ingress_node), and platform-config/cilium/l2-announcement-policy.yaml excludes
# that label from announcing ANY VIP, so the node can ARP-resolve all of them. A label rather than a
# cluster-vars knob because the alert that guards this has to read it too, and the alert unit-test
# harness refuses a postBuild placeholder in an expr. Failover is therefore TWO steps — move
# node_baseline_vpn_ingress_node and re-run the role, THEN re-point the router
# (runbooks/vpn-remote-access.md step 3).
# `VpnIngressNodeAnnouncesL2VIP` (platform/monitoring/alerts.yaml) fires if the two collide again.
#
# This pod deliberately does NOT probe that itself. A readinessProbe on VIP reachability would wedge
# a DaemonSet rollout behind an unrelated Gateway outage (maxUnavailable: 1 below), and a liveness
# probe would tear down a tunnel that is carrying LAN traffic perfectly well. The condition is a
# property of the CLUSTER's announcement layout, so it is watched there.
#
# All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the
# DaemonSet to cover the cluster; add them if a node is ever tainted.