fix(vpn-ingress): the VPN ingress node must announce no L2 VIPs, or clients route nothing
A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
holds itself: the announcer does not answer its own ARP request, so on the
lease holder the neighbour entry settles at FAILED and packets to that VIP die
before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes
x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced.
platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and
MASQUERADEs remote clients to the node's own address, so a VPN client takes
the node's LAN path and inherits its ARP failures. The router's udp/51820
forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16
Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering
(its VIP has two Services and k3s-prod-2 held the other lease). The tunnel
handshook, names resolved, every app hung, and nothing reported it.
on k3s-prod-1: ip neigh show 192.168.2.20 -> FAILED (its own VIP)
on prod-2/3: ip neigh show 192.168.2.20 -> lladdr ... (prod-1), 443 OK
Mark the node and exclude it from announcing:
* node_baseline labels the forwarded node (node_baseline_vpn_ingress_node,
k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes
the label from any node that no longer matches so moving it MOVES the label.
* l2-announcement-policy.yaml excludes that label, so the node ARP-resolves
every VIP. Fixes all VIPs for VPN clients, not just the one that collided.
Cost: announcement candidacy drops from 3 nodes to 2.
* VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels
(label added to the kube-state-metrics allowlist), with
VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches,
so a missing label would silently re-arm the whole failure AND leave the
critical alert inert. Four promtool cases, including that inertness.
A node LABEL rather than a cluster-vars knob because the alert must read it
too, and extract_rules.py rightly refuses a ${...} placeholder in an alert
expr -- the rule unit-tested would not be the rule deployed. One source of
truth, and it touches no sensitive path.
Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and
"failover is this one line". Both false; the first caused this.
UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the
handoff entry's third section.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
9a41a8e1e5
commit
f5353cbd30
13 files changed
+900
-11
No files matched your search
@@ -119,6 +119,11 @@ configMapGenerator:
|
||||
- update-rollback=update-rollback.md
|
||||
- vpn-down=vpn-down.md
|
||||
- vpn-egress-stale=vpn-egress-stale.md
|
||||
# `VpnIngressNodeAnnouncesL2VIP` — the INBOUND remote-access VPN, not the egress tunnel the two
|
||||
# pages above cover. A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease
|
||||
# it holds itself, and vpn-ingress MASQUERADEs clients to the node's own address, so the
|
||||
# forwarded node must announce nothing (measured 2026-10-08).
|
||||
- vpn-ingress-l2-collision=vpn-ingress-l2-collision.md
|
||||
# Operator procedure, not an alert page. Served because the component it completes is INERT
|
||||
# until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
|
||||
# and because the one check that actually proves it cannot be run from inside the house.
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
# vpn-ingress-l2-collision — the VPN ingress node is announcing a LoadBalancer VIP
|
||||
|
||||
**Alerts:** `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) and `VpnIngressNodeLabelMissing`
|
||||
(warning, 30m)
|
||||
**File:** `platform/monitoring/alerts.yaml`
|
||||
**Related:** `/runbooks/vpn-remote-access` is the setup and enrolment runbook for this component.
|
||||
`/runbooks/vpn-egress-stale` and `/runbooks/vpn-down` are the **other** VPN — the outbound PIA
|
||||
tunnel — and have nothing to do with this.
|
||||
|
||||
## What this means here
|
||||
|
||||
**A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
|
||||
announcer answers ARP for the VIP on behalf of other LAN hosts, but not for the announcing node's
|
||||
own request. On the lease holder the neighbour entry settles at `FAILED`, and every packet to that
|
||||
VIP is dropped before it leaves the node. Measured on prod 2026-10-08, symmetric across all three
|
||||
nodes and five VIPs — each node reached every VIP **except** the ones it alone announced.
|
||||
|
||||
For an ordinary workload this is harmless: pods reach Services through the cluster datapath and
|
||||
never ARP for a VIP. `platform/vpn-ingress` is the exception. It runs WireGuard with
|
||||
`hostNetwork: true` and MASQUERADEs remote clients to the node's own address, so a remote client's
|
||||
packets take that node's LAN path and inherit its ARP failures. **If the node the router forwards
|
||||
`udp/51820` to is also an L2 lease holder, every remote VPN client silently loses all access to the
|
||||
Services behind the VIPs it announces.**
|
||||
|
||||
The node terminating WireGuard is marked with `homelab.keithjackson.dev/vpn-ingress=true`, and
|
||||
`platform-config/cilium/l2-announcement-policy.yaml` excludes that label from announcing anything,
|
||||
so this should be unreachable.
|
||||
|
||||
**The symptom your users report will not sound like a network fault.** On 2026-10-08 the forward sat
|
||||
on `k3s-prod-1`, which held the `shared` Gateway lease, so all 16 Gateway hostnames were dead over
|
||||
the VPN — while PiHole DNS kept answering, because `192.168.2.23` is announced by *two* Services and
|
||||
`k3s-prod-2` held the other lease. So the tunnel handshakes, `wg show` looks perfect, names resolve,
|
||||
and **every app hangs on connect**. Expect "the VPN connects but nothing loads", not "the VPN is
|
||||
down".
|
||||
|
||||
Which Services are affected depends on which lease moved — read `{{ $labels.lease }}` from the
|
||||
alert. `cilium-l2announce-gateway-cilium-gateway-shared` is the worst case: it is every web app at
|
||||
once.
|
||||
|
||||
### If it is `VpnIngressNodeLabelMissing` that fired
|
||||
|
||||
That is the watchdog, and it means the protection is **switched off**, not that something is merely
|
||||
untidy. The exclusion is a `NotIn` on the label, and `NotIn` with an absent key **matches** — so
|
||||
with the label on no node, the policy excludes nothing and every node is an announcement candidate
|
||||
again. `VpnIngressNodeAnnouncesL2VIP` is simultaneously **inert**: its join matches nothing and
|
||||
returns no series, which is indistinguishable from healthy. Treat a quiet critical alert as
|
||||
meaningless until this warning is cleared. Usual cause: a node was rebuilt and `node_baseline` has
|
||||
not been re-run on it.
|
||||
|
||||
## First commands
|
||||
|
||||
Run cluster commands through the pinned toolbox, which supplies the kube context:
|
||||
|
||||
```bash
|
||||
# 1. WHO HOLDS WHAT, and WHO IS THE VPN NODE — the whole question, in two lines.
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n kube-system get lease | grep l2announce'
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/vpn-ingress'
|
||||
# Exactly ONE node should carry vpn-ingress=true, and NO l2announce lease should name it.
|
||||
|
||||
# 2. IS THE EXCLUSION STILL IN THE LIVE POLICY? Expect a matchExpressions list with a
|
||||
# homelab.keithjackson.dev/vpn-ingress NotIn [true] entry — not a bare matchLabels.
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod get ciliuml2announcementpolicy default-l2 -o jsonpath="{.spec.nodeSelector}"; echo'
|
||||
|
||||
# 3. IS THE LABEL REACHING PROMETHEUS? If this is empty the alerts are inert even when
|
||||
# the label is on the node (kube-state-metrics metricLabelsAllowlist).
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n monitoring exec deploy/kube-prometheus-stack-kube-state-metrics -- \
|
||||
wget -qO- localhost:8080/metrics | grep vpn_ingress | head'
|
||||
```
|
||||
|
||||
**Confirm the actual damage**, which is the one measurement that is unambiguous. From the WireGuard
|
||||
pod on the affected node (it shares the host netns, so this is the path a VPN client takes):
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress get pods -o wide' # pick the pod on that node
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- \
|
||||
sh -c "ping -c1 -W1 192.168.2.20 >/dev/null 2>&1; ip neigh show 192.168.2.20"'
|
||||
# BROKEN: 192.168.2.20 dev eno1 FAILED
|
||||
# HEALTHY: 192.168.2.20 dev eno1 lladdr bc:e9:2f:7e:c4:a6 STALE|REACHABLE|DELAY
|
||||
```
|
||||
|
||||
The `ping` is only there to trigger ARP — the VIP does not answer ICMP even when healthy, so
|
||||
**ignore its 100% packet loss**; the neighbour state is the answer. Substitute the VIP that belongs
|
||||
to `{{ $labels.lease }}` if the Gateway is not the one that moved.
|
||||
|
||||
**Which node actually holds the forward** (the router cannot be queried, but the cluster knows):
|
||||
only the forwarded node's `wg show` has a peer with an `endpoint` and a recent `latest handshake`.
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'for p in $(kubectl --context k3s-prod -n vpn-ingress get pods -o name); do
|
||||
echo "== $p"; kubectl --context k3s-prod -n vpn-ingress exec ${p#pod/} -c wireguard -- wg show; done'
|
||||
```
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
1. **The label is on the wrong node** — someone re-pointed the router's forward during a node
|
||||
failover and did not move `node_baseline_vpn_ingress_node`. This is the most likely cause by a
|
||||
wide margin, because the router is off-repo and nothing can check it. Confirm with the `wg show`
|
||||
loop above: the node with the live peer endpoint is the real forward target.
|
||||
*Fix:* merge a PR setting `node_baseline_vpn_ingress_node`
|
||||
(`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`) to that node, then re-run the role —
|
||||
`ansible-playbook -i inventory/hosts.ini <playbook> -e k3s_hosts=k3s_prod --tags vpn-ingress`.
|
||||
The role labels the new node and **removes** the label from the old one. Then confirm Cilium
|
||||
re-elected the lease off the newly-labelled node.
|
||||
|
||||
2. **The exclusion was edited out of `platform-config/cilium/l2-announcement-policy.yaml`**, or the
|
||||
file was reverted to a bare `matchLabels: {kubernetes.io/os: linux}`. Command 2 above shows this
|
||||
directly. *Fix:* restore the `matchExpressions` form; the file's own header explains why it is
|
||||
shaped that way.
|
||||
|
||||
3. **The label is missing entirely** (`VpnIngressNodeLabelMissing` firing) — a node rebuild without
|
||||
a `node_baseline` re-run. *Fix:* re-run the role as above.
|
||||
|
||||
4. **The label is on the node but not in Prometheus** — `metricLabelsAllowlist` in
|
||||
`platform/monitoring/kube-prometheus-stack.yaml` lost the `vpn-ingress` entry. Then the critical
|
||||
alert is inert *and* the watchdog fires. Command 3 above distinguishes this from cause 3.
|
||||
|
||||
5. **Cilium kept renewing a lease its holder is no longer eligible for.** The exclusion is correct
|
||||
and in place, but the lease never re-elected. This was never verified when the fix was authored
|
||||
(see `decisions/2026-10-17-vpn-ingress-l2-lease-collision.md`). *Fix:* delete that one Lease
|
||||
object to force an election — `kubectl -n kube-system delete lease <name>` — and watch it come
|
||||
back on a different node.
|
||||
|
||||
6. **A Cilium upgrade changed `NotIn` semantics or the `nodeSelector` field.** Least likely, but it
|
||||
is the reason this is a runtime alert rather than a static check on the policy text.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
**Act now, don't wait for a window.** Remote access is fully broken for the affected VIPs while this
|
||||
is firing, and it does **not** self-heal: a lease holder keeps renewing, so this condition persists
|
||||
indefinitely once entered. That is also why the 15m `for:` is safe — a transient move during a node
|
||||
reboot or a Cilium roll resolves on its own well inside it.
|
||||
|
||||
**The fastest honest mitigation is to move the forward, not the leases.** Re-pointing the router's
|
||||
`udp/51820` at a node that holds no lease for the affected VIP restores service in seconds, with no
|
||||
cluster change and no new client profiles — all three nodes answer the same keys. Follow it with the
|
||||
`node_baseline_vpn_ingress_node` PR and role re-run so the two agree again; until you do, this alert
|
||||
is watching the wrong node. Note each node's own announcements when choosing: as of 2026-10-08,
|
||||
prod-2 announced crafty `192.168.2.22` and unifi `192.168.2.21`, prod-3 announced plex
|
||||
`192.168.2.24`, so neither is a cost-free target.
|
||||
|
||||
**Do not** `kubectl edit` the `CiliumL2AnnouncementPolicy` to fix this — Flux owns it and will revert
|
||||
you, and the drift is the kind that outlives the memory of making it. The two acceptable hand
|
||||
actions are deleting a single Lease to force re-election (cause 5), and `kubectl label node` as a
|
||||
*temporary* stopgap while the PR lands — note that a node rebuild or a `node_baseline` run will undo
|
||||
a hand-applied label, so it is not a fix.
|
||||
|
||||
**Stop and escalate to the operator if** the leases will not leave the excluded node after a forced
|
||||
re-election, or if the intended forward target is unclear. Both need a decision about which node
|
||||
carries remote access, which is an operator call, not a runbook step.
|
||||
|
||||
**Known gap, so you are not misled:** if someone re-points the router without moving the label,
|
||||
`VpnIngressNodeAnnouncesL2VIP` watches the wrong node and **stays silent** while the VPN breaks in
|
||||
exactly the way described above. There is no in-cluster signal for that today; closing it needs a
|
||||
WireGuard exporter publishing which node is actually terminating peer traffic
|
||||
(`platform/vpn-ingress/README.md`, "Not built") — the `wg show` loop above is the manual substitute.
|
||||
Until then, if remote access is broken and both alerts are quiet, **check the forward against the
|
||||
label by hand before believing the alerts.**
|
||||
@@ -171,14 +171,57 @@ server without a `wg0` "ready".
|
||||
On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
|
||||
`192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.
|
||||
|
||||
Pick any node — all three run an identical listener. Note which one you chose here:
|
||||
> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node.
|
||||
>
|
||||
> This step used to say "pick any node — all three run an identical listener". **That sentence
|
||||
> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a
|
||||
> cluster-vars knob instead of a free choice here.
|
||||
>
|
||||
> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
|
||||
> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease
|
||||
> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they
|
||||
> leave the node. Measured on prod 2026-10-08:
|
||||
>
|
||||
> ```text
|
||||
> on k3s-prod-1 (held the shared Gateway lease): ip neigh show 192.168.2.20 -> FAILED
|
||||
> on k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr … (prod-1)
|
||||
> ```
|
||||
>
|
||||
> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN
|
||||
> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway
|
||||
> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so
|
||||
> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported
|
||||
> it.
|
||||
>
|
||||
`infra/ansible/roles/node_baseline` now labels the forwarded node
|
||||
> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in
|
||||
> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes
|
||||
> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI
|
||||
> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a
|
||||
> labelled node taking a lease, but it CANNOT catch the forward being moved without the label.
|
||||
|
||||
Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`).
|
||||
Note it here:
|
||||
|
||||
```
|
||||
# WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set <date>
|
||||
# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml
|
||||
```
|
||||
|
||||
**Node loss = failover is this one line.** Re-point the forward at another node address; every
|
||||
device keeps working with the same profile, because all three nodes answer the same keys.
|
||||
**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that
|
||||
was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and
|
||||
reintroduces the failure above against a different set of addresses (prod-2 announces crafty and
|
||||
unifi; prod-3 announces plex). Do both, in this order:
|
||||
|
||||
1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role
|
||||
(`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium
|
||||
then moves the leases off the newly-labelled node and onto its peers.
|
||||
2. **Re-point the router** forward at the new node's address.
|
||||
|
||||
Every device keeps working with the same profile either way; all three nodes answer the same keys,
|
||||
and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that
|
||||
handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because
|
||||
that alert watches the node the knob names, not the node your router actually points at.
|
||||
|
||||
Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
|
||||
already silent to unauthenticated probes (README, "Why this is safe to expose").
|
||||
|
||||
Reference in new issue
Block a user