From e82dab5df50b00e0783d09c44699d399d20fef29 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 17 Oct 2026 23:58:40 -0500 Subject: [PATCH] =?UTF-8?q?revert(vpn-ingress):=20withdraw=20the=20L2=20le?= =?UTF-8?q?ase-collision=20fix=20=E2=80=94=20a=20node=20CAN=20reach=20a=20?= =?UTF-8?q?VIP=20it=20announces?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #69 claimed a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds, and excluded the VPN ingress node from announcing on that basis. The claim makes a sharp prediction; today's power-cycle put ALL EIGHT cilium-l2announce-* leases on k3s-prod-1, and the prediction failed. Measured 2026-10-17 from the wireguard pod's host netns on each node: the holder reaches 192.168.2.20/.22/.23/.24 normally (curl 200 in 18ms on .20) while its ip neigh for every one of them sits at FAILED. The asserted cause is present without the asserted effect, which falsifies it. .21 (unifi) fails from all three nodes and is a separate pre-existing fault. It is inert because no path resolves a VIP by ARP: cilium-dbg reports KubeProxyReplacement on [eno1, wg0] and programs 192.168.2.20:443 => 127.0.0.1:13606, so VIPs are DNAT'd in BPF before any neighbour lookup, on the forwarded path as well as the host one. Both probes behind the original diagnosis were bad. `ip neigh` FAILED is permanent and harmless, and a ping artefact. `curl --interface wg0` — the control used to rule out a host-netns artefact — fails on all three nodes, including ones whose ARP resolves, so it discriminates nothing. The deeper error: a MASQUERADEd client is FORWARDED while every pod-runnable probe is socket-originated. Sharing a netns is not sharing a hook. Reverted the policy nodeSelector, the Ansible node label, both alerts, their promtool tests and the kube-state-metrics allowlist entry. Kept documentation only: the measurements, which probes lie, and the fact that the forwarded path remains unmeasured without a connected client. Also recorded, for the reported "LAN and cluster work, internet does not": new-peer.sh has emitted a FULL tunnel (0.0.0.0/0) by default since f92707f, so there is no full-tunnel option to add and no code change is made. The server side is capable — ip_forward=1, the 10.90.0.0/24 MASQUERADE present with an empty CILIUM_POST_nat ahead of it, FORWARD accepts, default route intact, nodes reach 1.1.1.1. A full tunnel egresses the HOUSE WAN, not PIA: the egress policy selects pods by egress-class=vpn, this pod is labelled internet, and tun0 exists on k3s-prod-3 only. Root cause NOT found; no client has connected since the power-cycle (0 forwarded packets, no handshake), so it is not reproducible here. Co-Authored-By: Claude Opus 5 (1M context) --- ...26-10-17-vpn-ingress-l2-lease-collision.md | 293 +++++++++++------- .../inventory/group_vars/k3s_prod/vars.yml | 19 -- .../roles/node_baseline/defaults/main.yml | 27 -- .../roles/node_baseline/tasks/main.yml | 42 --- .../cilium/l2-announcement-policy.yaml | 69 +---- platform/monitoring/alerts.yaml | 91 ------ .../monitoring/kube-prometheus-stack.yaml | 14 +- .../monitoring/tests/vpn-ingress-l2.test.yaml | 203 ------------ platform/vpn-ingress/README.md | 118 ++++--- platform/vpn-ingress/daemonset.yaml | 36 +-- runbooks/kustomization.yaml | 5 - runbooks/vpn-ingress-l2-collision.md | 165 ---------- runbooks/vpn-remote-access.md | 127 +++++--- 13 files changed, 338 insertions(+), 871 deletions(-) delete mode 100644 platform/monitoring/tests/vpn-ingress-l2.test.yaml delete mode 100644 runbooks/vpn-ingress-l2-collision.md diff --git a/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md b/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md index b21be0e..e0923b8 100644 --- a/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md +++ b/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md @@ -1,137 +1,198 @@ --- date: 2026-10-17 branch: fix/vpn-ingress-l2-lease-collision -title: "A node cannot reach a VIP it announces — so the VPN ingress node must announce nothing" +title: "WITHDRAWN: a node CAN reach a VIP it announces. My L2-lease diagnosis was wrong" --- ## What changed -The first device enrolled on the remote-access VPN handshook, resolved names, and could not open a -single connection to any cluster web app. The tunnel was not at fault. The fault is a property of -Cilium L2 announcement that nothing in this repo knew about: +**This branch previously proposed a fix. That fix has been reverted in full. The diagnosis behind it +was wrong.** What is left is documentation of what was actually measured, so the wrong conclusion is +not re-derived — it is very easy to re-derive from `ip neigh` alone. -**A node cannot reach a LoadBalancer VIP whose `CiliumL2AnnouncementPolicy` lease it holds itself.** -The announcer answers ARP for the VIP on behalf of other LAN hosts but not for its own request, so -on the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP is dropped -before it leaves the node. +### The claim, and how it died -`platform/vpn-ingress` is uniquely exposed because it runs WireGuard with `hostNetwork: true` and -MASQUERADEs remote clients to the node's own address — a VPN client's packets take the node's LAN -path and inherit its ARP failures. Ordinary pods never do. The router's `udp/51820` forward pointed -at `k3s-prod-1`, which held `cilium-l2announce-gateway-cilium-gateway-shared`, so **all 16 `shared` -Gateway hostnames were dead over the VPN** — while PiHole DNS kept answering, because -`192.168.2.23` is announced by *two* Services and `k3s-prod-2` held the other lease. Names resolved -and every app hung on connect. +The earlier version of this entry claimed: *a node cannot reach a LoadBalancer VIP whose Cilium L2 +announcement lease it holds itself* — the announcer does not answer its own ARP request, the +neighbour entry settles at `FAILED`, and packets to that VIP die before leaving the node. Because +`platform/vpn-ingress` runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's +own address, a VPN client was said to inherit that. The fix labelled the forwarded node and excluded +that label from announcing, with two alerts and four promtool tests behind it. -Seven files: +The claim makes a sharp prediction. After today's UPS power-cycle **all eight** `cilium-l2announce-*` +leases are held by `k3s-prod-1` (gateway, pihole tcp+udp, crafty tcp+udp, unifi tcp+udp, plex) — +the extreme case. The prediction was therefore that `k3s-prod-1` should reach **none** of the five +VIPs and the other two nodes should reach **all**. -* **`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`** — new - `node_baseline_vpn_ingress_node`; the matching node gets - `homelab.keithjackson.dev/vpn-ingress=true`, and a second task **removes** the label from any node - that no longer matches, so moving the value moves the label rather than adding a second one. Same - shape as `roles/vpn_egress_gateway`'s `egress-gateway` label. Tagged `vpn-ingress`; skipped - entirely when the value is empty. -* **`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`** — set to `k3s-prod-1`, with a - `[parity]` note that it must equal the router's forward. -* **`platform-config/cilium/l2-announcement-policy.yaml`** — `nodeSelector` becomes a - `matchExpressions` list keeping `kubernetes.io/os: linux` and adding - `homelab.keithjackson.dev/vpn-ingress NotIn ["true"]`. The VPN node then holds no lease, so it - ARP-resolves every VIP. This fixes **all** VIPs for VPN clients (Gateway, crafty, plex, pihole, - unifi), not just the one that happened to collide. -* **`platform/monitoring/kube-prometheus-stack.yaml`** — the label added to kube-state-metrics' - `metricLabelsAllowlist`, which is what lets the invariant be watched at all. -* **`platform/monitoring/alerts.yaml`** — `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) joins - `kube_lease_owner` against `kube_node_labels` via `label_replace`, plus - `VpnIngressNodeLabelMissing` (warning, 30m) as its watchdog. -* **`platform/monitoring/tests/vpn-ingress-l2.test.yaml`** — four promtool cases: fires on the real - collision, quiet in the fixed state, quiet across a transient 11-minute lease move (pinning the - `for: 15m`), and the label-absent case asserting the critical alert is inert *and* the watchdog - fires. -* **`runbooks/vpn-ingress-l2-collision.md`** (new, registered in `runbooks/kustomization.yaml`), - **`runbooks/vpn-remote-access.md`**, **`platform/vpn-ingress/{daemonset.yaml,README.md}`** — step 3 - of the setup runbook said *"pick any node — all three run an identical listener"* and *"node loss = - failover is this one line"*. Both were false and the first caused this. Failover is now documented - as two ordered changes. +**The prediction failed.** Measured from the host netns of the `wireguard` pod on each node: -**Design note — why a node label and not a `cluster-vars` knob.** The first attempt added -`WG_INGRESS_NODE` to `clusters/base/cluster-vars.yaml` and substituted it into both the policy and -the alert. The alert half is **impossible**: `platform/monitoring/tests/extract_rules.py` rejects any -alert whose `expr` contains a `${...}` placeholder, because the rule it would unit-test is not the -rule deployed. Rather than duplicate the node name into the alert as a literal (two sources that can -skew), the label is readable from both sides — the selector matches on it and `kube_node_labels` -carries it into Prometheus. One source of truth, and as a side effect the change touches no -sensitive path, so it merges on a green gate rather than needing a human review. +| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) | +|---|---|---| +| `ip neigh show 192.168.2.20` | `FAILED`, no lladdr | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` | +| `curl -k https://192.168.2.20/` | **`200` in 18 ms** | `200` in 13–40 ms | +| `nc -z 192.168.2.20 443` | **OK** | OK | +| `192.168.2.22:25565`, `.23:53`, `.24:32400` | **all OK / `200`** | all OK / `200` | +| `192.168.2.21:8443` | FAIL | **FAIL** (both) | -The DaemonSet deliberately gained **no** probe for this. A readinessProbe on VIP reachability would -wedge a rollout behind an unrelated Gateway outage (`maxUnavailable: 1`); a liveness probe would tear -down a tunnel carrying LAN traffic correctly. +The asserted **cause is still present** today — `FAILED` on the holder, for all five VIPs — while the +asserted **effect is absent**. That is sufficient to falsify the causal claim without having to +explain anything else. The ARP asymmetry is real and reproducible; it is also **inert**. + +### Why it is inert + +No traffic path resolves a LoadBalancer VIP by ARP: + +* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and + `Masquerading: BPF [eno1, wg0]` (Cilium v1.20.1, TCX attach, tunnel/vxlan). Cilium's LB datapath + is attached to **`wg0` as well as `eno1`**, so a remote client's packet arriving on `wg0` for a VIP + is DNAT'd **in BPF, before any route or neighbour lookup for the VIP**. +* `cilium-dbg service list` programs `192.168.2.20:443 => 127.0.0.1:13606` — the shared Gateway's + backend is a **node-local** Envoy socket, so every node serves that VIP without a LAN packet. +* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`), + which is why `curl` succeeds on the holder. Discriminator: a real service port connects while a + bogus one (`nc -z 192.168.2.20 9999`) fails, identically on all three nodes — service-aware DNAT, + not wire traffic. + +### The two probes that misled me + +Both were cited as evidence in the previous version of this entry. + +* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless, and is also a probing + artefact: VIPs answer no ICMP, so a `ping` leaves a failed entry whether or not anything is wrong. +* **`curl --interface wg0`.** The previous entry used this as the control that ruled out a + host-netns artefact. It is not a control: it fails (`rc=28`) from **all three** nodes, including + the two whose ARP for the VIP resolves perfectly. It is uncorrelated with lease holding, so it + discriminates nothing. + +**The deeper error:** a MASQUERADEd client's packets are **forwarded** (netfilter `FORWARD`, then +routing); everything runnable from the pod is **socket-originated**. Different hooks. A host-netns +probe cannot measure the forwarded path in either direction — it reports success where the forwarded +path might fail, and `--interface wg0` reports failure where it does not. Sharing a netns is not +sharing a hook, and that is the assumption the original diagnosis rested on. + +### Files + +Reverted to `origin/main`: `platform-config/cilium/l2-announcement-policy.yaml`, +`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`, +`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`, `platform/monitoring/alerts.yaml`, +`platform/monitoring/kube-prometheus-stack.yaml`, `runbooks/kustomization.yaml`. Deleted: +`platform/monitoring/tests/vpn-ingress-l2.test.yaml`, `runbooks/vpn-ingress-l2-collision.md`. + +Kept, rewritten from scratch: + +* **`platform/vpn-ingress/README.md`** — two measured sections: the L2 table above with why it is + inert and which probes lie, and the internet-egress facts below. +* **`runbooks/vpn-remote-access.md`** — a triage section for the reported symptom ("LAN and cluster + work, internet does not") that works outwards from the client profile, and a short section saying + the L2 lease is a dead end, with the unifi `.21` note. + +**Nothing in `platform/vpn-ingress` or `infra/vpn-ingress` changed behaviourally. No code fix is +proposed by this branch any more.** + +### The internet-routing report: no server-side fix was needed, and none was made + +The brief's prior was that `new-peer.sh` renders a split tunnel and that the operator wants a full +one, to be added as a documented option. **That prior does not match the tree.** `new-peer.sh` has +emitted `AllowedIPs = 0.0.0.0/0` **by default since the component's first commit** (`f92707f`, +2026-10-05, confirmed with `git log -S`); `--split` is the opt-in, and +`runbooks/vpn-remote-access.md:252-253` already documents both forms. So there is no full-tunnel +feature to add, and adding a `WG_FULL_TUNNEL` knob would be a second switch for a default that +already exists. **I deliberately shipped no code change here** — writing one would have meant +inventing a fix for a symptom I could not reproduce (see below). + +The server side was measured end to end and is capable and correct, so the cause is in the client +profile or the client. The triage section names the two candidates that fit the symptom exactly — +a `--split` profile, and the IPv4-only `AllowedIPs` letting a dual-stack client prefer AAAA and +leak public traffic outside the tunnel — each with a one-line test that settles it. ## What I verified -Measured live against prod, `--context k3s-prod`, on 2026-10-08: +Measured live against prod, `kubectl --context k3s-prod --request-timeout=15s`, read-only, on +2026-10-17: -* **The whole tunnel is healthy**, which is what made this hard: `wg show` on `k3s-prod-1` shows the - peer with a real endpoint and bidirectional transfer (346 KiB in / 200 KiB out); - `net.ipv4.ip_forward=1`; `wg0` up at `10.90.0.1/24` MTU 1420; `iptables -t nat -S POSTROUTING` - carries the `MASQUERADE` rule with 26 packets; `FORWARD -i/-o wg0` ACCEPT with 108 / 171 packets; - DaemonSet generation 2, 3/3 ready on the content-hashed `wg-peers-m9kcckcb5t`. -* **The ARP failure and its symmetry.** On `k3s-prod-1`: `ip neigh show 192.168.2.20` → `FAILED`, - while `.21`/`.23` resolve to prod-2's MAC `04:0e:3c:96:b0:94`. On prod-2/prod-3, `.20` resolves to - prod-1's `bc:e9:2f:7e:c4:a6`. TCP agrees: `.20:443` and `.20:80` FAIL from prod-1, OK from the - other two. Repeated across 3 nodes × 5 VIPs — each node reached every VIP except the ones it alone - announced (prod-2 ✗ `.22`/`.21`; prod-3 ✗ `.24`). -* **Not an artefact of probing from the host netns.** `curl --interface wg0` from prod-1 fails the - same way, and a neighbour entry is node-global state, so the forwarded path cannot differ in ARP - resolution. -* **The Gateway itself was never broken:** from the workstation (192.168.2.133) - `home.keithjackson.dev` → `200`, `plex.home.keithjackson.dev` → `401`, both at `192.168.2.20`. - From prod-1 the router (`.1`) and TrueNAS (`.30`) are `REACHABLE`. -* **`NotIn` semantics, empirically** — the basis for the fail-back claim in the policy's comment: - `kubectl get nodes -l 'kubernetes.io/hostname notin (k3s-prod-1)'` returns prod-2 and prod-3, and - `NotIn` on a key no node carries returns **all three**. So an unapplied label excludes nothing and - degrades to the old behaviour rather than taking every VIP off the LAN. -* **The alert metric is really scraped:** `kube_lease_owner` is served by kube-state-metrics (55 - series, every `cilium-l2announce-*` lease with its holder in `lease_holder`), and `kube_node_labels` - already carries the two `egress-*` labels through the same allowlist (`k3s-prod-3` shows - `egress-gateway=true`), so the third entry follows a proven path. -* **The four promtool cases pass** (`platform/monitoring/tests/run.sh` → `SUCCESS`, 55 rules - checked), including that `VpnIngressNodeAnnouncesL2VIP` is inert when the label is absent. -* All edited YAML parses; the rendered `nodeSelector` is the intended two-expression list. +* **The lease distribution.** All eight `cilium-l2announce-*` leases held by `k3s-prod-1` + (`.spec.holderIdentity`). The brief's observation reproduced; it had not moved again. +* **The falsification, in full**, as tabulated above: 3 nodes x 5 VIPs, `ip neigh` + `nc -z` + a + real `curl` for each. The holder reaches four of five VIPs; `.21` fails from everywhere. +* **The socket-LB discriminator.** Real service port OK, bogus port FAIL, identically on all three + nodes — so the successes are service-aware DNAT, not wire delivery. +* **`curl --interface wg0` fails on all three nodes**, including non-announcing ones with + `REACHABLE` neighbour entries. This is the measurement that destroys the original control. +* **The datapath facts** quoted above from `cilium-dbg status --verbose` and + `cilium-dbg service list` on `cilium-55jv8` (`k3s-prod-1`), plus `cilium-config`: + `kube-proxy-replacement=true`, `enable-l2-announcements=true`, `enable-bpf-masquerade=true`, + `routing-mode=tunnel`, `devices=` (empty, auto-detected). Cilium `v1.20.1` on all three. +* **The VIP is not a local address and routes on-link.** `ip route get 192.168.2.20` → + `dev eno1 src 192.168.2.10` on the holder; nothing in `table local`; `ip -4 addr` shows no VIP. + So the 200 is not loopback delivery by the host stack — it is BPF. +* **The internet path, server side, on all three nodes:** `/proc/sys/net/ipv4/ip_forward` = `1` + (read, never written); `iptables -t nat -S POSTROUTING` carries + `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE`; the preceding `CILIUM_POST_nat` chain is **empty with 0 + packets**, so it cannot short-circuit that rule; `FORWARD -i wg0` and `-o wg0` ACCEPT both + present; `default via 192.168.2.1 dev eno1 onlink` intact; `ip route get 1.1.1.1` resolves via it; + `curl http://1.1.1.1/` returns `301` from each node. +* **The PIA / egress-gateway answer.** `platform/vpn-egress/egress-steering.yaml` selects by + `podSelector` on `egress-class: vpn`; `platform/vpn-ingress/daemonset.yaml:130` labels this pod + `egress-class: internet`; `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`), absent on + `-1` and `-2`. So a full tunnel egresses the **house WAN**, and a client's apparent address is the + home WAN IP, not PIA's. +* **PiHole resolves public names** — `nslookup google.com 192.168.2.23` from the pod returned + `142.250.113.138`. +* **No client is connected, and none has been since the power-cycle.** `wg show wg0` lists the one + enrolled peer (`10.90.0.2/32`) with **no endpoint and no handshake** on all three nodes, and both + `FORWARD -i wg0` and the `MASQUERADE` rule read **0 packets / 0 bytes** on all three. The + `wireguard` pods restarted 153 min ago (2 restarts, 11d old), which reset the counters. +* **The full-tunnel default's age**, from `git log -S` on `new-peer.sh`: introduced in `f92707f` + (2026-10-05) and never changed. +* **The branch rebased cleanly** onto `origin/main` (`1a0b208`, PR #70 merged) with no conflicts. ## What I did NOT verify -* **The fix is UNAPPLIED and therefore UNPROVEN end to end.** Applying it needs a mutating - `kubectl`/`flux reconcile` plus an Ansible run, which are operator actions (CLAUDE.md §5). I have - **not** observed `k3s-prod-1` resolve `192.168.2.20` after the exclusion lands, nor a VPN client - reaching an app. The prediction rests on the measured symmetry (non-announcing nodes resolve every - VIP, 15/15 data points), not on a test of the changed state. **Verify after merge, in this order:** - re-run `node_baseline --tags vpn-ingress`; `kubectl get nodes -L - homelab.keithjackson.dev/vpn-ingress` (exactly one node); `kubectl -n kube-system get lease | grep - l2announce` (no holder is that node); `ip neigh show 192.168.2.20` from the prod-1 wireguard pod - (expect a `lladdr`, not `FAILED`); then a client. -* **That Cilium actually releases a lease when its holder stops matching the `nodeSelector`.** I - assumed re-election and did not test it. If Cilium keeps renewing, the affected Leases must be - deleted once by hand to force an election. Check this first if the fix appears not to work. -* **The brief interruption on apply.** Moving the Gateway and `pihole-dns-udp` leases off prod-1 will - blip `192.168.2.20` and `192.168.2.23` for as long as re-election plus gratuitous ARP takes. Not - measured, and I did not test whether in-flight LAN sessions survive it. -* **The Ansible tasks have never been executed.** They parse as YAML and follow - `roles/vpn_egress_gateway`'s shape, but neither the label task nor the label-removal task has run, - so the `changed_when`/`failed_when` conditions and the `k3s kubectl label ... -` removal path are - unproven. The removal task's `failed_when` tolerates `'not found'` in stderr; that string was - assumed from kubectl's behaviour, not observed. -* **The residual off-repo coupling is NOT closed and cannot be, as built.** The router's forward is - unreadable from the cluster. If someone re-points it without moving the label, the alert watches - the wrong node and stays silent while the VPN breaks identically against a different set of VIPs. - Only the runbook guards it. Closing it needs a WireGuard exporter publishing which node is - actually terminating peer traffic (`wg show` knows — only the forwarded node has a peer - `endpoint`); listed as "Not built" in the component README and **recommended follow-up, not done - here**. -* **Why Cilium behaves this way** — intended, a known upstream issue, or fixed in a later version. I - measured the behaviour and did not read Cilium's source or issue tracker, so I cannot say whether - an upgrade would remove the need for this exclusion. -* **`192.168.2.21:8443` (unifi) fails from all three nodes**, including ones where ARP resolves, so - it is a different, pre-existing fault. Not investigated, not addressed. -* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 offline - subset only; none of the above is covered by a test that runs in the gate. +* **THE FORWARDED PATH IS STILL UNMEASURED, in both directions.** No client is connected, so I could + not test what a real remote client does, and — as established above — **no probe runnable from the + pod can substitute for it**. Specifically I did **not** prove that a forwarded client packet to a + VIP is DNAT'd by Cilium's `wg0` program. That inference rests on `cilium-dbg` listing `wg0` as a + LB device plus the node-local Gateway backend, not on an observed client packet. **It is the one + remaining way the withdrawn diagnosis could be partly right** — if `wg0` BPF were somehow not in + the forwarded path, forwarded traffic would need ARP and would die on the holder. I judge this + unlikely but it is unproven. *To close it: connect a client, then from the node whose `wg show` + shows the peer endpoint, watch `FORWARD -i wg0` and the `MASQUERADE` counters move while the + client loads a cluster app — and repeat it on a node that holds leases and one that does not.* +* **WHICH NODE THE ROUTER FORWARDS `udp/51820` TO IS UNDETERMINABLE RIGHT NOW**, for the same + reason. **I need a client connected** to learn it; the router is unreadable from the cluster. +* **I cannot reproduce or explain the 2026-10-08 measurements.** That entry recorded `nc -z` TCP + failures correlated with lease holding across 15 node x VIP pairs. Today the same test passes. + My leading hypothesis is **stale or incorrectly-programmed BPF LB state on `k3s-prod-1`**, + rebuilt by today's full power-cycle — which would make the lease correlation coincidental. A + competing hypothesis is that Cilium had not auto-detected `wg0` then (`devices` is empty, and + `wg0` is created by this DaemonSet after Cilium starts). **Both are unproven, and neither is + needed for the falsification** — the asserted cause is present today without its effect. I did not + retro-measure anything, and I did not read Cilium's source or issue tracker. +* **The operator's report cannot be a description of "right now."** No client has connected since + the power-cycle (0 forwarded packets, no handshake on any node), so "clients can reach cluster + services and the LAN but not the internet" must describe a session before the power-cycle or one + that has since dropped. **I did not establish when it was observed, or with which profile.** +* **THE INTERNET ROOT CAUSE IS NOT DIAGNOSED.** I proved the server side is *capable*; I did **not** + find the fault, and I shipped no fix for it. The two candidates in the triage section (a + `--split` profile; IPv6 preference leaking outside an IPv4-only `AllowedIPs`) are **reasoned, not + measured**. *What I need:* the `AllowedIPs` line from the client's own profile — that one line + decides split vs full and halves the search — and a connected client for step 2. +* **Whether `--split` was in fact used for the enrolled peer.** The profile holds a private key and + lives only on the operator's machine and the device; it is not in the repo, and I did not look for + it. Server-side `AllowedIPs` is a `/32` source filter and says nothing about the client's routing. +* **Whether `MTU = 1420` holds on a full-tunnel WAN path.** Not measured, and not measurable without + a client. It is the standard 1500 − 60; a client behind PPPoE or CGNAT with a smaller path MTU + would see large transfers hang while small requests succeed. The runbook's existing MTU note + (drop to 1280) already covers the remedy. +* **Adding `::/0` to `AllowedIPs` is untested advice.** The triage section offers it to blackhole + IPv6 rather than leak it, and states that the tunnel has no IPv6 egress. I did not try it on a + client, and I do not know how each WireGuard client behaves when handed an IPv6 route on an + interface with no IPv6 address. +* **`192.168.2.21:8443` (unifi) fails from all three nodes**, as it did on 2026-10-08 — a separate, + pre-existing fault. Not investigated, not addressed, and now recorded in the runbook as such. +* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so the gate proves the L0/L1 + offline subset only. Nothing above is covered by a test that runs in CI. +* **Nothing was applied, and nothing needs to be.** The branch is now documentation-only, so there + is no Flux reconcile, Ansible run or `kubectl` step to perform after merge. I ran no mutating + `kubectl`, `flux` or Ansible, and did not run `new-peer.sh` (it mints key material). * I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b). diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index d286c9d..2b865a6 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -420,22 +420,3 @@ flux_bootstrap_env_path: "clusters/prod" # rather than leaving an inert drop-in behind, so this group-wide `true` is still correct # for it. etcd_io_priority_enabled: true - -# --- The node terminating remote-access WireGuard — DECIDED (measured 2026-10-08) --------------- -# k3s-prod-1 (192.168.2.10) is what the SOHO router forwards WAN udp/51820 to, confirmed from the -# cluster rather than assumed: only prod-1's `wg show` had a peer with an `endpoint` and a recent -# handshake, the other two listeners were idle. -# -# node_baseline labels it `homelab.keithjackson.dev/vpn-ingress=true`, and -# platform-config/cilium/l2-announcement-policy.yaml excludes that label from announcing any -# LoadBalancer VIP -- because a node CANNOT reach a VIP whose Cilium L2 announcement lease it holds -# itself (the announcer does not answer its own ARP request; the neighbour entry goes to `FAILED`). -# prod-1 held the `shared` Gateway lease, and since vpn-ingress MASQUERADEs clients to the node's -# own address, all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved. -# -# [parity] MUST equal the node named by the router's forward (runbooks/vpn-remote-access.md step 3). -# Nothing in CI can read that forward, so re-pointing it means changing BOTH: this value plus a -# node_baseline re-run, and the router line. `VpnIngressNodeAnnouncesL2VIP` catches the case where -# the label lands on a node that then takes a lease; it CANNOT catch the forward being moved -# without this value following, which is why the runbook spells the two-step out. -node_baseline_vpn_ingress_node: k3s-prod-1 diff --git a/infra/ansible/roles/node_baseline/defaults/main.yml b/infra/ansible/roles/node_baseline/defaults/main.yml index ae7a493..b6a7f4b 100644 --- a/infra/ansible/roles/node_baseline/defaults/main.yml +++ b/infra/ansible/roles/node_baseline/defaults/main.yml @@ -259,30 +259,3 @@ node_baseline_dns_options: # Written as ONE marked block, so removing an entry here removes it from the node, and # an empty list removes the block entirely. node_baseline_hosts_pins: [] - -# --- The node that terminates remote-access WireGuard (tags: vpn-ingress) ---------------------- -# Hostname of the node the SOHO router forwards WAN udp/51820 to, i.e. the one platform/vpn-ingress -# actually serves remote clients from. Set it in group_vars for the environment; EMPTY by default, -# because dev has no such forward and because a wrong value is worse than none. -# -# WHAT IT IS FOR, AND WHY IT IS NOT COSMETIC. The matching node gets -# `homelab.keithjackson.dev/vpn-ingress=true`, and -# platform-config/cilium/l2-announcement-policy.yaml EXCLUDES that label from announcing any -# LoadBalancer VIP. MEASURED on prod 2026-10-08: a node cannot reach a VIP whose Cilium L2 -# announcement lease it holds itself (the announcer does not answer its own ARP request -- the -# neighbour entry goes to `FAILED`). Because vpn-ingress runs on `hostNetwork` and MASQUERADEs -# clients to the node's own address, a VPN client inherits that: with the forward on the node -# holding the `shared` Gateway lease, all 16 Gateway hostnames were dead over the VPN while PiHole -# DNS still resolved. Names worked, every app hung, nothing alerted. -# -# [parity] MUST name the node the router's forward actually points at. The forward is off-repo and -# no check can read it (runbooks/vpn-remote-access.md step 3). Changing the forward is therefore a -# two-part change: this value (then re-run the role), and the router line. -# -# Only ONE node should ever carry the label: it removes that node from VIP announcement candidacy, -# so labelling all three would leave nothing announcing any VIP at all. The task below removes the -# label from a node that no longer matches, so moving the value moves the label. -node_baseline_vpn_ingress_node: "" -node_baseline_vpn_ingress_label_key: homelab.keithjackson.dev/vpn-ingress -# k3s ships its kubeconfig here; every prod node is a k3s server, so `k3s kubectl` works locally. -node_baseline_vpn_ingress_kubeconfig: /etc/rancher/k3s/k3s.yaml diff --git a/infra/ansible/roles/node_baseline/tasks/main.yml b/infra/ansible/roles/node_baseline/tasks/main.yml index de317f4..1422573 100644 --- a/infra/ansible/roles/node_baseline/tasks/main.yml +++ b/infra/ansible/roles/node_baseline/tasks/main.yml @@ -1109,45 +1109,3 @@ state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}" backup: true tags: [dns, hosts] - -# ── vpn-ingress node label — tag `vpn-ingress` ──────────────────────────────────────────────── -# Marks the node terminating remote-access WireGuard so Cilium's L2 announcement policy can EXCLUDE -# it from announcing LoadBalancer VIPs (a node cannot reach a VIP it announces itself -- see -# defaults/main.yml). Same shape as roles/vpn_egress_gateway's egress-gateway label, and for the -# same reason: the label is the one place BOTH a nodeSelector and a PromQL alert can read it from. -# -# Skipped entirely when node_baseline_vpn_ingress_node is empty (dev, or before the forward exists), -# so this adds nothing to an environment that has no WAN forward. -- name: Label this node as the vpn-ingress node - ansible.builtin.command: - cmd: >- - k3s kubectl label node {{ inventory_hostname }} - {{ node_baseline_vpn_ingress_label_key }}=true --overwrite - environment: - KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}" - register: node_baseline_vpn_ingress_label_apply - changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_apply.stdout" - when: - - node_baseline_vpn_ingress_node | length > 0 - - inventory_hostname == node_baseline_vpn_ingress_node - tags: [vpn-ingress] - -# The other half: moving the knob must MOVE the label, not add a second one. Two labelled nodes -# would remove two of three nodes from announcement candidacy; three would leave no announcer at -# all and take every VIP off the LAN. -- name: Remove the vpn-ingress label from a node that no longer holds the forward - ansible.builtin.command: - cmd: >- - k3s kubectl label node {{ inventory_hostname }} - {{ node_baseline_vpn_ingress_label_key }}- - environment: - KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}" - register: node_baseline_vpn_ingress_label_remove - changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_remove.stdout" - failed_when: - - node_baseline_vpn_ingress_label_remove.rc != 0 - - "'not found' not in node_baseline_vpn_ingress_label_remove.stderr" - when: - - node_baseline_vpn_ingress_node | length > 0 - - inventory_hostname != node_baseline_vpn_ingress_node - tags: [vpn-ingress] diff --git a/platform-config/cilium/l2-announcement-policy.yaml b/platform-config/cilium/l2-announcement-policy.yaml index c9229f5..d65e0ff 100644 --- a/platform-config/cilium/l2-announcement-policy.yaml +++ b/platform-config/cilium/l2-announcement-policy.yaml @@ -16,71 +16,6 @@ spec: externalIPs: false interfaces: - ^e(n|th) - # =============================================================================================== - # WHY THIS IS A matchExpressions LIST AND NOT `matchLabels: {kubernetes.io/os: linux}` - # =============================================================================================== - # The second expression excludes every node labelled `homelab.keithjackson.dev/vpn-ingress=true` - # from announcing ANY LoadBalancer VIP. That is the node the SOHO router forwards WAN udp/51820 to. - # It is not tuning; it is the fix for a measured, silent, total failure of remote access. - # - # MEASURED ON PROD 2026-10-08: A NODE CANNOT REACH A VIP IT ANNOUNCES ITSELF. Cilium's L2 - # announcer answers ARP for the VIP on behalf of OTHER LAN hosts, but not for the announcing node - # itself. On the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP - # is dropped before it leaves the node: - # - # k3s-prod-1, holding cilium-l2announce-gateway-cilium-gateway-shared: - # ip neigh show 192.168.2.20 -> 192.168.2.20 dev eno1 FAILED (its own VIP) - # nc -z 192.168.2.20 443 -> FAIL nc -z 192.168.2.20 80 -> FAIL - # k3s-prod-2 and k3s-prod-3, holding no lease for it: - # ip neigh show 192.168.2.20 -> lladdr bc:e9:2f:7e:c4:a6 (prod-1) nc 443/80 -> OK - # - # Symmetric across all 3 nodes x 5 VIPs: every node reached every VIP EXCEPT the ones it alone - # announced (prod-2 could not reach crafty 192.168.2.22 or unifi 192.168.2.21; prod-3 could not - # reach plex 192.168.2.24). So this is a property of L2 announcement, not of one Service. - # - # WHY IT ONLY HURTS THE VPN NODE. For an ordinary workload this is harmless: pods reach Services - # through the cluster datapath and never ARP for a VIP. platform/vpn-ingress is the exception -- - # it runs WireGuard with `hostNetwork: true` and MASQUERADEs remote clients to the node's own - # address, so a VPN client's packets take the node's LAN path and inherit exactly this ARP - # failure. On 2026-10-08 the forward pointed at k3s-prod-1, which held the Gateway lease, and all - # 16 Gateway hostnames were unreachable over the VPN while PiHole DNS still answered -- so names - # resolved and every app hung. Nothing in the cluster reported it; the alerts below now do. - # - # WHY A NODE LABEL AND NOT A cluster-vars KNOB. A knob would put the node's identity in git, which - # is more reviewable -- but it CANNOT BE USED BY THE ALERT that guards this, and an unguarded - # invariant here is worth less than a reviewable one. The alert harness - # (platform/monitoring/tests/extract_rules.py) REFUSES any alert whose `expr` contains a - # dollar-brace postBuild placeholder (spelled without braces here ON PURPOSE: this file is itself - # envsubst'd, and a literal dollar-brace-dot-dot-dot is UNPARSEABLE to envsubst, which fails the - # WHOLE Kustomization -- ci: flux-postbuild-substitution caught exactly this comment), because the - # rule it would unit-test is not the rule that gets - # deployed. A label is readable from BOTH sides: this selector matches on it, and - # `kube_node_labels` carries it into Prometheus so `VpnIngressNodeAnnouncesL2VIP` can join lease - # holders against it. One source of truth, no git-vs-cluster skew, and the policy stays - # env-agnostic as its header above claims. The label is set by - # infra/ansible/roles/node_baseline (node_baseline_vpn_ingress_node), the same shape - # roles/vpn_egress_gateway already uses for `egress-gateway`. - # - # THE COST, STATED: VIP announcement candidacy drops from 3 nodes to 2. That is still redundant, - # and the trade is deliberate -- the excluded node is the one whose *outbound* reach to every VIP - # matters most, because it carries every remote client. If BOTH remaining nodes are down the VIPs - # go dark where previously one node might have served them, but at that point the cluster has one - # node left and far worse problems. - # - # `NotIn` with an ABSENT key MATCHES (k8s apimachinery: a missing label is "not in the set"), so - # the other two nodes need no label and nothing has to be applied to them. If the label is never - # applied anywhere, this excludes nothing and the policy degrades to its pre-2026-10-08 behaviour - # -- the status quo, not an outage: it fails back, not down. That silent-degradation path is - # exactly why `VpnIngressNodeLabelMissing` exists as a watchdog: without it, the alert below - # would be structurally INERT (its join would match nothing) and would look identical to healthy. nodeSelector: - matchExpressions: - - key: kubernetes.io/os - operator: In - values: - - linux - # The node terminating remote-access WireGuard must announce nothing -- see above. - - key: homelab.keithjackson.dev/vpn-ingress - operator: NotIn - values: - - "true" + matchLabels: + kubernetes.io/os: linux diff --git a/platform/monitoring/alerts.yaml b/platform/monitoring/alerts.yaml index ef18519..e007f2d 100644 --- a/platform/monitoring/alerts.yaml +++ b/platform/monitoring/alerts.yaml @@ -1147,97 +1147,6 @@ spec: metricLabelsAllowlist. runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale" # ----------------------------------------------------------------------------------------- - # REMOTE-ACCESS VPN (platform/vpn-ingress) -- NOT the egress tunnel above. Different - # direction, different workload: this is inbound WireGuard for remote devices. - # ----------------------------------------------------------------------------------------- - - alert: VpnIngressNodeAnnouncesL2VIP - # THE FAILURE THIS CATCHES. MEASURED on prod 2026-10-08: a node CANNOT reach a - # LoadBalancer VIP whose Cilium L2 announcement lease it holds itself -- the announcer - # does not answer its own ARP request, so the neighbour entry settles at `FAILED` and - # packets die before leaving the node (`ip neigh show 192.168.2.20` -> FAILED on the - # lease holder, resolved on both others). platform/vpn-ingress runs WireGuard on - # `hostNetwork` and MASQUERADEs remote clients to the node's own address, so if the node - # the router forwards udp/51820 to is also a lease holder, VPN clients cannot reach the - # VIPs it announces. On 2026-10-08 that was the `shared` Gateway: all 16 app hostnames - # dead over the VPN, while PiHole DNS kept answering (its VIP has two Services and the - # other lease sat on prod-2) -- so the tunnel handshook, names resolved, and every app - # hung. Nothing reported it for ~2 days. - # - # platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled - # `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so this should be - # permanently unsatisfiable. The alert exists because the exclusion has ways to stop - # working that no static check on that file's TEXT can see: the label being removed or - # moved to the wrong node, a Cilium upgrade changing `NotIn` semantics or the - # nodeSelector field, and the selector silently failing to match. This watches the - # OUTCOME -- who actually holds the lease -- so all of them surface the same way. - # - # WHY THE LABEL JOIN RATHER THAN A NODE NAME. The node's identity is NOT written into - # this expr, because the harness (tests/extract_rules.py) refuses any alert whose expr - # contains a dollar-brace placeholder -- spelled in words because a literal one here is - # unparseable to envsubst and would fail this whole Kustomization. Rightly so: the rule it - # unit-tests would not be the - # rule deployed. Joining `kube_lease_owner` against `kube_node_labels` keeps ONE source - # of truth (the label itself, which is also what the policy selects on), so this alert - # cannot drift out of agreement with the exclusion it guards. `label_replace` is needed - # because kube_lease_owner names the holder `lease_holder` while kube_node_labels names - # it `node`; the join is then on a shared `node` label. The label is in - # kube-state-metrics' metricLabelsAllowlist (kube-prometheus-stack.yaml) -- without that - # line this rule is inert, which `VpnIngressNodeLabelMissing` below exists to catch. - # - # `for: 15m` because leases legitimately move during a node reboot or a Cilium roll and - # Cilium re-elects within seconds; a real collision persists, since the lease holder - # keeps renewing and it never self-heals. - expr: | - max by (lease) ( - label_replace( - kube_lease_owner{namespace="kube-system", lease=~"cilium-l2announce-.*"}, - "node", "$1", "lease_holder", "(.+)" - ) - * on (node) group_left () - kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"} - ) == 1 - for: 15m - labels: - severity: critical - annotations: - summary: "VPN ingress node is announcing a LoadBalancer VIP ({{ $labels.lease }})" - description: >- - The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access - WireGuard, and it now holds the Cilium L2 announcement lease `{{ $labels.lease }}`. A - node cannot reach a VIP it announces itself, so every remote VPN client has just lost - all access to that VIP's Service -- silently, with DNS still resolving. Check that - platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that - the label is still on the node the router forwards udp/51820 to. - runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" - - alert: VpnIngressNodeLabelMissing - # THE WATCHDOG for the rule above, and the reason it is not optional. The exclusion in - # l2-announcement-policy.yaml is a `NotIn` on this label, and `NotIn` with an ABSENT key - # MATCHES -- so if the label is on NO node, the policy excludes nothing, every node is an - # announcement candidate again, and the 2026-10-08 failure is fully re-armed. The alert - # above would not fire either: its join would match nothing and return no series. Present, - # evaluating, and incapable of firing -- the exact "inert critical alert" shape that - # VPNEgressUplinkLabelMissing exists for on the egress side, and that this repo has been - # burned by before. - # - # The label is applied by infra/ansible/roles/node_baseline - # (node_baseline_vpn_ingress_node); the most likely cause of it going absent is a node - # rebuild where that role has not been re-run. - expr: absent(kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"}) - for: 30m - labels: - severity: warning - annotations: - summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF" - description: >- - No node publishes homelab.keithjackson.dev/vpn-ingress=true, so - l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none: - any node may now announce a LoadBalancer VIP, including the one terminating - remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so - the warning matters more than its severity suggests. Re-run the node_baseline role on - the forwarded node, and confirm the label is in kube-state-metrics' - metricLabelsAllowlist. - runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" - # ----------------------------------------------------------------------------------------- # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown` diff --git a/platform/monitoring/kube-prometheus-stack.yaml b/platform/monitoring/kube-prometheus-stack.yaml index 0daf563..1f75c1c 100644 --- a/platform/monitoring/kube-prometheus-stack.yaml +++ b/platform/monitoring/kube-prometheus-stack.yaml @@ -281,20 +281,8 @@ spec: # without anyone having to remember to re-enable it later. Scoped to one label on one resource: # 3 extra series, no cardinality concern. Label name is sanitized by KSM to # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`). - # `vpn-ingress` (added 2026-10-17) marks the node the SOHO router forwards WAN udp/51820 to. - # platform-config/cilium/l2-announcement-policy.yaml excludes that node from announcing any - # LoadBalancer VIP, because a node cannot reach a VIP whose L2 announcement lease it holds - # itself and vpn-ingress MASQUERADEs remote clients to the node's own address (measured - # 2026-10-08: all 16 Gateway hostnames dead over the VPN while DNS still resolved). It is - # allowlisted HERE because that is the only way the invariant can be WATCHED: - # `VpnIngressNodeAnnouncesL2VIP` joins `kube_lease_owner` against this label, and the alert - # unit-test harness refuses a dollar-brace placeholder in an alert expr (written out in words: - # a literal one here is unparseable to envsubst and fails the whole Kustomization), so a knob - # could not have been used. Without this line BOTH that alert and its - # `VpnIngressNodeLabelMissing` watchdog are structurally inert. Same reasoning, and the same - # 3-extra-series cost, as the two egress labels below. metricLabelsAllowlist: - - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink,homelab.keithjackson.dev/vpn-ingress]" + - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]" # --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) --------------------------------- # Velero exports NOTHING that enumerates Schedule objects and nothing that exposes # `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that diff --git a/platform/monitoring/tests/vpn-ingress-l2.test.yaml b/platform/monitoring/tests/vpn-ingress-l2.test.yaml deleted file mode 100644 index 4033449..0000000 --- a/platform/monitoring/tests/vpn-ingress-l2.test.yaml +++ /dev/null @@ -1,203 +0,0 @@ -# promtool rule-unit tests for VpnIngressNodeAnnouncesL2VIP + VpnIngressNodeLabelMissing -# (platform/monitoring/alerts.yaml) -# -# Run with: platform/monitoring/tests/run.sh -# (the merge gate runs the same harness via -# conformance/tests/l0-static/alert-rules-unit-tested) -# -# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS -# MEASURED on prod 2026-10-08: a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement -# lease it holds ITSELF -- the announcer does not answer its own ARP request, so on the lease holder -# the neighbour entry settles at `FAILED` and packets to that VIP die before leaving the node. -# platform/vpn-ingress runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's -# own address, so a VPN client takes that node's LAN path and inherits the failure. With the -# router's udp/51820 forward on k3s-prod-1, which held the `shared` Gateway lease, all 16 Gateway -# hostnames were dead over the VPN -- while PiHole DNS kept answering, because its VIP has two -# Services and k3s-prod-2 held the other lease. The tunnel handshook, names resolved, every app -# hung, and nothing reported it for ~2 days. -# -# platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled -# `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so VpnIngressNodeAnnouncesL2VIP -# should be permanently unsatisfiable. THAT IS EXACTLY WHY IT NEEDS A TEST: an alert expected never -# to fire is indistinguishable, in production, from an alert that CANNOT fire. Nothing would ever -# reveal a mistyped label, a broken `label_replace`, or a join on the wrong label. Test 1 proves it -# fires on the real failure; test 2 proves it is quiet in the fixed state, so it is not a rule that -# fires on anything; test 3 pins the `for: 15m` against routine lease churn. -# -# THE JOIN IS THE FRAGILE PART AND IS TESTED DELIBERATELY. `kube_lease_owner` names the holder in -# `lease_holder`; `kube_node_labels` names it `node`. The rule bridges them with `label_replace` -# and joins `on (node) group_left ()`. Fixtures therefore carry BOTH series shapes with real label -# sets, because a join that silently matches nothing returns no series -- which looks exactly like -# "healthy" and is the failure mode this whole group exists to prevent. -# -# Fixture values are prod's real shape (measured 2026-10-08): leases -# `cilium-l2announce-gateway-cilium-gateway-shared` (k3s-prod-1) and -# `cilium-l2announce-pihole-pihole-dns-tcp` (k3s-prod-2), in namespace kube-system, with -# k3s-prod-1 the vpn-ingress node. - -rule_files: - - generated.rules.yaml - -evaluation_interval: 1m - -tests: - # ============================================================================================= - # 1. THE BUG, EXACTLY AS IT HAPPENED -- the vpn-ingress node holds the Gateway lease: FIRES. - # ============================================================================================= - - interval: 1m - name: vpn-ingress node holds the shared Gateway L2 lease - VpnIngressNodeAnnouncesL2VIP fires - input_series: - # The collision. A lease holder keeps renewing, so this series does not lapse on its own -- - # which is why the condition never self-heals and why a 15m `for:` is safe. - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}' - values: '1+0x40' - # A second lease held by a DIFFERENT node, present throughout. This is the discriminator that - # stops the rule from being "any l2announce lease exists anywhere": if the join were dropped, - # this series alone would fire the alert, and test 2 would fail. - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-pihole-pihole-dns-tcp",lease_holder="k3s-prod-2"}' - values: '1+0x40' - # k3s-prod-1 is the vpn-ingress node. Only this node carries the label. - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}' - values: '1+0x40' - # The other two nodes publish kube_node_labels WITHOUT the vpn-ingress label. Present so the - # join is exercised against non-matching nodes rather than against nothing at all. - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}' - values: '1+0x40' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}' - values: '1+0x40' - alert_rule_test: - # Held 15m, so nothing before then. - - eval_time: 14m - alertname: VpnIngressNodeAnnouncesL2VIP - exp_alerts: [] - - eval_time: 20m - alertname: VpnIngressNodeAnnouncesL2VIP - exp_alerts: - # EXACTLY ONE alert: the pihole lease on prod-2 must NOT produce a second one. - - exp_labels: - severity: critical - lease: cilium-l2announce-gateway-cilium-gateway-shared - exp_annotations: - summary: "VPN ingress node is announcing a LoadBalancer VIP (cilium-l2announce-gateway-cilium-gateway-shared)" - description: >- - The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access - WireGuard, and it now holds the Cilium L2 announcement lease `cilium-l2announce-gateway-cilium-gateway-shared`. A - node cannot reach a VIP it announces itself, so every remote VPN client has just lost - all access to that VIP's Service -- silently, with DNS still resolving. Check that - platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that - the label is still on the node the router forwards udp/51820 to. - runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" - # The label IS present, so the watchdog must stay quiet here. - - eval_time: 40m - alertname: VpnIngressNodeLabelMissing - exp_alerts: [] - - # ============================================================================================= - # 2. THE FIXED STATE -- every lease is held by a node that is NOT the vpn-ingress node: QUIET. - # This is what l2-announcement-policy.yaml's exclusion produces, and the assertion that keeps - # test 1 meaningful rather than an alert that fires on anything. - # ============================================================================================= - - interval: 1m - name: all l2 leases held by other nodes - VpnIngressNodeAnnouncesL2VIP stays quiet - input_series: - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-2"}' - values: '1+0x40' - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-plex-plex-l4-tcp",lease_holder="k3s-prod-3"}' - values: '1+0x40' - # A lease the vpn-ingress node DOES legitimately hold, in a different family -- proves the - # `cilium-l2announce-.*` matcher is doing work. Every node holds operator/controller leases; - # only the ANNOUNCEMENT ones matter here, and a matcher broad enough to catch this would page - # on a healthy cluster forever. - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-operator-resource-lock",lease_holder="k3s-prod-1"}' - values: '1+0x40' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}' - values: '1+0x40' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}' - values: '1+0x40' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}' - values: '1+0x40' - alert_rule_test: - - eval_time: 40m - alertname: VpnIngressNodeAnnouncesL2VIP - exp_alerts: [] - - eval_time: 40m - alertname: VpnIngressNodeLabelMissing - exp_alerts: [] - - # ============================================================================================= - # 3. A LEGITIMATE, TRANSIENT LEASE MOVE -- the vpn-ingress node holds it for 11m during a node - # reboot or a Cilium roll, then Cilium re-elects away: QUIET. This pins the `for: 15m` as - # deliberate. Without it, every rolling restart of the cilium DaemonSet would page, and a paging - # alert that cries wolf on routine maintenance gets muted -- which is how the real collision - # would go unseen again. - # ============================================================================================= - - interval: 1m - name: transient lease move shorter than for 15m - VpnIngressNodeAnnouncesL2VIP stays quiet - input_series: - # Held by the vpn-ingress node for 11 samples, then the series goes absent (the lease moved, - # so kube-state-metrics retires this label set and starts a new one). - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}' - values: '1+0x10 _x30' - # Re-elected onto prod-3 from t=11m. A label VALUE change retires one series and starts - # another, which is why this is a second series and not a changing value. - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-3"}' - values: '_x11 1+0x29' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}' - values: '1+0x40' - alert_rule_test: - # Inside the transient window: pending at most, never firing. - - eval_time: 10m - alertname: VpnIngressNodeAnnouncesL2VIP - exp_alerts: [] - # Well past it, with the lease settled on another node. - - eval_time: 30m - alertname: VpnIngressNodeAnnouncesL2VIP - exp_alerts: [] - - # ============================================================================================= - # 4. THE WATCHDOG -- no node carries the label at all. The policy's `NotIn` then excludes - # NOTHING (a missing key matches), so the 2026-10-08 failure is fully re-armed, AND - # VpnIngressNodeAnnouncesL2VIP is structurally INERT: its join matches nothing and returns no - # series, which is indistinguishable from healthy. Both halves are asserted here, because an - # inert critical alert is the exact shape this repo has been burned by before. - # ============================================================================================= - - interval: 1m - name: vpn-ingress label absent everywhere - watchdog fires and the critical alert is inert - input_series: - # A lease held by the node that IS (in reality) the forwarded one -- but with no label - # anywhere, nothing can tell. This is the dangerous state. - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}' - values: '1+0x60' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}' - values: '1+0x60' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}' - values: '1+0x60' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}' - values: '1+0x60' - alert_rule_test: - # The critical alert CANNOT fire -- this is the inertness being pinned, not an aside. - - eval_time: 45m - alertname: VpnIngressNodeAnnouncesL2VIP - exp_alerts: [] - # ... which is precisely why the watchdog must. - - eval_time: 45m - alertname: VpnIngressNodeLabelMissing - exp_alerts: - - exp_labels: - severity: warning - # absent() propagates the matcher's own labels onto the series it returns, so the - # alert inherits this one. Asserted rather than worked around: it is harmless, and - # pinning it means a future change to the matcher cannot alter the alert's identity - # (and therefore its Alertmanager grouping) unnoticed. - label_homelab_keithjackson_dev_vpn_ingress: "true" - exp_annotations: - summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF" - description: >- - No node publishes homelab.keithjackson.dev/vpn-ingress=true, so - l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none: - any node may now announce a LoadBalancer VIP, including the one terminating - remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so - the warning matters more than its severity suggests. Re-run the node_baseline role on - the forwarded node, and confirm the label is in kube-state-metrics' - metricLabelsAllowlist. - runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" diff --git a/platform/vpn-ingress/README.md b/platform/vpn-ingress/README.md index add6522..b0598be 100644 --- a/platform/vpn-ingress/README.md +++ b/platform/vpn-ingress/README.md @@ -117,19 +117,8 @@ boundary around this port is WireGuard's peer authentication, full stop. The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400 forward). A single-replica Deployment would float on every reschedule and silently invalidate it. A DaemonSet instead puts an identical listener on all three nodes — every node answers the same -keys — so losing the forwarded node never costs a re-issued client profile. The two idle listeners -cost ~15 MiB each. - -> **This section used to say "losing the forwarded node is a one-line router change, with no cluster -> work". It was wrong, and it cost two days of silently broken remote access on 2026-10-08.** The -> nodes are warm, but they are **not interchangeable**: the node the forward points at must be the -> one labelled `homelab.keithjackson.dev/vpn-ingress=true`, because that is the node -> `platform-config/cilium/l2-announcement-policy.yaml` excludes from announcing LoadBalancer VIPs. -> **A node cannot reach a VIP whose Cilium L2 announcement lease it holds itself** (see "Measured on -> prod, 2026-10-08" below), and since this pod MASQUERADEs clients to the node's own address, a VPN -> client inherits that. Failover is therefore **two** changes: move -> `node_baseline_vpn_ingress_node` and re-run that role, then re-point the router — -> `runbooks/vpn-remote-access.md` step 3. +keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no +re-issued client profiles. The two idle listeners cost ~15 MiB each. A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node @@ -197,44 +186,85 @@ right way; one was a defect in my own manifest. | Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. | | Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. | -## Measured on prod, 2026-10-08 — why the forwarded node cannot be any node +## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach -The first device enrolled, handshook, resolved names, and could not open a single connection to any -cluster web app. The tunnel was not at fault — every part of it measured healthy: -`net.ipv4.ip_forward=1`, `wg0` up at `10.90.0.1/24` MTU 1420, the peer authorised with real -bidirectional transfer (346 KiB in / 200 KiB out), `MASQUERADE` present and counting 26 flows, -`FORWARD -i/-o wg0 ACCEPT` counting 108 / 171 packets, and PiHole DNS answering from the node. +A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose +Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from +announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is +real but inert, and the distinction matters enough to record, because the wrong version is very +easy to re-derive from `ip neigh` alone. -**The fault was that a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it -holds itself.** Cilium's announcer answers ARP for the VIP on behalf of other LAN hosts, but not for -the announcing node's own request, so the neighbour entry settles at `FAILED`: +Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case, +after the UPS power-cycle — from the host netns of the `wireguard` pod on each node: -| from | `192.168.2.20` (Gateway) | `192.168.2.22` (crafty) | `192.168.2.24` (plex) | announces | -|---|---|---|---|---| -| `k3s-prod-1` | **FAILED** | — | — | `.20` | -| `k3s-prod-2` | ok | **INCOMPLETE** | ok | `.21`, `.22`, `.23-tcp` | -| `k3s-prod-3` | ok | ok | **INCOMPLETE** | `.24` | +| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) | +|---|---|---| +| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` | +| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms | +| `nc -z 192.168.2.20 443` | **OK** | OK | +| plex `192.168.2.24:32400` | **`200`** | `200` | -Symmetric across 3 nodes × 5 VIPs: each node reached every VIP **except** the ones it alone -announced. It is a property of L2 announcement, not of any one Service, and it is harmless for -ordinary pods — they reach Services through the cluster datapath and never ARP for a VIP. This -component is the exception, because `hostNetwork` + `MASQUERADE` put remote clients on the node's -own LAN path. +So the announcer really does **not** answer its own ARP request — that part of the observation is +reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**: -The forward pointed at `k3s-prod-1`, which held the `shared` Gateway lease, so **all 16 Gateway -hostnames were dead over the VPN**. Two things hid it: +* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and + `Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**, + so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before + any route or neighbour lookup for the VIP happens**. +* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the + shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without + a packet crossing the LAN at all. +* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`), + which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real** + service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes — + that is service-aware DNAT, not wire traffic. -* **PiHole survived by accident.** `192.168.2.23` is announced by *two* Services - (`pihole-dns-udp` on prod-1, `pihole-dns-tcp` on prod-2), so prod-2 answered the ARP prod-1 could - not. Names resolved perfectly and every app hung — the hardest shape to diagnose from a client. -* **Nothing reported it.** No probe in this repo compares the forwarded node against the lease - holders, and from outside WireGuard is silent to unauthenticated traffic anyway. +**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis: -Closed by labelling the forwarded node `homelab.keithjackson.dev/vpn-ingress=true` and excluding -that label in `platform-config/cilium/l2-announcement-policy.yaml` (the node then ARP-resolves every -VIP), with `VpnIngressNodeAnnouncesL2VIP` watching the lease holders so a recurrence is loud and -`VpnIngressNodeLabelMissing` watching for the label going absent, which would silently re-arm the -whole failure. Cost: VIP announcement candidacy drops from three nodes to two. +* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an + artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not + anything is wrong. +* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the + two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it + discriminates nothing — and it is not the client path either (see below). + +**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are +**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is +**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the +forwarded path in either direction: it reports success where the forwarded path might fail, and +`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with +a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE` +counters move. + +## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses + +`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the +component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature +to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel +client cannot reach the internet, nothing in the list below is the reason: + +| checked | result on all three nodes | +|---|---| +| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` | +| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change | +| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE | +| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present | +| default route | `default via 192.168.2.1 dev eno1 onlink`, intact | +| node's own internet reachability | `http://1.1.1.1/` returns `301` | + +**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is +closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by +`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled +`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity +traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway +node. A remote client's apparent public address is therefore the **home WAN IP**, which is the +intended behaviour and the thing to state when enrolling a device. + +**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one +enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and +the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the +power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the +forwarded node shows a peer with an `endpoint` and a recent `latest handshake`. ## Prerequisites outside this directory diff --git a/platform/vpn-ingress/daemonset.yaml b/platform/vpn-ingress/daemonset.yaml index 64a07e4..c1a203d 100644 --- a/platform/vpn-ingress/daemonset.yaml +++ b/platform/vpn-ingress/daemonset.yaml @@ -62,39 +62,9 @@ # as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to # whichever node the scheduler picked and silently invalidate that forward on every reschedule. A # DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of -# them, and if that node is lost every other node is already warm and answers the same keys, so no -# client profile ever changes. Each node binds udp/51820 in its own netns, so there is no conflict, -# and the two idle listeners cost ~15 MiB each. -# -# BUT THE NODES ARE NOT INTERCHANGEABLE, AND THIS COMMENT USED TO CLAIM THEY WERE. -# It said "FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work". That was wrong, and the -# wording cost two days of silently broken remote access on 2026-10-08. MEASURED that day: a node -# cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds ITSELF — the announcer -# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and packets to -# that VIP die before leaving the node: -# -# k3s-prod-1, holding the shared Gateway lease: ip neigh show 192.168.2.20 -> FAILED -# k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr …(prod-1) -# -# Because this pod MASQUERADEs clients to the node's own address (see PostUp below), a remote client -# inherits exactly that failure. With the forward on the Gateway's announcing node, all 16 Gateway -# hostnames were unreachable over the VPN while PiHole DNS still resolved — a tunnel that -# handshakes, resolves names, and then hangs on every single app. -# -# THE FIX IS DECLARATIVE AND LIVES OUTSIDE THIS FILE: the forwarded node is labelled -# homelab.keithjackson.dev/vpn-ingress=true by infra/ansible/roles/node_baseline (from -# node_baseline_vpn_ingress_node), and platform-config/cilium/l2-announcement-policy.yaml excludes -# that label from announcing ANY VIP, so the node can ARP-resolve all of them. A label rather than a -# cluster-vars knob because the alert that guards this has to read it too, and the alert unit-test -# harness refuses a postBuild placeholder in an expr. Failover is therefore TWO steps — move -# node_baseline_vpn_ingress_node and re-run the role, THEN re-point the router -# (runbooks/vpn-remote-access.md step 3). -# `VpnIngressNodeAnnouncesL2VIP` (platform/monitoring/alerts.yaml) fires if the two collide again. -# -# This pod deliberately does NOT probe that itself. A readinessProbe on VIP reachability would wedge -# a DaemonSet rollout behind an unrelated Gateway outage (maxUnavailable: 1 below), and a liveness -# probe would tear down a tunnel that is carrying LAN traffic perfectly well. The condition is a -# property of the CLUSTER's announcement layout, so it is watched there. +# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no +# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own +# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each. # # All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the # DaemonSet to cover the cluster; add them if a node is ever tainted. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index 7adc398..f01643a 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -119,11 +119,6 @@ configMapGenerator: - update-rollback=update-rollback.md - vpn-down=vpn-down.md - vpn-egress-stale=vpn-egress-stale.md - # `VpnIngressNodeAnnouncesL2VIP` — the INBOUND remote-access VPN, not the egress tunnel the two - # pages above cover. A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease - # it holds itself, and vpn-ingress MASQUERADEs clients to the node's own address, so the - # forwarded node must announce nothing (measured 2026-10-08). - - vpn-ingress-l2-collision=vpn-ingress-l2-collision.md # Operator procedure, not an alert page. Served because the component it completes is INERT # until three off-repo manual steps are done (kernel module, server key, WAN port-forward) — # and because the one check that actually proves it cannot be run from inside the house. diff --git a/runbooks/vpn-ingress-l2-collision.md b/runbooks/vpn-ingress-l2-collision.md deleted file mode 100644 index 38bb489..0000000 --- a/runbooks/vpn-ingress-l2-collision.md +++ /dev/null @@ -1,165 +0,0 @@ -# vpn-ingress-l2-collision — the VPN ingress node is announcing a LoadBalancer VIP - -**Alerts:** `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) and `VpnIngressNodeLabelMissing` -(warning, 30m) -**File:** `platform/monitoring/alerts.yaml` -**Related:** `/runbooks/vpn-remote-access` is the setup and enrolment runbook for this component. -`/runbooks/vpn-egress-stale` and `/runbooks/vpn-down` are the **other** VPN — the outbound PIA -tunnel — and have nothing to do with this. - -## What this means here - -**A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The -announcer answers ARP for the VIP on behalf of other LAN hosts, but not for the announcing node's -own request. On the lease holder the neighbour entry settles at `FAILED`, and every packet to that -VIP is dropped before it leaves the node. Measured on prod 2026-10-08, symmetric across all three -nodes and five VIPs — each node reached every VIP **except** the ones it alone announced. - -For an ordinary workload this is harmless: pods reach Services through the cluster datapath and -never ARP for a VIP. `platform/vpn-ingress` is the exception. It runs WireGuard with -`hostNetwork: true` and MASQUERADEs remote clients to the node's own address, so a remote client's -packets take that node's LAN path and inherit its ARP failures. **If the node the router forwards -`udp/51820` to is also an L2 lease holder, every remote VPN client silently loses all access to the -Services behind the VIPs it announces.** - -The node terminating WireGuard is marked with `homelab.keithjackson.dev/vpn-ingress=true`, and -`platform-config/cilium/l2-announcement-policy.yaml` excludes that label from announcing anything, -so this should be unreachable. - -**The symptom your users report will not sound like a network fault.** On 2026-10-08 the forward sat -on `k3s-prod-1`, which held the `shared` Gateway lease, so all 16 Gateway hostnames were dead over -the VPN — while PiHole DNS kept answering, because `192.168.2.23` is announced by *two* Services and -`k3s-prod-2` held the other lease. So the tunnel handshakes, `wg show` looks perfect, names resolve, -and **every app hangs on connect**. Expect "the VPN connects but nothing loads", not "the VPN is -down". - -Which Services are affected depends on which lease moved — read `{{ $labels.lease }}` from the -alert. `cilium-l2announce-gateway-cilium-gateway-shared` is the worst case: it is every web app at -once. - -### If it is `VpnIngressNodeLabelMissing` that fired - -That is the watchdog, and it means the protection is **switched off**, not that something is merely -untidy. The exclusion is a `NotIn` on the label, and `NotIn` with an absent key **matches** — so -with the label on no node, the policy excludes nothing and every node is an announcement candidate -again. `VpnIngressNodeAnnouncesL2VIP` is simultaneously **inert**: its join matches nothing and -returns no series, which is indistinguishable from healthy. Treat a quiet critical alert as -meaningless until this warning is cleared. Usual cause: a node was rebuilt and `node_baseline` has -not been re-run on it. - -## First commands - -Run cluster commands through the pinned toolbox, which supplies the kube context: - -```bash -# 1. WHO HOLDS WHAT, and WHO IS THE VPN NODE — the whole question, in two lines. -./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n kube-system get lease | grep l2announce' -./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/vpn-ingress' -# Exactly ONE node should carry vpn-ingress=true, and NO l2announce lease should name it. - -# 2. IS THE EXCLUSION STILL IN THE LIVE POLICY? Expect a matchExpressions list with a -# homelab.keithjackson.dev/vpn-ingress NotIn [true] entry — not a bare matchLabels. -./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod get ciliuml2announcementpolicy default-l2 -o jsonpath="{.spec.nodeSelector}"; echo' - -# 3. IS THE LABEL REACHING PROMETHEUS? If this is empty the alerts are inert even when -# the label is on the node (kube-state-metrics metricLabelsAllowlist). -./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n monitoring exec deploy/kube-prometheus-stack-kube-state-metrics -- \ - wget -qO- localhost:8080/metrics | grep vpn_ingress | head' -``` - -**Confirm the actual damage**, which is the one measurement that is unambiguous. From the WireGuard -pod on the affected node (it shares the host netns, so this is the path a VPN client takes): - -```bash -./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n vpn-ingress get pods -o wide' # pick the pod on that node - -./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n vpn-ingress exec -c wireguard -- \ - sh -c "ping -c1 -W1 192.168.2.20 >/dev/null 2>&1; ip neigh show 192.168.2.20"' -# BROKEN: 192.168.2.20 dev eno1 FAILED -# HEALTHY: 192.168.2.20 dev eno1 lladdr bc:e9:2f:7e:c4:a6 STALE|REACHABLE|DELAY -``` - -The `ping` is only there to trigger ARP — the VIP does not answer ICMP even when healthy, so -**ignore its 100% packet loss**; the neighbour state is the answer. Substitute the VIP that belongs -to `{{ $labels.lease }}` if the Gateway is not the one that moved. - -**Which node actually holds the forward** (the router cannot be queried, but the cluster knows): -only the forwarded node's `wg show` has a peer with an `endpoint` and a recent `latest handshake`. - -```bash -./infra/devcontainer/dev.sh run bash -c \ - 'for p in $(kubectl --context k3s-prod -n vpn-ingress get pods -o name); do - echo "== $p"; kubectl --context k3s-prod -n vpn-ingress exec ${p#pod/} -c wireguard -- wg show; done' -``` - -## Likely causes, most likely first - -1. **The label is on the wrong node** — someone re-pointed the router's forward during a node - failover and did not move `node_baseline_vpn_ingress_node`. This is the most likely cause by a - wide margin, because the router is off-repo and nothing can check it. Confirm with the `wg show` - loop above: the node with the live peer endpoint is the real forward target. - *Fix:* merge a PR setting `node_baseline_vpn_ingress_node` - (`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`) to that node, then re-run the role — - `ansible-playbook -i inventory/hosts.ini -e k3s_hosts=k3s_prod --tags vpn-ingress`. - The role labels the new node and **removes** the label from the old one. Then confirm Cilium - re-elected the lease off the newly-labelled node. - -2. **The exclusion was edited out of `platform-config/cilium/l2-announcement-policy.yaml`**, or the - file was reverted to a bare `matchLabels: {kubernetes.io/os: linux}`. Command 2 above shows this - directly. *Fix:* restore the `matchExpressions` form; the file's own header explains why it is - shaped that way. - -3. **The label is missing entirely** (`VpnIngressNodeLabelMissing` firing) — a node rebuild without - a `node_baseline` re-run. *Fix:* re-run the role as above. - -4. **The label is on the node but not in Prometheus** — `metricLabelsAllowlist` in - `platform/monitoring/kube-prometheus-stack.yaml` lost the `vpn-ingress` entry. Then the critical - alert is inert *and* the watchdog fires. Command 3 above distinguishes this from cause 3. - -5. **Cilium kept renewing a lease its holder is no longer eligible for.** The exclusion is correct - and in place, but the lease never re-elected. This was never verified when the fix was authored - (see `decisions/2026-10-17-vpn-ingress-l2-lease-collision.md`). *Fix:* delete that one Lease - object to force an election — `kubectl -n kube-system delete lease ` — and watch it come - back on a different node. - -6. **A Cilium upgrade changed `NotIn` semantics or the `nodeSelector` field.** Least likely, but it - is the reason this is a runtime alert rather than a static check on the policy text. - -## When to escalate or stop - -**Act now, don't wait for a window.** Remote access is fully broken for the affected VIPs while this -is firing, and it does **not** self-heal: a lease holder keeps renewing, so this condition persists -indefinitely once entered. That is also why the 15m `for:` is safe — a transient move during a node -reboot or a Cilium roll resolves on its own well inside it. - -**The fastest honest mitigation is to move the forward, not the leases.** Re-pointing the router's -`udp/51820` at a node that holds no lease for the affected VIP restores service in seconds, with no -cluster change and no new client profiles — all three nodes answer the same keys. Follow it with the -`node_baseline_vpn_ingress_node` PR and role re-run so the two agree again; until you do, this alert -is watching the wrong node. Note each node's own announcements when choosing: as of 2026-10-08, -prod-2 announced crafty `192.168.2.22` and unifi `192.168.2.21`, prod-3 announced plex -`192.168.2.24`, so neither is a cost-free target. - -**Do not** `kubectl edit` the `CiliumL2AnnouncementPolicy` to fix this — Flux owns it and will revert -you, and the drift is the kind that outlives the memory of making it. The two acceptable hand -actions are deleting a single Lease to force re-election (cause 5), and `kubectl label node` as a -*temporary* stopgap while the PR lands — note that a node rebuild or a `node_baseline` run will undo -a hand-applied label, so it is not a fix. - -**Stop and escalate to the operator if** the leases will not leave the excluded node after a forced -re-election, or if the intended forward target is unclear. Both need a decision about which node -carries remote access, which is an operator call, not a runbook step. - -**Known gap, so you are not misled:** if someone re-points the router without moving the label, -`VpnIngressNodeAnnouncesL2VIP` watches the wrong node and **stays silent** while the VPN breaks in -exactly the way described above. There is no in-cluster signal for that today; closing it needs a -WireGuard exporter publishing which node is actually terminating peer traffic -(`platform/vpn-ingress/README.md`, "Not built") — the `wg show` loop above is the manual substitute. -Until then, if remote access is broken and both alerts are quiet, **check the forward against the -label by hand before believing the alerts.** diff --git a/runbooks/vpn-remote-access.md b/runbooks/vpn-remote-access.md index dfdd874..cba1887 100644 --- a/runbooks/vpn-remote-access.md +++ b/runbooks/vpn-remote-access.md @@ -171,57 +171,14 @@ server without a `wg0` "ready". On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g. `192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check. -> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node. -> -> This step used to say "pick any node — all three run an identical listener". **That sentence -> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a -> cluster-vars knob instead of a free choice here. -> -> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The -> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease -> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they -> leave the node. Measured on prod 2026-10-08: -> -> ```text -> on k3s-prod-1 (held the shared Gateway lease): ip neigh show 192.168.2.20 -> FAILED -> on k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr … (prod-1) -> ``` -> -> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN -> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway -> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so -> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported -> it. -> -`infra/ansible/roles/node_baseline` now labels the forwarded node -> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in -> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes -> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI -> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a -> labelled node taking a lease, but it CANNOT catch the forward being moved without the label. - -Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`). -Note it here: +Pick any node — all three run an identical listener. Note which one you chose here: ``` # WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set -# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml ``` -**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that -was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and -reintroduces the failure above against a different set of addresses (prod-2 announces crafty and -unifi; prod-3 announces plex). Do both, in this order: - -1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role - (`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium - then moves the leases off the newly-labelled node and onto its peers. -2. **Re-point the router** forward at the new node's address. - -Every device keeps working with the same profile either way; all three nodes answer the same keys, -and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that -handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because -that alert watches the node the knob names, not the node your router actually points at. +**Node loss = failover is this one line.** Re-point the forward at another node address; every +device keeps working with the same profile, because all three nodes answer the same keys. Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is already silent to unauthenticated probes (README, "Why this is safe to expose"). @@ -316,6 +273,84 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test. by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes if the router is not delivering the packets, and a `handshake` with no completed session if it is. +## Triage — "the client reaches the LAN and cluster apps, but not the internet" + +Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom +almost always lives, and because the server side has been measured and is not the cause. + +**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.** + +| the profile says | meaning | +|---|---| +| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. | +| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. | + +`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit — +`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client +profile only**, so an already-issued profile keeps whatever it was rendered with until it is +re-rendered and re-imported on the device. The two forms differ by one flag: + +```bash +./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh ' # full tunnel +./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh --split' # LAN only +``` + +Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into +`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is +editing `AllowedIPs` in the profile **on the device** — the server does not care. + +**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the +client connected, then watch the two counters that only forwarded traffic moves: + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c " + wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5; + iptables -t nat -L POSTROUTING -n -v | head -5"' +``` + +* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which + node the router's `udp/51820` forward points at (the router is unreadable from the cluster). +* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is + upstream of the cluster (router forward, client, or carrier), not here. +* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side + failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there, + which breaks internet while node-local cluster VIPs still work. It has not been observed. + +**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly: + +* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack + client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel — + so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4: + `curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it, + and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is + blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it). +* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified + 2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a + blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on + a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all. + +**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the +`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain +**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route +`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the +PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled +`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the +**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6). + +## Triage — which L2 announcement lease a node holds does NOT matter + +Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node +could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`: +that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh` +for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer +VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend +is a node-local socket. Full measurements and the two misleading probes +(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`. + +`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a +separate, pre-existing fault in that service and has nothing to do with the VPN. + ## Revoking a device Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the