diff --git a/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md b/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md index b21be0e..e0923b8 100644 --- a/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md +++ b/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md @@ -1,137 +1,198 @@ --- date: 2026-10-17 branch: fix/vpn-ingress-l2-lease-collision -title: "A node cannot reach a VIP it announces — so the VPN ingress node must announce nothing" +title: "WITHDRAWN: a node CAN reach a VIP it announces. My L2-lease diagnosis was wrong" --- ## What changed -The first device enrolled on the remote-access VPN handshook, resolved names, and could not open a -single connection to any cluster web app. The tunnel was not at fault. The fault is a property of -Cilium L2 announcement that nothing in this repo knew about: +**This branch previously proposed a fix. That fix has been reverted in full. The diagnosis behind it +was wrong.** What is left is documentation of what was actually measured, so the wrong conclusion is +not re-derived — it is very easy to re-derive from `ip neigh` alone. -**A node cannot reach a LoadBalancer VIP whose `CiliumL2AnnouncementPolicy` lease it holds itself.** -The announcer answers ARP for the VIP on behalf of other LAN hosts but not for its own request, so -on the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP is dropped -before it leaves the node. +### The claim, and how it died -`platform/vpn-ingress` is uniquely exposed because it runs WireGuard with `hostNetwork: true` and -MASQUERADEs remote clients to the node's own address — a VPN client's packets take the node's LAN -path and inherit its ARP failures. Ordinary pods never do. The router's `udp/51820` forward pointed -at `k3s-prod-1`, which held `cilium-l2announce-gateway-cilium-gateway-shared`, so **all 16 `shared` -Gateway hostnames were dead over the VPN** — while PiHole DNS kept answering, because -`192.168.2.23` is announced by *two* Services and `k3s-prod-2` held the other lease. Names resolved -and every app hung on connect. +The earlier version of this entry claimed: *a node cannot reach a LoadBalancer VIP whose Cilium L2 +announcement lease it holds itself* — the announcer does not answer its own ARP request, the +neighbour entry settles at `FAILED`, and packets to that VIP die before leaving the node. Because +`platform/vpn-ingress` runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's +own address, a VPN client was said to inherit that. The fix labelled the forwarded node and excluded +that label from announcing, with two alerts and four promtool tests behind it. -Seven files: +The claim makes a sharp prediction. After today's UPS power-cycle **all eight** `cilium-l2announce-*` +leases are held by `k3s-prod-1` (gateway, pihole tcp+udp, crafty tcp+udp, unifi tcp+udp, plex) — +the extreme case. The prediction was therefore that `k3s-prod-1` should reach **none** of the five +VIPs and the other two nodes should reach **all**. -* **`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`** — new - `node_baseline_vpn_ingress_node`; the matching node gets - `homelab.keithjackson.dev/vpn-ingress=true`, and a second task **removes** the label from any node - that no longer matches, so moving the value moves the label rather than adding a second one. Same - shape as `roles/vpn_egress_gateway`'s `egress-gateway` label. Tagged `vpn-ingress`; skipped - entirely when the value is empty. -* **`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`** — set to `k3s-prod-1`, with a - `[parity]` note that it must equal the router's forward. -* **`platform-config/cilium/l2-announcement-policy.yaml`** — `nodeSelector` becomes a - `matchExpressions` list keeping `kubernetes.io/os: linux` and adding - `homelab.keithjackson.dev/vpn-ingress NotIn ["true"]`. The VPN node then holds no lease, so it - ARP-resolves every VIP. This fixes **all** VIPs for VPN clients (Gateway, crafty, plex, pihole, - unifi), not just the one that happened to collide. -* **`platform/monitoring/kube-prometheus-stack.yaml`** — the label added to kube-state-metrics' - `metricLabelsAllowlist`, which is what lets the invariant be watched at all. -* **`platform/monitoring/alerts.yaml`** — `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) joins - `kube_lease_owner` against `kube_node_labels` via `label_replace`, plus - `VpnIngressNodeLabelMissing` (warning, 30m) as its watchdog. -* **`platform/monitoring/tests/vpn-ingress-l2.test.yaml`** — four promtool cases: fires on the real - collision, quiet in the fixed state, quiet across a transient 11-minute lease move (pinning the - `for: 15m`), and the label-absent case asserting the critical alert is inert *and* the watchdog - fires. -* **`runbooks/vpn-ingress-l2-collision.md`** (new, registered in `runbooks/kustomization.yaml`), - **`runbooks/vpn-remote-access.md`**, **`platform/vpn-ingress/{daemonset.yaml,README.md}`** — step 3 - of the setup runbook said *"pick any node — all three run an identical listener"* and *"node loss = - failover is this one line"*. Both were false and the first caused this. Failover is now documented - as two ordered changes. +**The prediction failed.** Measured from the host netns of the `wireguard` pod on each node: -**Design note — why a node label and not a `cluster-vars` knob.** The first attempt added -`WG_INGRESS_NODE` to `clusters/base/cluster-vars.yaml` and substituted it into both the policy and -the alert. The alert half is **impossible**: `platform/monitoring/tests/extract_rules.py` rejects any -alert whose `expr` contains a `${...}` placeholder, because the rule it would unit-test is not the -rule deployed. Rather than duplicate the node name into the alert as a literal (two sources that can -skew), the label is readable from both sides — the selector matches on it and `kube_node_labels` -carries it into Prometheus. One source of truth, and as a side effect the change touches no -sensitive path, so it merges on a green gate rather than needing a human review. +| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) | +|---|---|---| +| `ip neigh show 192.168.2.20` | `FAILED`, no lladdr | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` | +| `curl -k https://192.168.2.20/` | **`200` in 18 ms** | `200` in 13–40 ms | +| `nc -z 192.168.2.20 443` | **OK** | OK | +| `192.168.2.22:25565`, `.23:53`, `.24:32400` | **all OK / `200`** | all OK / `200` | +| `192.168.2.21:8443` | FAIL | **FAIL** (both) | -The DaemonSet deliberately gained **no** probe for this. A readinessProbe on VIP reachability would -wedge a rollout behind an unrelated Gateway outage (`maxUnavailable: 1`); a liveness probe would tear -down a tunnel carrying LAN traffic correctly. +The asserted **cause is still present** today — `FAILED` on the holder, for all five VIPs — while the +asserted **effect is absent**. That is sufficient to falsify the causal claim without having to +explain anything else. The ARP asymmetry is real and reproducible; it is also **inert**. + +### Why it is inert + +No traffic path resolves a LoadBalancer VIP by ARP: + +* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and + `Masquerading: BPF [eno1, wg0]` (Cilium v1.20.1, TCX attach, tunnel/vxlan). Cilium's LB datapath + is attached to **`wg0` as well as `eno1`**, so a remote client's packet arriving on `wg0` for a VIP + is DNAT'd **in BPF, before any route or neighbour lookup for the VIP**. +* `cilium-dbg service list` programs `192.168.2.20:443 => 127.0.0.1:13606` — the shared Gateway's + backend is a **node-local** Envoy socket, so every node serves that VIP without a LAN packet. +* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`), + which is why `curl` succeeds on the holder. Discriminator: a real service port connects while a + bogus one (`nc -z 192.168.2.20 9999`) fails, identically on all three nodes — service-aware DNAT, + not wire traffic. + +### The two probes that misled me + +Both were cited as evidence in the previous version of this entry. + +* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless, and is also a probing + artefact: VIPs answer no ICMP, so a `ping` leaves a failed entry whether or not anything is wrong. +* **`curl --interface wg0`.** The previous entry used this as the control that ruled out a + host-netns artefact. It is not a control: it fails (`rc=28`) from **all three** nodes, including + the two whose ARP for the VIP resolves perfectly. It is uncorrelated with lease holding, so it + discriminates nothing. + +**The deeper error:** a MASQUERADEd client's packets are **forwarded** (netfilter `FORWARD`, then +routing); everything runnable from the pod is **socket-originated**. Different hooks. A host-netns +probe cannot measure the forwarded path in either direction — it reports success where the forwarded +path might fail, and `--interface wg0` reports failure where it does not. Sharing a netns is not +sharing a hook, and that is the assumption the original diagnosis rested on. + +### Files + +Reverted to `origin/main`: `platform-config/cilium/l2-announcement-policy.yaml`, +`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`, +`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`, `platform/monitoring/alerts.yaml`, +`platform/monitoring/kube-prometheus-stack.yaml`, `runbooks/kustomization.yaml`. Deleted: +`platform/monitoring/tests/vpn-ingress-l2.test.yaml`, `runbooks/vpn-ingress-l2-collision.md`. + +Kept, rewritten from scratch: + +* **`platform/vpn-ingress/README.md`** — two measured sections: the L2 table above with why it is + inert and which probes lie, and the internet-egress facts below. +* **`runbooks/vpn-remote-access.md`** — a triage section for the reported symptom ("LAN and cluster + work, internet does not") that works outwards from the client profile, and a short section saying + the L2 lease is a dead end, with the unifi `.21` note. + +**Nothing in `platform/vpn-ingress` or `infra/vpn-ingress` changed behaviourally. No code fix is +proposed by this branch any more.** + +### The internet-routing report: no server-side fix was needed, and none was made + +The brief's prior was that `new-peer.sh` renders a split tunnel and that the operator wants a full +one, to be added as a documented option. **That prior does not match the tree.** `new-peer.sh` has +emitted `AllowedIPs = 0.0.0.0/0` **by default since the component's first commit** (`f92707f`, +2026-10-05, confirmed with `git log -S`); `--split` is the opt-in, and +`runbooks/vpn-remote-access.md:252-253` already documents both forms. So there is no full-tunnel +feature to add, and adding a `WG_FULL_TUNNEL` knob would be a second switch for a default that +already exists. **I deliberately shipped no code change here** — writing one would have meant +inventing a fix for a symptom I could not reproduce (see below). + +The server side was measured end to end and is capable and correct, so the cause is in the client +profile or the client. The triage section names the two candidates that fit the symptom exactly — +a `--split` profile, and the IPv4-only `AllowedIPs` letting a dual-stack client prefer AAAA and +leak public traffic outside the tunnel — each with a one-line test that settles it. ## What I verified -Measured live against prod, `--context k3s-prod`, on 2026-10-08: +Measured live against prod, `kubectl --context k3s-prod --request-timeout=15s`, read-only, on +2026-10-17: -* **The whole tunnel is healthy**, which is what made this hard: `wg show` on `k3s-prod-1` shows the - peer with a real endpoint and bidirectional transfer (346 KiB in / 200 KiB out); - `net.ipv4.ip_forward=1`; `wg0` up at `10.90.0.1/24` MTU 1420; `iptables -t nat -S POSTROUTING` - carries the `MASQUERADE` rule with 26 packets; `FORWARD -i/-o wg0` ACCEPT with 108 / 171 packets; - DaemonSet generation 2, 3/3 ready on the content-hashed `wg-peers-m9kcckcb5t`. -* **The ARP failure and its symmetry.** On `k3s-prod-1`: `ip neigh show 192.168.2.20` → `FAILED`, - while `.21`/`.23` resolve to prod-2's MAC `04:0e:3c:96:b0:94`. On prod-2/prod-3, `.20` resolves to - prod-1's `bc:e9:2f:7e:c4:a6`. TCP agrees: `.20:443` and `.20:80` FAIL from prod-1, OK from the - other two. Repeated across 3 nodes × 5 VIPs — each node reached every VIP except the ones it alone - announced (prod-2 ✗ `.22`/`.21`; prod-3 ✗ `.24`). -* **Not an artefact of probing from the host netns.** `curl --interface wg0` from prod-1 fails the - same way, and a neighbour entry is node-global state, so the forwarded path cannot differ in ARP - resolution. -* **The Gateway itself was never broken:** from the workstation (192.168.2.133) - `home.keithjackson.dev` → `200`, `plex.home.keithjackson.dev` → `401`, both at `192.168.2.20`. - From prod-1 the router (`.1`) and TrueNAS (`.30`) are `REACHABLE`. -* **`NotIn` semantics, empirically** — the basis for the fail-back claim in the policy's comment: - `kubectl get nodes -l 'kubernetes.io/hostname notin (k3s-prod-1)'` returns prod-2 and prod-3, and - `NotIn` on a key no node carries returns **all three**. So an unapplied label excludes nothing and - degrades to the old behaviour rather than taking every VIP off the LAN. -* **The alert metric is really scraped:** `kube_lease_owner` is served by kube-state-metrics (55 - series, every `cilium-l2announce-*` lease with its holder in `lease_holder`), and `kube_node_labels` - already carries the two `egress-*` labels through the same allowlist (`k3s-prod-3` shows - `egress-gateway=true`), so the third entry follows a proven path. -* **The four promtool cases pass** (`platform/monitoring/tests/run.sh` → `SUCCESS`, 55 rules - checked), including that `VpnIngressNodeAnnouncesL2VIP` is inert when the label is absent. -* All edited YAML parses; the rendered `nodeSelector` is the intended two-expression list. +* **The lease distribution.** All eight `cilium-l2announce-*` leases held by `k3s-prod-1` + (`.spec.holderIdentity`). The brief's observation reproduced; it had not moved again. +* **The falsification, in full**, as tabulated above: 3 nodes x 5 VIPs, `ip neigh` + `nc -z` + a + real `curl` for each. The holder reaches four of five VIPs; `.21` fails from everywhere. +* **The socket-LB discriminator.** Real service port OK, bogus port FAIL, identically on all three + nodes — so the successes are service-aware DNAT, not wire delivery. +* **`curl --interface wg0` fails on all three nodes**, including non-announcing ones with + `REACHABLE` neighbour entries. This is the measurement that destroys the original control. +* **The datapath facts** quoted above from `cilium-dbg status --verbose` and + `cilium-dbg service list` on `cilium-55jv8` (`k3s-prod-1`), plus `cilium-config`: + `kube-proxy-replacement=true`, `enable-l2-announcements=true`, `enable-bpf-masquerade=true`, + `routing-mode=tunnel`, `devices=` (empty, auto-detected). Cilium `v1.20.1` on all three. +* **The VIP is not a local address and routes on-link.** `ip route get 192.168.2.20` → + `dev eno1 src 192.168.2.10` on the holder; nothing in `table local`; `ip -4 addr` shows no VIP. + So the 200 is not loopback delivery by the host stack — it is BPF. +* **The internet path, server side, on all three nodes:** `/proc/sys/net/ipv4/ip_forward` = `1` + (read, never written); `iptables -t nat -S POSTROUTING` carries + `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE`; the preceding `CILIUM_POST_nat` chain is **empty with 0 + packets**, so it cannot short-circuit that rule; `FORWARD -i wg0` and `-o wg0` ACCEPT both + present; `default via 192.168.2.1 dev eno1 onlink` intact; `ip route get 1.1.1.1` resolves via it; + `curl http://1.1.1.1/` returns `301` from each node. +* **The PIA / egress-gateway answer.** `platform/vpn-egress/egress-steering.yaml` selects by + `podSelector` on `egress-class: vpn`; `platform/vpn-ingress/daemonset.yaml:130` labels this pod + `egress-class: internet`; `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`), absent on + `-1` and `-2`. So a full tunnel egresses the **house WAN**, and a client's apparent address is the + home WAN IP, not PIA's. +* **PiHole resolves public names** — `nslookup google.com 192.168.2.23` from the pod returned + `142.250.113.138`. +* **No client is connected, and none has been since the power-cycle.** `wg show wg0` lists the one + enrolled peer (`10.90.0.2/32`) with **no endpoint and no handshake** on all three nodes, and both + `FORWARD -i wg0` and the `MASQUERADE` rule read **0 packets / 0 bytes** on all three. The + `wireguard` pods restarted 153 min ago (2 restarts, 11d old), which reset the counters. +* **The full-tunnel default's age**, from `git log -S` on `new-peer.sh`: introduced in `f92707f` + (2026-10-05) and never changed. +* **The branch rebased cleanly** onto `origin/main` (`1a0b208`, PR #70 merged) with no conflicts. ## What I did NOT verify -* **The fix is UNAPPLIED and therefore UNPROVEN end to end.** Applying it needs a mutating - `kubectl`/`flux reconcile` plus an Ansible run, which are operator actions (CLAUDE.md §5). I have - **not** observed `k3s-prod-1` resolve `192.168.2.20` after the exclusion lands, nor a VPN client - reaching an app. The prediction rests on the measured symmetry (non-announcing nodes resolve every - VIP, 15/15 data points), not on a test of the changed state. **Verify after merge, in this order:** - re-run `node_baseline --tags vpn-ingress`; `kubectl get nodes -L - homelab.keithjackson.dev/vpn-ingress` (exactly one node); `kubectl -n kube-system get lease | grep - l2announce` (no holder is that node); `ip neigh show 192.168.2.20` from the prod-1 wireguard pod - (expect a `lladdr`, not `FAILED`); then a client. -* **That Cilium actually releases a lease when its holder stops matching the `nodeSelector`.** I - assumed re-election and did not test it. If Cilium keeps renewing, the affected Leases must be - deleted once by hand to force an election. Check this first if the fix appears not to work. -* **The brief interruption on apply.** Moving the Gateway and `pihole-dns-udp` leases off prod-1 will - blip `192.168.2.20` and `192.168.2.23` for as long as re-election plus gratuitous ARP takes. Not - measured, and I did not test whether in-flight LAN sessions survive it. -* **The Ansible tasks have never been executed.** They parse as YAML and follow - `roles/vpn_egress_gateway`'s shape, but neither the label task nor the label-removal task has run, - so the `changed_when`/`failed_when` conditions and the `k3s kubectl label ... -` removal path are - unproven. The removal task's `failed_when` tolerates `'not found'` in stderr; that string was - assumed from kubectl's behaviour, not observed. -* **The residual off-repo coupling is NOT closed and cannot be, as built.** The router's forward is - unreadable from the cluster. If someone re-points it without moving the label, the alert watches - the wrong node and stays silent while the VPN breaks identically against a different set of VIPs. - Only the runbook guards it. Closing it needs a WireGuard exporter publishing which node is - actually terminating peer traffic (`wg show` knows — only the forwarded node has a peer - `endpoint`); listed as "Not built" in the component README and **recommended follow-up, not done - here**. -* **Why Cilium behaves this way** — intended, a known upstream issue, or fixed in a later version. I - measured the behaviour and did not read Cilium's source or issue tracker, so I cannot say whether - an upgrade would remove the need for this exclusion. -* **`192.168.2.21:8443` (unifi) fails from all three nodes**, including ones where ARP resolves, so - it is a different, pre-existing fault. Not investigated, not addressed. -* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 offline - subset only; none of the above is covered by a test that runs in the gate. +* **THE FORWARDED PATH IS STILL UNMEASURED, in both directions.** No client is connected, so I could + not test what a real remote client does, and — as established above — **no probe runnable from the + pod can substitute for it**. Specifically I did **not** prove that a forwarded client packet to a + VIP is DNAT'd by Cilium's `wg0` program. That inference rests on `cilium-dbg` listing `wg0` as a + LB device plus the node-local Gateway backend, not on an observed client packet. **It is the one + remaining way the withdrawn diagnosis could be partly right** — if `wg0` BPF were somehow not in + the forwarded path, forwarded traffic would need ARP and would die on the holder. I judge this + unlikely but it is unproven. *To close it: connect a client, then from the node whose `wg show` + shows the peer endpoint, watch `FORWARD -i wg0` and the `MASQUERADE` counters move while the + client loads a cluster app — and repeat it on a node that holds leases and one that does not.* +* **WHICH NODE THE ROUTER FORWARDS `udp/51820` TO IS UNDETERMINABLE RIGHT NOW**, for the same + reason. **I need a client connected** to learn it; the router is unreadable from the cluster. +* **I cannot reproduce or explain the 2026-10-08 measurements.** That entry recorded `nc -z` TCP + failures correlated with lease holding across 15 node x VIP pairs. Today the same test passes. + My leading hypothesis is **stale or incorrectly-programmed BPF LB state on `k3s-prod-1`**, + rebuilt by today's full power-cycle — which would make the lease correlation coincidental. A + competing hypothesis is that Cilium had not auto-detected `wg0` then (`devices` is empty, and + `wg0` is created by this DaemonSet after Cilium starts). **Both are unproven, and neither is + needed for the falsification** — the asserted cause is present today without its effect. I did not + retro-measure anything, and I did not read Cilium's source or issue tracker. +* **The operator's report cannot be a description of "right now."** No client has connected since + the power-cycle (0 forwarded packets, no handshake on any node), so "clients can reach cluster + services and the LAN but not the internet" must describe a session before the power-cycle or one + that has since dropped. **I did not establish when it was observed, or with which profile.** +* **THE INTERNET ROOT CAUSE IS NOT DIAGNOSED.** I proved the server side is *capable*; I did **not** + find the fault, and I shipped no fix for it. The two candidates in the triage section (a + `--split` profile; IPv6 preference leaking outside an IPv4-only `AllowedIPs`) are **reasoned, not + measured**. *What I need:* the `AllowedIPs` line from the client's own profile — that one line + decides split vs full and halves the search — and a connected client for step 2. +* **Whether `--split` was in fact used for the enrolled peer.** The profile holds a private key and + lives only on the operator's machine and the device; it is not in the repo, and I did not look for + it. Server-side `AllowedIPs` is a `/32` source filter and says nothing about the client's routing. +* **Whether `MTU = 1420` holds on a full-tunnel WAN path.** Not measured, and not measurable without + a client. It is the standard 1500 − 60; a client behind PPPoE or CGNAT with a smaller path MTU + would see large transfers hang while small requests succeed. The runbook's existing MTU note + (drop to 1280) already covers the remedy. +* **Adding `::/0` to `AllowedIPs` is untested advice.** The triage section offers it to blackhole + IPv6 rather than leak it, and states that the tunnel has no IPv6 egress. I did not try it on a + client, and I do not know how each WireGuard client behaves when handed an IPv6 route on an + interface with no IPv6 address. +* **`192.168.2.21:8443` (unifi) fails from all three nodes**, as it did on 2026-10-08 — a separate, + pre-existing fault. Not investigated, not addressed, and now recorded in the runbook as such. +* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so the gate proves the L0/L1 + offline subset only. Nothing above is covered by a test that runs in CI. +* **Nothing was applied, and nothing needs to be.** The branch is now documentation-only, so there + is no Flux reconcile, Ansible run or `kubectl` step to perform after merge. I ran no mutating + `kubectl`, `flux` or Ansible, and did not run `new-peer.sh` (it mints key material). * I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b). diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index d286c9d..2b865a6 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -420,22 +420,3 @@ flux_bootstrap_env_path: "clusters/prod" # rather than leaving an inert drop-in behind, so this group-wide `true` is still correct # for it. etcd_io_priority_enabled: true - -# --- The node terminating remote-access WireGuard — DECIDED (measured 2026-10-08) --------------- -# k3s-prod-1 (192.168.2.10) is what the SOHO router forwards WAN udp/51820 to, confirmed from the -# cluster rather than assumed: only prod-1's `wg show` had a peer with an `endpoint` and a recent -# handshake, the other two listeners were idle. -# -# node_baseline labels it `homelab.keithjackson.dev/vpn-ingress=true`, and -# platform-config/cilium/l2-announcement-policy.yaml excludes that label from announcing any -# LoadBalancer VIP -- because a node CANNOT reach a VIP whose Cilium L2 announcement lease it holds -# itself (the announcer does not answer its own ARP request; the neighbour entry goes to `FAILED`). -# prod-1 held the `shared` Gateway lease, and since vpn-ingress MASQUERADEs clients to the node's -# own address, all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved. -# -# [parity] MUST equal the node named by the router's forward (runbooks/vpn-remote-access.md step 3). -# Nothing in CI can read that forward, so re-pointing it means changing BOTH: this value plus a -# node_baseline re-run, and the router line. `VpnIngressNodeAnnouncesL2VIP` catches the case where -# the label lands on a node that then takes a lease; it CANNOT catch the forward being moved -# without this value following, which is why the runbook spells the two-step out. -node_baseline_vpn_ingress_node: k3s-prod-1 diff --git a/infra/ansible/roles/node_baseline/defaults/main.yml b/infra/ansible/roles/node_baseline/defaults/main.yml index ae7a493..b6a7f4b 100644 --- a/infra/ansible/roles/node_baseline/defaults/main.yml +++ b/infra/ansible/roles/node_baseline/defaults/main.yml @@ -259,30 +259,3 @@ node_baseline_dns_options: # Written as ONE marked block, so removing an entry here removes it from the node, and # an empty list removes the block entirely. node_baseline_hosts_pins: [] - -# --- The node that terminates remote-access WireGuard (tags: vpn-ingress) ---------------------- -# Hostname of the node the SOHO router forwards WAN udp/51820 to, i.e. the one platform/vpn-ingress -# actually serves remote clients from. Set it in group_vars for the environment; EMPTY by default, -# because dev has no such forward and because a wrong value is worse than none. -# -# WHAT IT IS FOR, AND WHY IT IS NOT COSMETIC. The matching node gets -# `homelab.keithjackson.dev/vpn-ingress=true`, and -# platform-config/cilium/l2-announcement-policy.yaml EXCLUDES that label from announcing any -# LoadBalancer VIP. MEASURED on prod 2026-10-08: a node cannot reach a VIP whose Cilium L2 -# announcement lease it holds itself (the announcer does not answer its own ARP request -- the -# neighbour entry goes to `FAILED`). Because vpn-ingress runs on `hostNetwork` and MASQUERADEs -# clients to the node's own address, a VPN client inherits that: with the forward on the node -# holding the `shared` Gateway lease, all 16 Gateway hostnames were dead over the VPN while PiHole -# DNS still resolved. Names worked, every app hung, nothing alerted. -# -# [parity] MUST name the node the router's forward actually points at. The forward is off-repo and -# no check can read it (runbooks/vpn-remote-access.md step 3). Changing the forward is therefore a -# two-part change: this value (then re-run the role), and the router line. -# -# Only ONE node should ever carry the label: it removes that node from VIP announcement candidacy, -# so labelling all three would leave nothing announcing any VIP at all. The task below removes the -# label from a node that no longer matches, so moving the value moves the label. -node_baseline_vpn_ingress_node: "" -node_baseline_vpn_ingress_label_key: homelab.keithjackson.dev/vpn-ingress -# k3s ships its kubeconfig here; every prod node is a k3s server, so `k3s kubectl` works locally. -node_baseline_vpn_ingress_kubeconfig: /etc/rancher/k3s/k3s.yaml diff --git a/infra/ansible/roles/node_baseline/tasks/main.yml b/infra/ansible/roles/node_baseline/tasks/main.yml index de317f4..1422573 100644 --- a/infra/ansible/roles/node_baseline/tasks/main.yml +++ b/infra/ansible/roles/node_baseline/tasks/main.yml @@ -1109,45 +1109,3 @@ state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}" backup: true tags: [dns, hosts] - -# ── vpn-ingress node label — tag `vpn-ingress` ──────────────────────────────────────────────── -# Marks the node terminating remote-access WireGuard so Cilium's L2 announcement policy can EXCLUDE -# it from announcing LoadBalancer VIPs (a node cannot reach a VIP it announces itself -- see -# defaults/main.yml). Same shape as roles/vpn_egress_gateway's egress-gateway label, and for the -# same reason: the label is the one place BOTH a nodeSelector and a PromQL alert can read it from. -# -# Skipped entirely when node_baseline_vpn_ingress_node is empty (dev, or before the forward exists), -# so this adds nothing to an environment that has no WAN forward. -- name: Label this node as the vpn-ingress node - ansible.builtin.command: - cmd: >- - k3s kubectl label node {{ inventory_hostname }} - {{ node_baseline_vpn_ingress_label_key }}=true --overwrite - environment: - KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}" - register: node_baseline_vpn_ingress_label_apply - changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_apply.stdout" - when: - - node_baseline_vpn_ingress_node | length > 0 - - inventory_hostname == node_baseline_vpn_ingress_node - tags: [vpn-ingress] - -# The other half: moving the knob must MOVE the label, not add a second one. Two labelled nodes -# would remove two of three nodes from announcement candidacy; three would leave no announcer at -# all and take every VIP off the LAN. -- name: Remove the vpn-ingress label from a node that no longer holds the forward - ansible.builtin.command: - cmd: >- - k3s kubectl label node {{ inventory_hostname }} - {{ node_baseline_vpn_ingress_label_key }}- - environment: - KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}" - register: node_baseline_vpn_ingress_label_remove - changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_remove.stdout" - failed_when: - - node_baseline_vpn_ingress_label_remove.rc != 0 - - "'not found' not in node_baseline_vpn_ingress_label_remove.stderr" - when: - - node_baseline_vpn_ingress_node | length > 0 - - inventory_hostname != node_baseline_vpn_ingress_node - tags: [vpn-ingress] diff --git a/platform-config/cilium/l2-announcement-policy.yaml b/platform-config/cilium/l2-announcement-policy.yaml index c9229f5..d65e0ff 100644 --- a/platform-config/cilium/l2-announcement-policy.yaml +++ b/platform-config/cilium/l2-announcement-policy.yaml @@ -16,71 +16,6 @@ spec: externalIPs: false interfaces: - ^e(n|th) - # =============================================================================================== - # WHY THIS IS A matchExpressions LIST AND NOT `matchLabels: {kubernetes.io/os: linux}` - # =============================================================================================== - # The second expression excludes every node labelled `homelab.keithjackson.dev/vpn-ingress=true` - # from announcing ANY LoadBalancer VIP. That is the node the SOHO router forwards WAN udp/51820 to. - # It is not tuning; it is the fix for a measured, silent, total failure of remote access. - # - # MEASURED ON PROD 2026-10-08: A NODE CANNOT REACH A VIP IT ANNOUNCES ITSELF. Cilium's L2 - # announcer answers ARP for the VIP on behalf of OTHER LAN hosts, but not for the announcing node - # itself. On the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP - # is dropped before it leaves the node: - # - # k3s-prod-1, holding cilium-l2announce-gateway-cilium-gateway-shared: - # ip neigh show 192.168.2.20 -> 192.168.2.20 dev eno1 FAILED (its own VIP) - # nc -z 192.168.2.20 443 -> FAIL nc -z 192.168.2.20 80 -> FAIL - # k3s-prod-2 and k3s-prod-3, holding no lease for it: - # ip neigh show 192.168.2.20 -> lladdr bc:e9:2f:7e:c4:a6 (prod-1) nc 443/80 -> OK - # - # Symmetric across all 3 nodes x 5 VIPs: every node reached every VIP EXCEPT the ones it alone - # announced (prod-2 could not reach crafty 192.168.2.22 or unifi 192.168.2.21; prod-3 could not - # reach plex 192.168.2.24). So this is a property of L2 announcement, not of one Service. - # - # WHY IT ONLY HURTS THE VPN NODE. For an ordinary workload this is harmless: pods reach Services - # through the cluster datapath and never ARP for a VIP. platform/vpn-ingress is the exception -- - # it runs WireGuard with `hostNetwork: true` and MASQUERADEs remote clients to the node's own - # address, so a VPN client's packets take the node's LAN path and inherit exactly this ARP - # failure. On 2026-10-08 the forward pointed at k3s-prod-1, which held the Gateway lease, and all - # 16 Gateway hostnames were unreachable over the VPN while PiHole DNS still answered -- so names - # resolved and every app hung. Nothing in the cluster reported it; the alerts below now do. - # - # WHY A NODE LABEL AND NOT A cluster-vars KNOB. A knob would put the node's identity in git, which - # is more reviewable -- but it CANNOT BE USED BY THE ALERT that guards this, and an unguarded - # invariant here is worth less than a reviewable one. The alert harness - # (platform/monitoring/tests/extract_rules.py) REFUSES any alert whose `expr` contains a - # dollar-brace postBuild placeholder (spelled without braces here ON PURPOSE: this file is itself - # envsubst'd, and a literal dollar-brace-dot-dot-dot is UNPARSEABLE to envsubst, which fails the - # WHOLE Kustomization -- ci: flux-postbuild-substitution caught exactly this comment), because the - # rule it would unit-test is not the rule that gets - # deployed. A label is readable from BOTH sides: this selector matches on it, and - # `kube_node_labels` carries it into Prometheus so `VpnIngressNodeAnnouncesL2VIP` can join lease - # holders against it. One source of truth, no git-vs-cluster skew, and the policy stays - # env-agnostic as its header above claims. The label is set by - # infra/ansible/roles/node_baseline (node_baseline_vpn_ingress_node), the same shape - # roles/vpn_egress_gateway already uses for `egress-gateway`. - # - # THE COST, STATED: VIP announcement candidacy drops from 3 nodes to 2. That is still redundant, - # and the trade is deliberate -- the excluded node is the one whose *outbound* reach to every VIP - # matters most, because it carries every remote client. If BOTH remaining nodes are down the VIPs - # go dark where previously one node might have served them, but at that point the cluster has one - # node left and far worse problems. - # - # `NotIn` with an ABSENT key MATCHES (k8s apimachinery: a missing label is "not in the set"), so - # the other two nodes need no label and nothing has to be applied to them. If the label is never - # applied anywhere, this excludes nothing and the policy degrades to its pre-2026-10-08 behaviour - # -- the status quo, not an outage: it fails back, not down. That silent-degradation path is - # exactly why `VpnIngressNodeLabelMissing` exists as a watchdog: without it, the alert below - # would be structurally INERT (its join would match nothing) and would look identical to healthy. nodeSelector: - matchExpressions: - - key: kubernetes.io/os - operator: In - values: - - linux - # The node terminating remote-access WireGuard must announce nothing -- see above. - - key: homelab.keithjackson.dev/vpn-ingress - operator: NotIn - values: - - "true" + matchLabels: + kubernetes.io/os: linux diff --git a/platform/monitoring/alerts.yaml b/platform/monitoring/alerts.yaml index ef18519..e007f2d 100644 --- a/platform/monitoring/alerts.yaml +++ b/platform/monitoring/alerts.yaml @@ -1147,97 +1147,6 @@ spec: metricLabelsAllowlist. runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale" # ----------------------------------------------------------------------------------------- - # REMOTE-ACCESS VPN (platform/vpn-ingress) -- NOT the egress tunnel above. Different - # direction, different workload: this is inbound WireGuard for remote devices. - # ----------------------------------------------------------------------------------------- - - alert: VpnIngressNodeAnnouncesL2VIP - # THE FAILURE THIS CATCHES. MEASURED on prod 2026-10-08: a node CANNOT reach a - # LoadBalancer VIP whose Cilium L2 announcement lease it holds itself -- the announcer - # does not answer its own ARP request, so the neighbour entry settles at `FAILED` and - # packets die before leaving the node (`ip neigh show 192.168.2.20` -> FAILED on the - # lease holder, resolved on both others). platform/vpn-ingress runs WireGuard on - # `hostNetwork` and MASQUERADEs remote clients to the node's own address, so if the node - # the router forwards udp/51820 to is also a lease holder, VPN clients cannot reach the - # VIPs it announces. On 2026-10-08 that was the `shared` Gateway: all 16 app hostnames - # dead over the VPN, while PiHole DNS kept answering (its VIP has two Services and the - # other lease sat on prod-2) -- so the tunnel handshook, names resolved, and every app - # hung. Nothing reported it for ~2 days. - # - # platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled - # `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so this should be - # permanently unsatisfiable. The alert exists because the exclusion has ways to stop - # working that no static check on that file's TEXT can see: the label being removed or - # moved to the wrong node, a Cilium upgrade changing `NotIn` semantics or the - # nodeSelector field, and the selector silently failing to match. This watches the - # OUTCOME -- who actually holds the lease -- so all of them surface the same way. - # - # WHY THE LABEL JOIN RATHER THAN A NODE NAME. The node's identity is NOT written into - # this expr, because the harness (tests/extract_rules.py) refuses any alert whose expr - # contains a dollar-brace placeholder -- spelled in words because a literal one here is - # unparseable to envsubst and would fail this whole Kustomization. Rightly so: the rule it - # unit-tests would not be the - # rule deployed. Joining `kube_lease_owner` against `kube_node_labels` keeps ONE source - # of truth (the label itself, which is also what the policy selects on), so this alert - # cannot drift out of agreement with the exclusion it guards. `label_replace` is needed - # because kube_lease_owner names the holder `lease_holder` while kube_node_labels names - # it `node`; the join is then on a shared `node` label. The label is in - # kube-state-metrics' metricLabelsAllowlist (kube-prometheus-stack.yaml) -- without that - # line this rule is inert, which `VpnIngressNodeLabelMissing` below exists to catch. - # - # `for: 15m` because leases legitimately move during a node reboot or a Cilium roll and - # Cilium re-elects within seconds; a real collision persists, since the lease holder - # keeps renewing and it never self-heals. - expr: | - max by (lease) ( - label_replace( - kube_lease_owner{namespace="kube-system", lease=~"cilium-l2announce-.*"}, - "node", "$1", "lease_holder", "(.+)" - ) - * on (node) group_left () - kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"} - ) == 1 - for: 15m - labels: - severity: critical - annotations: - summary: "VPN ingress node is announcing a LoadBalancer VIP ({{ $labels.lease }})" - description: >- - The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access - WireGuard, and it now holds the Cilium L2 announcement lease `{{ $labels.lease }}`. A - node cannot reach a VIP it announces itself, so every remote VPN client has just lost - all access to that VIP's Service -- silently, with DNS still resolving. Check that - platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that - the label is still on the node the router forwards udp/51820 to. - runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" - - alert: VpnIngressNodeLabelMissing - # THE WATCHDOG for the rule above, and the reason it is not optional. The exclusion in - # l2-announcement-policy.yaml is a `NotIn` on this label, and `NotIn` with an ABSENT key - # MATCHES -- so if the label is on NO node, the policy excludes nothing, every node is an - # announcement candidate again, and the 2026-10-08 failure is fully re-armed. The alert - # above would not fire either: its join would match nothing and return no series. Present, - # evaluating, and incapable of firing -- the exact "inert critical alert" shape that - # VPNEgressUplinkLabelMissing exists for on the egress side, and that this repo has been - # burned by before. - # - # The label is applied by infra/ansible/roles/node_baseline - # (node_baseline_vpn_ingress_node); the most likely cause of it going absent is a node - # rebuild where that role has not been re-run. - expr: absent(kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"}) - for: 30m - labels: - severity: warning - annotations: - summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF" - description: >- - No node publishes homelab.keithjackson.dev/vpn-ingress=true, so - l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none: - any node may now announce a LoadBalancer VIP, including the one terminating - remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so - the warning matters more than its severity suggests. Re-run the node_baseline role on - the forwarded node, and confirm the label is in kube-state-metrics' - metricLabelsAllowlist. - runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" - # ----------------------------------------------------------------------------------------- # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown` diff --git a/platform/monitoring/kube-prometheus-stack.yaml b/platform/monitoring/kube-prometheus-stack.yaml index 0daf563..1f75c1c 100644 --- a/platform/monitoring/kube-prometheus-stack.yaml +++ b/platform/monitoring/kube-prometheus-stack.yaml @@ -281,20 +281,8 @@ spec: # without anyone having to remember to re-enable it later. Scoped to one label on one resource: # 3 extra series, no cardinality concern. Label name is sanitized by KSM to # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`). - # `vpn-ingress` (added 2026-10-17) marks the node the SOHO router forwards WAN udp/51820 to. - # platform-config/cilium/l2-announcement-policy.yaml excludes that node from announcing any - # LoadBalancer VIP, because a node cannot reach a VIP whose L2 announcement lease it holds - # itself and vpn-ingress MASQUERADEs remote clients to the node's own address (measured - # 2026-10-08: all 16 Gateway hostnames dead over the VPN while DNS still resolved). It is - # allowlisted HERE because that is the only way the invariant can be WATCHED: - # `VpnIngressNodeAnnouncesL2VIP` joins `kube_lease_owner` against this label, and the alert - # unit-test harness refuses a dollar-brace placeholder in an alert expr (written out in words: - # a literal one here is unparseable to envsubst and fails the whole Kustomization), so a knob - # could not have been used. Without this line BOTH that alert and its - # `VpnIngressNodeLabelMissing` watchdog are structurally inert. Same reasoning, and the same - # 3-extra-series cost, as the two egress labels below. metricLabelsAllowlist: - - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink,homelab.keithjackson.dev/vpn-ingress]" + - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]" # --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) --------------------------------- # Velero exports NOTHING that enumerates Schedule objects and nothing that exposes # `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that diff --git a/platform/monitoring/tests/vpn-ingress-l2.test.yaml b/platform/monitoring/tests/vpn-ingress-l2.test.yaml deleted file mode 100644 index 4033449..0000000 --- a/platform/monitoring/tests/vpn-ingress-l2.test.yaml +++ /dev/null @@ -1,203 +0,0 @@ -# promtool rule-unit tests for VpnIngressNodeAnnouncesL2VIP + VpnIngressNodeLabelMissing -# (platform/monitoring/alerts.yaml) -# -# Run with: platform/monitoring/tests/run.sh -# (the merge gate runs the same harness via -# conformance/tests/l0-static/alert-rules-unit-tested) -# -# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS -# MEASURED on prod 2026-10-08: a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement -# lease it holds ITSELF -- the announcer does not answer its own ARP request, so on the lease holder -# the neighbour entry settles at `FAILED` and packets to that VIP die before leaving the node. -# platform/vpn-ingress runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's -# own address, so a VPN client takes that node's LAN path and inherits the failure. With the -# router's udp/51820 forward on k3s-prod-1, which held the `shared` Gateway lease, all 16 Gateway -# hostnames were dead over the VPN -- while PiHole DNS kept answering, because its VIP has two -# Services and k3s-prod-2 held the other lease. The tunnel handshook, names resolved, every app -# hung, and nothing reported it for ~2 days. -# -# platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled -# `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so VpnIngressNodeAnnouncesL2VIP -# should be permanently unsatisfiable. THAT IS EXACTLY WHY IT NEEDS A TEST: an alert expected never -# to fire is indistinguishable, in production, from an alert that CANNOT fire. Nothing would ever -# reveal a mistyped label, a broken `label_replace`, or a join on the wrong label. Test 1 proves it -# fires on the real failure; test 2 proves it is quiet in the fixed state, so it is not a rule that -# fires on anything; test 3 pins the `for: 15m` against routine lease churn. -# -# THE JOIN IS THE FRAGILE PART AND IS TESTED DELIBERATELY. `kube_lease_owner` names the holder in -# `lease_holder`; `kube_node_labels` names it `node`. The rule bridges them with `label_replace` -# and joins `on (node) group_left ()`. Fixtures therefore carry BOTH series shapes with real label -# sets, because a join that silently matches nothing returns no series -- which looks exactly like -# "healthy" and is the failure mode this whole group exists to prevent. -# -# Fixture values are prod's real shape (measured 2026-10-08): leases -# `cilium-l2announce-gateway-cilium-gateway-shared` (k3s-prod-1) and -# `cilium-l2announce-pihole-pihole-dns-tcp` (k3s-prod-2), in namespace kube-system, with -# k3s-prod-1 the vpn-ingress node. - -rule_files: - - generated.rules.yaml - -evaluation_interval: 1m - -tests: - # ============================================================================================= - # 1. THE BUG, EXACTLY AS IT HAPPENED -- the vpn-ingress node holds the Gateway lease: FIRES. - # ============================================================================================= - - interval: 1m - name: vpn-ingress node holds the shared Gateway L2 lease - VpnIngressNodeAnnouncesL2VIP fires - input_series: - # The collision. A lease holder keeps renewing, so this series does not lapse on its own -- - # which is why the condition never self-heals and why a 15m `for:` is safe. - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}' - values: '1+0x40' - # A second lease held by a DIFFERENT node, present throughout. This is the discriminator that - # stops the rule from being "any l2announce lease exists anywhere": if the join were dropped, - # this series alone would fire the alert, and test 2 would fail. - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-pihole-pihole-dns-tcp",lease_holder="k3s-prod-2"}' - values: '1+0x40' - # k3s-prod-1 is the vpn-ingress node. Only this node carries the label. - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}' - values: '1+0x40' - # The other two nodes publish kube_node_labels WITHOUT the vpn-ingress label. Present so the - # join is exercised against non-matching nodes rather than against nothing at all. - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}' - values: '1+0x40' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}' - values: '1+0x40' - alert_rule_test: - # Held 15m, so nothing before then. - - eval_time: 14m - alertname: VpnIngressNodeAnnouncesL2VIP - exp_alerts: [] - - eval_time: 20m - alertname: VpnIngressNodeAnnouncesL2VIP - exp_alerts: - # EXACTLY ONE alert: the pihole lease on prod-2 must NOT produce a second one. - - exp_labels: - severity: critical - lease: cilium-l2announce-gateway-cilium-gateway-shared - exp_annotations: - summary: "VPN ingress node is announcing a LoadBalancer VIP (cilium-l2announce-gateway-cilium-gateway-shared)" - description: >- - The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access - WireGuard, and it now holds the Cilium L2 announcement lease `cilium-l2announce-gateway-cilium-gateway-shared`. A - node cannot reach a VIP it announces itself, so every remote VPN client has just lost - all access to that VIP's Service -- silently, with DNS still resolving. Check that - platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that - the label is still on the node the router forwards udp/51820 to. - runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" - # The label IS present, so the watchdog must stay quiet here. - - eval_time: 40m - alertname: VpnIngressNodeLabelMissing - exp_alerts: [] - - # ============================================================================================= - # 2. THE FIXED STATE -- every lease is held by a node that is NOT the vpn-ingress node: QUIET. - # This is what l2-announcement-policy.yaml's exclusion produces, and the assertion that keeps - # test 1 meaningful rather than an alert that fires on anything. - # ============================================================================================= - - interval: 1m - name: all l2 leases held by other nodes - VpnIngressNodeAnnouncesL2VIP stays quiet - input_series: - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-2"}' - values: '1+0x40' - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-plex-plex-l4-tcp",lease_holder="k3s-prod-3"}' - values: '1+0x40' - # A lease the vpn-ingress node DOES legitimately hold, in a different family -- proves the - # `cilium-l2announce-.*` matcher is doing work. Every node holds operator/controller leases; - # only the ANNOUNCEMENT ones matter here, and a matcher broad enough to catch this would page - # on a healthy cluster forever. - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-operator-resource-lock",lease_holder="k3s-prod-1"}' - values: '1+0x40' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}' - values: '1+0x40' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}' - values: '1+0x40' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}' - values: '1+0x40' - alert_rule_test: - - eval_time: 40m - alertname: VpnIngressNodeAnnouncesL2VIP - exp_alerts: [] - - eval_time: 40m - alertname: VpnIngressNodeLabelMissing - exp_alerts: [] - - # ============================================================================================= - # 3. A LEGITIMATE, TRANSIENT LEASE MOVE -- the vpn-ingress node holds it for 11m during a node - # reboot or a Cilium roll, then Cilium re-elects away: QUIET. This pins the `for: 15m` as - # deliberate. Without it, every rolling restart of the cilium DaemonSet would page, and a paging - # alert that cries wolf on routine maintenance gets muted -- which is how the real collision - # would go unseen again. - # ============================================================================================= - - interval: 1m - name: transient lease move shorter than for 15m - VpnIngressNodeAnnouncesL2VIP stays quiet - input_series: - # Held by the vpn-ingress node for 11 samples, then the series goes absent (the lease moved, - # so kube-state-metrics retires this label set and starts a new one). - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}' - values: '1+0x10 _x30' - # Re-elected onto prod-3 from t=11m. A label VALUE change retires one series and starts - # another, which is why this is a second series and not a changing value. - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-3"}' - values: '_x11 1+0x29' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}' - values: '1+0x40' - alert_rule_test: - # Inside the transient window: pending at most, never firing. - - eval_time: 10m - alertname: VpnIngressNodeAnnouncesL2VIP - exp_alerts: [] - # Well past it, with the lease settled on another node. - - eval_time: 30m - alertname: VpnIngressNodeAnnouncesL2VIP - exp_alerts: [] - - # ============================================================================================= - # 4. THE WATCHDOG -- no node carries the label at all. The policy's `NotIn` then excludes - # NOTHING (a missing key matches), so the 2026-10-08 failure is fully re-armed, AND - # VpnIngressNodeAnnouncesL2VIP is structurally INERT: its join matches nothing and returns no - # series, which is indistinguishable from healthy. Both halves are asserted here, because an - # inert critical alert is the exact shape this repo has been burned by before. - # ============================================================================================= - - interval: 1m - name: vpn-ingress label absent everywhere - watchdog fires and the critical alert is inert - input_series: - # A lease held by the node that IS (in reality) the forwarded one -- but with no label - # anywhere, nothing can tell. This is the dangerous state. - - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}' - values: '1+0x60' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}' - values: '1+0x60' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}' - values: '1+0x60' - - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}' - values: '1+0x60' - alert_rule_test: - # The critical alert CANNOT fire -- this is the inertness being pinned, not an aside. - - eval_time: 45m - alertname: VpnIngressNodeAnnouncesL2VIP - exp_alerts: [] - # ... which is precisely why the watchdog must. - - eval_time: 45m - alertname: VpnIngressNodeLabelMissing - exp_alerts: - - exp_labels: - severity: warning - # absent() propagates the matcher's own labels onto the series it returns, so the - # alert inherits this one. Asserted rather than worked around: it is harmless, and - # pinning it means a future change to the matcher cannot alter the alert's identity - # (and therefore its Alertmanager grouping) unnoticed. - label_homelab_keithjackson_dev_vpn_ingress: "true" - exp_annotations: - summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF" - description: >- - No node publishes homelab.keithjackson.dev/vpn-ingress=true, so - l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none: - any node may now announce a LoadBalancer VIP, including the one terminating - remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so - the warning matters more than its severity suggests. Re-run the node_baseline role on - the forwarded node, and confirm the label is in kube-state-metrics' - metricLabelsAllowlist. - runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" diff --git a/platform/vpn-ingress/README.md b/platform/vpn-ingress/README.md index add6522..b0598be 100644 --- a/platform/vpn-ingress/README.md +++ b/platform/vpn-ingress/README.md @@ -117,19 +117,8 @@ boundary around this port is WireGuard's peer authentication, full stop. The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400 forward). A single-replica Deployment would float on every reschedule and silently invalidate it. A DaemonSet instead puts an identical listener on all three nodes — every node answers the same -keys — so losing the forwarded node never costs a re-issued client profile. The two idle listeners -cost ~15 MiB each. - -> **This section used to say "losing the forwarded node is a one-line router change, with no cluster -> work". It was wrong, and it cost two days of silently broken remote access on 2026-10-08.** The -> nodes are warm, but they are **not interchangeable**: the node the forward points at must be the -> one labelled `homelab.keithjackson.dev/vpn-ingress=true`, because that is the node -> `platform-config/cilium/l2-announcement-policy.yaml` excludes from announcing LoadBalancer VIPs. -> **A node cannot reach a VIP whose Cilium L2 announcement lease it holds itself** (see "Measured on -> prod, 2026-10-08" below), and since this pod MASQUERADEs clients to the node's own address, a VPN -> client inherits that. Failover is therefore **two** changes: move -> `node_baseline_vpn_ingress_node` and re-run that role, then re-point the router — -> `runbooks/vpn-remote-access.md` step 3. +keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no +re-issued client profiles. The two idle listeners cost ~15 MiB each. A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node @@ -197,44 +186,85 @@ right way; one was a defect in my own manifest. | Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. | | Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. | -## Measured on prod, 2026-10-08 — why the forwarded node cannot be any node +## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach -The first device enrolled, handshook, resolved names, and could not open a single connection to any -cluster web app. The tunnel was not at fault — every part of it measured healthy: -`net.ipv4.ip_forward=1`, `wg0` up at `10.90.0.1/24` MTU 1420, the peer authorised with real -bidirectional transfer (346 KiB in / 200 KiB out), `MASQUERADE` present and counting 26 flows, -`FORWARD -i/-o wg0 ACCEPT` counting 108 / 171 packets, and PiHole DNS answering from the node. +A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose +Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from +announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is +real but inert, and the distinction matters enough to record, because the wrong version is very +easy to re-derive from `ip neigh` alone. -**The fault was that a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it -holds itself.** Cilium's announcer answers ARP for the VIP on behalf of other LAN hosts, but not for -the announcing node's own request, so the neighbour entry settles at `FAILED`: +Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case, +after the UPS power-cycle — from the host netns of the `wireguard` pod on each node: -| from | `192.168.2.20` (Gateway) | `192.168.2.22` (crafty) | `192.168.2.24` (plex) | announces | -|---|---|---|---|---| -| `k3s-prod-1` | **FAILED** | — | — | `.20` | -| `k3s-prod-2` | ok | **INCOMPLETE** | ok | `.21`, `.22`, `.23-tcp` | -| `k3s-prod-3` | ok | ok | **INCOMPLETE** | `.24` | +| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) | +|---|---|---| +| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` | +| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms | +| `nc -z 192.168.2.20 443` | **OK** | OK | +| plex `192.168.2.24:32400` | **`200`** | `200` | -Symmetric across 3 nodes × 5 VIPs: each node reached every VIP **except** the ones it alone -announced. It is a property of L2 announcement, not of any one Service, and it is harmless for -ordinary pods — they reach Services through the cluster datapath and never ARP for a VIP. This -component is the exception, because `hostNetwork` + `MASQUERADE` put remote clients on the node's -own LAN path. +So the announcer really does **not** answer its own ARP request — that part of the observation is +reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**: -The forward pointed at `k3s-prod-1`, which held the `shared` Gateway lease, so **all 16 Gateway -hostnames were dead over the VPN**. Two things hid it: +* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and + `Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**, + so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before + any route or neighbour lookup for the VIP happens**. +* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the + shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without + a packet crossing the LAN at all. +* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`), + which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real** + service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes — + that is service-aware DNAT, not wire traffic. -* **PiHole survived by accident.** `192.168.2.23` is announced by *two* Services - (`pihole-dns-udp` on prod-1, `pihole-dns-tcp` on prod-2), so prod-2 answered the ARP prod-1 could - not. Names resolved perfectly and every app hung — the hardest shape to diagnose from a client. -* **Nothing reported it.** No probe in this repo compares the forwarded node against the lease - holders, and from outside WireGuard is silent to unauthenticated traffic anyway. +**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis: -Closed by labelling the forwarded node `homelab.keithjackson.dev/vpn-ingress=true` and excluding -that label in `platform-config/cilium/l2-announcement-policy.yaml` (the node then ARP-resolves every -VIP), with `VpnIngressNodeAnnouncesL2VIP` watching the lease holders so a recurrence is loud and -`VpnIngressNodeLabelMissing` watching for the label going absent, which would silently re-arm the -whole failure. Cost: VIP announcement candidacy drops from three nodes to two. +* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an + artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not + anything is wrong. +* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the + two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it + discriminates nothing — and it is not the client path either (see below). + +**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are +**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is +**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the +forwarded path in either direction: it reports success where the forwarded path might fail, and +`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with +a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE` +counters move. + +## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses + +`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the +component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature +to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel +client cannot reach the internet, nothing in the list below is the reason: + +| checked | result on all three nodes | +|---|---| +| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` | +| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change | +| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE | +| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present | +| default route | `default via 192.168.2.1 dev eno1 onlink`, intact | +| node's own internet reachability | `http://1.1.1.1/` returns `301` | + +**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is +closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by +`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled +`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity +traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway +node. A remote client's apparent public address is therefore the **home WAN IP**, which is the +intended behaviour and the thing to state when enrolling a device. + +**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one +enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and +the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the +power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the +forwarded node shows a peer with an `endpoint` and a recent `latest handshake`. ## Prerequisites outside this directory diff --git a/platform/vpn-ingress/daemonset.yaml b/platform/vpn-ingress/daemonset.yaml index 64a07e4..c1a203d 100644 --- a/platform/vpn-ingress/daemonset.yaml +++ b/platform/vpn-ingress/daemonset.yaml @@ -62,39 +62,9 @@ # as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to # whichever node the scheduler picked and silently invalidate that forward on every reschedule. A # DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of -# them, and if that node is lost every other node is already warm and answers the same keys, so no -# client profile ever changes. Each node binds udp/51820 in its own netns, so there is no conflict, -# and the two idle listeners cost ~15 MiB each. -# -# BUT THE NODES ARE NOT INTERCHANGEABLE, AND THIS COMMENT USED TO CLAIM THEY WERE. -# It said "FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work". That was wrong, and the -# wording cost two days of silently broken remote access on 2026-10-08. MEASURED that day: a node -# cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds ITSELF — the announcer -# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and packets to -# that VIP die before leaving the node: -# -# k3s-prod-1, holding the shared Gateway lease: ip neigh show 192.168.2.20 -> FAILED -# k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr …(prod-1) -# -# Because this pod MASQUERADEs clients to the node's own address (see PostUp below), a remote client -# inherits exactly that failure. With the forward on the Gateway's announcing node, all 16 Gateway -# hostnames were unreachable over the VPN while PiHole DNS still resolved — a tunnel that -# handshakes, resolves names, and then hangs on every single app. -# -# THE FIX IS DECLARATIVE AND LIVES OUTSIDE THIS FILE: the forwarded node is labelled -# homelab.keithjackson.dev/vpn-ingress=true by infra/ansible/roles/node_baseline (from -# node_baseline_vpn_ingress_node), and platform-config/cilium/l2-announcement-policy.yaml excludes -# that label from announcing ANY VIP, so the node can ARP-resolve all of them. A label rather than a -# cluster-vars knob because the alert that guards this has to read it too, and the alert unit-test -# harness refuses a postBuild placeholder in an expr. Failover is therefore TWO steps — move -# node_baseline_vpn_ingress_node and re-run the role, THEN re-point the router -# (runbooks/vpn-remote-access.md step 3). -# `VpnIngressNodeAnnouncesL2VIP` (platform/monitoring/alerts.yaml) fires if the two collide again. -# -# This pod deliberately does NOT probe that itself. A readinessProbe on VIP reachability would wedge -# a DaemonSet rollout behind an unrelated Gateway outage (maxUnavailable: 1 below), and a liveness -# probe would tear down a tunnel that is carrying LAN traffic perfectly well. The condition is a -# property of the CLUSTER's announcement layout, so it is watched there. +# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no +# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own +# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each. # # All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the # DaemonSet to cover the cluster; add them if a node is ever tainted. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index 7adc398..f01643a 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -119,11 +119,6 @@ configMapGenerator: - update-rollback=update-rollback.md - vpn-down=vpn-down.md - vpn-egress-stale=vpn-egress-stale.md - # `VpnIngressNodeAnnouncesL2VIP` — the INBOUND remote-access VPN, not the egress tunnel the two - # pages above cover. A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease - # it holds itself, and vpn-ingress MASQUERADEs clients to the node's own address, so the - # forwarded node must announce nothing (measured 2026-10-08). - - vpn-ingress-l2-collision=vpn-ingress-l2-collision.md # Operator procedure, not an alert page. Served because the component it completes is INERT # until three off-repo manual steps are done (kernel module, server key, WAN port-forward) — # and because the one check that actually proves it cannot be run from inside the house. diff --git a/runbooks/vpn-ingress-l2-collision.md b/runbooks/vpn-ingress-l2-collision.md deleted file mode 100644 index 38bb489..0000000 --- a/runbooks/vpn-ingress-l2-collision.md +++ /dev/null @@ -1,165 +0,0 @@ -# vpn-ingress-l2-collision — the VPN ingress node is announcing a LoadBalancer VIP - -**Alerts:** `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) and `VpnIngressNodeLabelMissing` -(warning, 30m) -**File:** `platform/monitoring/alerts.yaml` -**Related:** `/runbooks/vpn-remote-access` is the setup and enrolment runbook for this component. -`/runbooks/vpn-egress-stale` and `/runbooks/vpn-down` are the **other** VPN — the outbound PIA -tunnel — and have nothing to do with this. - -## What this means here - -**A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The -announcer answers ARP for the VIP on behalf of other LAN hosts, but not for the announcing node's -own request. On the lease holder the neighbour entry settles at `FAILED`, and every packet to that -VIP is dropped before it leaves the node. Measured on prod 2026-10-08, symmetric across all three -nodes and five VIPs — each node reached every VIP **except** the ones it alone announced. - -For an ordinary workload this is harmless: pods reach Services through the cluster datapath and -never ARP for a VIP. `platform/vpn-ingress` is the exception. It runs WireGuard with -`hostNetwork: true` and MASQUERADEs remote clients to the node's own address, so a remote client's -packets take that node's LAN path and inherit its ARP failures. **If the node the router forwards -`udp/51820` to is also an L2 lease holder, every remote VPN client silently loses all access to the -Services behind the VIPs it announces.** - -The node terminating WireGuard is marked with `homelab.keithjackson.dev/vpn-ingress=true`, and -`platform-config/cilium/l2-announcement-policy.yaml` excludes that label from announcing anything, -so this should be unreachable. - -**The symptom your users report will not sound like a network fault.** On 2026-10-08 the forward sat -on `k3s-prod-1`, which held the `shared` Gateway lease, so all 16 Gateway hostnames were dead over -the VPN — while PiHole DNS kept answering, because `192.168.2.23` is announced by *two* Services and -`k3s-prod-2` held the other lease. So the tunnel handshakes, `wg show` looks perfect, names resolve, -and **every app hangs on connect**. Expect "the VPN connects but nothing loads", not "the VPN is -down". - -Which Services are affected depends on which lease moved — read `{{ $labels.lease }}` from the -alert. `cilium-l2announce-gateway-cilium-gateway-shared` is the worst case: it is every web app at -once. - -### If it is `VpnIngressNodeLabelMissing` that fired - -That is the watchdog, and it means the protection is **switched off**, not that something is merely -untidy. The exclusion is a `NotIn` on the label, and `NotIn` with an absent key **matches** — so -with the label on no node, the policy excludes nothing and every node is an announcement candidate -again. `VpnIngressNodeAnnouncesL2VIP` is simultaneously **inert**: its join matches nothing and -returns no series, which is indistinguishable from healthy. Treat a quiet critical alert as -meaningless until this warning is cleared. Usual cause: a node was rebuilt and `node_baseline` has -not been re-run on it. - -## First commands - -Run cluster commands through the pinned toolbox, which supplies the kube context: - -```bash -# 1. WHO HOLDS WHAT, and WHO IS THE VPN NODE — the whole question, in two lines. -./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n kube-system get lease | grep l2announce' -./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/vpn-ingress' -# Exactly ONE node should carry vpn-ingress=true, and NO l2announce lease should name it. - -# 2. IS THE EXCLUSION STILL IN THE LIVE POLICY? Expect a matchExpressions list with a -# homelab.keithjackson.dev/vpn-ingress NotIn [true] entry — not a bare matchLabels. -./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod get ciliuml2announcementpolicy default-l2 -o jsonpath="{.spec.nodeSelector}"; echo' - -# 3. IS THE LABEL REACHING PROMETHEUS? If this is empty the alerts are inert even when -# the label is on the node (kube-state-metrics metricLabelsAllowlist). -./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n monitoring exec deploy/kube-prometheus-stack-kube-state-metrics -- \ - wget -qO- localhost:8080/metrics | grep vpn_ingress | head' -``` - -**Confirm the actual damage**, which is the one measurement that is unambiguous. From the WireGuard -pod on the affected node (it shares the host netns, so this is the path a VPN client takes): - -```bash -./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n vpn-ingress get pods -o wide' # pick the pod on that node - -./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n vpn-ingress exec -c wireguard -- \ - sh -c "ping -c1 -W1 192.168.2.20 >/dev/null 2>&1; ip neigh show 192.168.2.20"' -# BROKEN: 192.168.2.20 dev eno1 FAILED -# HEALTHY: 192.168.2.20 dev eno1 lladdr bc:e9:2f:7e:c4:a6 STALE|REACHABLE|DELAY -``` - -The `ping` is only there to trigger ARP — the VIP does not answer ICMP even when healthy, so -**ignore its 100% packet loss**; the neighbour state is the answer. Substitute the VIP that belongs -to `{{ $labels.lease }}` if the Gateway is not the one that moved. - -**Which node actually holds the forward** (the router cannot be queried, but the cluster knows): -only the forwarded node's `wg show` has a peer with an `endpoint` and a recent `latest handshake`. - -```bash -./infra/devcontainer/dev.sh run bash -c \ - 'for p in $(kubectl --context k3s-prod -n vpn-ingress get pods -o name); do - echo "== $p"; kubectl --context k3s-prod -n vpn-ingress exec ${p#pod/} -c wireguard -- wg show; done' -``` - -## Likely causes, most likely first - -1. **The label is on the wrong node** — someone re-pointed the router's forward during a node - failover and did not move `node_baseline_vpn_ingress_node`. This is the most likely cause by a - wide margin, because the router is off-repo and nothing can check it. Confirm with the `wg show` - loop above: the node with the live peer endpoint is the real forward target. - *Fix:* merge a PR setting `node_baseline_vpn_ingress_node` - (`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`) to that node, then re-run the role — - `ansible-playbook -i inventory/hosts.ini -e k3s_hosts=k3s_prod --tags vpn-ingress`. - The role labels the new node and **removes** the label from the old one. Then confirm Cilium - re-elected the lease off the newly-labelled node. - -2. **The exclusion was edited out of `platform-config/cilium/l2-announcement-policy.yaml`**, or the - file was reverted to a bare `matchLabels: {kubernetes.io/os: linux}`. Command 2 above shows this - directly. *Fix:* restore the `matchExpressions` form; the file's own header explains why it is - shaped that way. - -3. **The label is missing entirely** (`VpnIngressNodeLabelMissing` firing) — a node rebuild without - a `node_baseline` re-run. *Fix:* re-run the role as above. - -4. **The label is on the node but not in Prometheus** — `metricLabelsAllowlist` in - `platform/monitoring/kube-prometheus-stack.yaml` lost the `vpn-ingress` entry. Then the critical - alert is inert *and* the watchdog fires. Command 3 above distinguishes this from cause 3. - -5. **Cilium kept renewing a lease its holder is no longer eligible for.** The exclusion is correct - and in place, but the lease never re-elected. This was never verified when the fix was authored - (see `decisions/2026-10-17-vpn-ingress-l2-lease-collision.md`). *Fix:* delete that one Lease - object to force an election — `kubectl -n kube-system delete lease ` — and watch it come - back on a different node. - -6. **A Cilium upgrade changed `NotIn` semantics or the `nodeSelector` field.** Least likely, but it - is the reason this is a runtime alert rather than a static check on the policy text. - -## When to escalate or stop - -**Act now, don't wait for a window.** Remote access is fully broken for the affected VIPs while this -is firing, and it does **not** self-heal: a lease holder keeps renewing, so this condition persists -indefinitely once entered. That is also why the 15m `for:` is safe — a transient move during a node -reboot or a Cilium roll resolves on its own well inside it. - -**The fastest honest mitigation is to move the forward, not the leases.** Re-pointing the router's -`udp/51820` at a node that holds no lease for the affected VIP restores service in seconds, with no -cluster change and no new client profiles — all three nodes answer the same keys. Follow it with the -`node_baseline_vpn_ingress_node` PR and role re-run so the two agree again; until you do, this alert -is watching the wrong node. Note each node's own announcements when choosing: as of 2026-10-08, -prod-2 announced crafty `192.168.2.22` and unifi `192.168.2.21`, prod-3 announced plex -`192.168.2.24`, so neither is a cost-free target. - -**Do not** `kubectl edit` the `CiliumL2AnnouncementPolicy` to fix this — Flux owns it and will revert -you, and the drift is the kind that outlives the memory of making it. The two acceptable hand -actions are deleting a single Lease to force re-election (cause 5), and `kubectl label node` as a -*temporary* stopgap while the PR lands — note that a node rebuild or a `node_baseline` run will undo -a hand-applied label, so it is not a fix. - -**Stop and escalate to the operator if** the leases will not leave the excluded node after a forced -re-election, or if the intended forward target is unclear. Both need a decision about which node -carries remote access, which is an operator call, not a runbook step. - -**Known gap, so you are not misled:** if someone re-points the router without moving the label, -`VpnIngressNodeAnnouncesL2VIP` watches the wrong node and **stays silent** while the VPN breaks in -exactly the way described above. There is no in-cluster signal for that today; closing it needs a -WireGuard exporter publishing which node is actually terminating peer traffic -(`platform/vpn-ingress/README.md`, "Not built") — the `wg show` loop above is the manual substitute. -Until then, if remote access is broken and both alerts are quiet, **check the forward against the -label by hand before believing the alerts.** diff --git a/runbooks/vpn-remote-access.md b/runbooks/vpn-remote-access.md index dfdd874..cba1887 100644 --- a/runbooks/vpn-remote-access.md +++ b/runbooks/vpn-remote-access.md @@ -171,57 +171,14 @@ server without a `wg0` "ready". On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g. `192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check. -> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node. -> -> This step used to say "pick any node — all three run an identical listener". **That sentence -> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a -> cluster-vars knob instead of a free choice here. -> -> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The -> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease -> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they -> leave the node. Measured on prod 2026-10-08: -> -> ```text -> on k3s-prod-1 (held the shared Gateway lease): ip neigh show 192.168.2.20 -> FAILED -> on k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr … (prod-1) -> ``` -> -> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN -> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway -> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so -> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported -> it. -> -`infra/ansible/roles/node_baseline` now labels the forwarded node -> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in -> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes -> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI -> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a -> labelled node taking a lease, but it CANNOT catch the forward being moved without the label. - -Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`). -Note it here: +Pick any node — all three run an identical listener. Note which one you chose here: ``` # WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set -# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml ``` -**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that -was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and -reintroduces the failure above against a different set of addresses (prod-2 announces crafty and -unifi; prod-3 announces plex). Do both, in this order: - -1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role - (`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium - then moves the leases off the newly-labelled node and onto its peers. -2. **Re-point the router** forward at the new node's address. - -Every device keeps working with the same profile either way; all three nodes answer the same keys, -and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that -handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because -that alert watches the node the knob names, not the node your router actually points at. +**Node loss = failover is this one line.** Re-point the forward at another node address; every +device keeps working with the same profile, because all three nodes answer the same keys. Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is already silent to unauthenticated probes (README, "Why this is safe to expose"). @@ -316,6 +273,84 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test. by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes if the router is not delivering the packets, and a `handshake` with no completed session if it is. +## Triage — "the client reaches the LAN and cluster apps, but not the internet" + +Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom +almost always lives, and because the server side has been measured and is not the cause. + +**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.** + +| the profile says | meaning | +|---|---| +| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. | +| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. | + +`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit — +`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client +profile only**, so an already-issued profile keeps whatever it was rendered with until it is +re-rendered and re-imported on the device. The two forms differ by one flag: + +```bash +./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh ' # full tunnel +./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh --split' # LAN only +``` + +Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into +`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is +editing `AllowedIPs` in the profile **on the device** — the server does not care. + +**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the +client connected, then watch the two counters that only forwarded traffic moves: + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c " + wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5; + iptables -t nat -L POSTROUTING -n -v | head -5"' +``` + +* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which + node the router's `udp/51820` forward points at (the router is unreadable from the cluster). +* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is + upstream of the cluster (router forward, client, or carrier), not here. +* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side + failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there, + which breaks internet while node-local cluster VIPs still work. It has not been observed. + +**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly: + +* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack + client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel — + so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4: + `curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it, + and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is + blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it). +* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified + 2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a + blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on + a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all. + +**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the +`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain +**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route +`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the +PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled +`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the +**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6). + +## Triage — which L2 announcement lease a node holds does NOT matter + +Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node +could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`: +that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh` +for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer +VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend +is a node-local socket. Full measurements and the two misleading probes +(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`. + +`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a +separate, pre-existing fault in that service and has nothing to do with the VPN. + ## Revoking a device Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the