From f5353cbd30ed90bfa2512fc314e99634ca08cbe6 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 17 Oct 2026 21:55:25 -0500 Subject: [PATCH] fix(vpn-ingress): the VPN ingress node must announce no L2 VIPs, or clients route nothing A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself: the announcer does not answer its own ARP request, so on the lease holder the neighbour entry settles at FAILED and packets to that VIP die before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced. platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and MASQUERADEs remote clients to the node's own address, so a VPN client takes the node's LAN path and inherits its ARP failures. The router's udp/51820 forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16 Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering (its VIP has two Services and k3s-prod-2 held the other lease). The tunnel handshook, names resolved, every app hung, and nothing reported it. on k3s-prod-1: ip neigh show 192.168.2.20 -> FAILED (its own VIP) on prod-2/3: ip neigh show 192.168.2.20 -> lladdr ... (prod-1), 443 OK Mark the node and exclude it from announcing: * node_baseline labels the forwarded node (node_baseline_vpn_ingress_node, k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes the label from any node that no longer matches so moving it MOVES the label. * l2-announcement-policy.yaml excludes that label, so the node ARP-resolves every VIP. Fixes all VIPs for VPN clients, not just the one that collided. Cost: announcement candidacy drops from 3 nodes to 2. * VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels (label added to the kube-state-metrics allowlist), with VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches, so a missing label would silently re-arm the whole failure AND leave the critical alert inert. Four promtool cases, including that inertness. A node LABEL rather than a cluster-vars knob because the alert must read it too, and extract_rules.py rightly refuses a ${...} placeholder in an alert expr -- the rule unit-tested would not be the rule deployed. One source of truth, and it touches no sensitive path. Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and "failover is this one line". Both false; the first caused this. UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the handoff entry's third section. Co-Authored-By: Claude Opus 5 (1M context) --- ...26-10-17-vpn-ingress-l2-lease-collision.md | 137 ++++++++++++ .../inventory/group_vars/k3s_prod/vars.yml | 19 ++ .../roles/node_baseline/defaults/main.yml | 27 +++ .../roles/node_baseline/tasks/main.yml | 42 ++++ .../cilium/l2-announcement-policy.yaml | 69 +++++- platform/monitoring/alerts.yaml | 91 ++++++++ .../monitoring/kube-prometheus-stack.yaml | 14 +- .../monitoring/tests/vpn-ingress-l2.test.yaml | 203 ++++++++++++++++++ platform/vpn-ingress/README.md | 54 ++++- platform/vpn-ingress/daemonset.yaml | 36 +++- runbooks/kustomization.yaml | 5 + runbooks/vpn-ingress-l2-collision.md | 165 ++++++++++++++ runbooks/vpn-remote-access.md | 49 ++++- 13 files changed, 900 insertions(+), 11 deletions(-) create mode 100644 decisions/2026-10-17-vpn-ingress-l2-lease-collision.md create mode 100644 platform/monitoring/tests/vpn-ingress-l2.test.yaml create mode 100644 runbooks/vpn-ingress-l2-collision.md diff --git a/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md b/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md new file mode 100644 index 0000000..b21be0e --- /dev/null +++ b/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md @@ -0,0 +1,137 @@ +--- +date: 2026-10-17 +branch: fix/vpn-ingress-l2-lease-collision +title: "A node cannot reach a VIP it announces — so the VPN ingress node must announce nothing" +--- + +## What changed + +The first device enrolled on the remote-access VPN handshook, resolved names, and could not open a +single connection to any cluster web app. The tunnel was not at fault. The fault is a property of +Cilium L2 announcement that nothing in this repo knew about: + +**A node cannot reach a LoadBalancer VIP whose `CiliumL2AnnouncementPolicy` lease it holds itself.** +The announcer answers ARP for the VIP on behalf of other LAN hosts but not for its own request, so +on the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP is dropped +before it leaves the node. + +`platform/vpn-ingress` is uniquely exposed because it runs WireGuard with `hostNetwork: true` and +MASQUERADEs remote clients to the node's own address — a VPN client's packets take the node's LAN +path and inherit its ARP failures. Ordinary pods never do. The router's `udp/51820` forward pointed +at `k3s-prod-1`, which held `cilium-l2announce-gateway-cilium-gateway-shared`, so **all 16 `shared` +Gateway hostnames were dead over the VPN** — while PiHole DNS kept answering, because +`192.168.2.23` is announced by *two* Services and `k3s-prod-2` held the other lease. Names resolved +and every app hung on connect. + +Seven files: + +* **`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`** — new + `node_baseline_vpn_ingress_node`; the matching node gets + `homelab.keithjackson.dev/vpn-ingress=true`, and a second task **removes** the label from any node + that no longer matches, so moving the value moves the label rather than adding a second one. Same + shape as `roles/vpn_egress_gateway`'s `egress-gateway` label. Tagged `vpn-ingress`; skipped + entirely when the value is empty. +* **`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`** — set to `k3s-prod-1`, with a + `[parity]` note that it must equal the router's forward. +* **`platform-config/cilium/l2-announcement-policy.yaml`** — `nodeSelector` becomes a + `matchExpressions` list keeping `kubernetes.io/os: linux` and adding + `homelab.keithjackson.dev/vpn-ingress NotIn ["true"]`. The VPN node then holds no lease, so it + ARP-resolves every VIP. This fixes **all** VIPs for VPN clients (Gateway, crafty, plex, pihole, + unifi), not just the one that happened to collide. +* **`platform/monitoring/kube-prometheus-stack.yaml`** — the label added to kube-state-metrics' + `metricLabelsAllowlist`, which is what lets the invariant be watched at all. +* **`platform/monitoring/alerts.yaml`** — `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) joins + `kube_lease_owner` against `kube_node_labels` via `label_replace`, plus + `VpnIngressNodeLabelMissing` (warning, 30m) as its watchdog. +* **`platform/monitoring/tests/vpn-ingress-l2.test.yaml`** — four promtool cases: fires on the real + collision, quiet in the fixed state, quiet across a transient 11-minute lease move (pinning the + `for: 15m`), and the label-absent case asserting the critical alert is inert *and* the watchdog + fires. +* **`runbooks/vpn-ingress-l2-collision.md`** (new, registered in `runbooks/kustomization.yaml`), + **`runbooks/vpn-remote-access.md`**, **`platform/vpn-ingress/{daemonset.yaml,README.md}`** — step 3 + of the setup runbook said *"pick any node — all three run an identical listener"* and *"node loss = + failover is this one line"*. Both were false and the first caused this. Failover is now documented + as two ordered changes. + +**Design note — why a node label and not a `cluster-vars` knob.** The first attempt added +`WG_INGRESS_NODE` to `clusters/base/cluster-vars.yaml` and substituted it into both the policy and +the alert. The alert half is **impossible**: `platform/monitoring/tests/extract_rules.py` rejects any +alert whose `expr` contains a `${...}` placeholder, because the rule it would unit-test is not the +rule deployed. Rather than duplicate the node name into the alert as a literal (two sources that can +skew), the label is readable from both sides — the selector matches on it and `kube_node_labels` +carries it into Prometheus. One source of truth, and as a side effect the change touches no +sensitive path, so it merges on a green gate rather than needing a human review. + +The DaemonSet deliberately gained **no** probe for this. A readinessProbe on VIP reachability would +wedge a rollout behind an unrelated Gateway outage (`maxUnavailable: 1`); a liveness probe would tear +down a tunnel carrying LAN traffic correctly. + +## What I verified + +Measured live against prod, `--context k3s-prod`, on 2026-10-08: + +* **The whole tunnel is healthy**, which is what made this hard: `wg show` on `k3s-prod-1` shows the + peer with a real endpoint and bidirectional transfer (346 KiB in / 200 KiB out); + `net.ipv4.ip_forward=1`; `wg0` up at `10.90.0.1/24` MTU 1420; `iptables -t nat -S POSTROUTING` + carries the `MASQUERADE` rule with 26 packets; `FORWARD -i/-o wg0` ACCEPT with 108 / 171 packets; + DaemonSet generation 2, 3/3 ready on the content-hashed `wg-peers-m9kcckcb5t`. +* **The ARP failure and its symmetry.** On `k3s-prod-1`: `ip neigh show 192.168.2.20` → `FAILED`, + while `.21`/`.23` resolve to prod-2's MAC `04:0e:3c:96:b0:94`. On prod-2/prod-3, `.20` resolves to + prod-1's `bc:e9:2f:7e:c4:a6`. TCP agrees: `.20:443` and `.20:80` FAIL from prod-1, OK from the + other two. Repeated across 3 nodes × 5 VIPs — each node reached every VIP except the ones it alone + announced (prod-2 ✗ `.22`/`.21`; prod-3 ✗ `.24`). +* **Not an artefact of probing from the host netns.** `curl --interface wg0` from prod-1 fails the + same way, and a neighbour entry is node-global state, so the forwarded path cannot differ in ARP + resolution. +* **The Gateway itself was never broken:** from the workstation (192.168.2.133) + `home.keithjackson.dev` → `200`, `plex.home.keithjackson.dev` → `401`, both at `192.168.2.20`. + From prod-1 the router (`.1`) and TrueNAS (`.30`) are `REACHABLE`. +* **`NotIn` semantics, empirically** — the basis for the fail-back claim in the policy's comment: + `kubectl get nodes -l 'kubernetes.io/hostname notin (k3s-prod-1)'` returns prod-2 and prod-3, and + `NotIn` on a key no node carries returns **all three**. So an unapplied label excludes nothing and + degrades to the old behaviour rather than taking every VIP off the LAN. +* **The alert metric is really scraped:** `kube_lease_owner` is served by kube-state-metrics (55 + series, every `cilium-l2announce-*` lease with its holder in `lease_holder`), and `kube_node_labels` + already carries the two `egress-*` labels through the same allowlist (`k3s-prod-3` shows + `egress-gateway=true`), so the third entry follows a proven path. +* **The four promtool cases pass** (`platform/monitoring/tests/run.sh` → `SUCCESS`, 55 rules + checked), including that `VpnIngressNodeAnnouncesL2VIP` is inert when the label is absent. +* All edited YAML parses; the rendered `nodeSelector` is the intended two-expression list. + +## What I did NOT verify + +* **The fix is UNAPPLIED and therefore UNPROVEN end to end.** Applying it needs a mutating + `kubectl`/`flux reconcile` plus an Ansible run, which are operator actions (CLAUDE.md §5). I have + **not** observed `k3s-prod-1` resolve `192.168.2.20` after the exclusion lands, nor a VPN client + reaching an app. The prediction rests on the measured symmetry (non-announcing nodes resolve every + VIP, 15/15 data points), not on a test of the changed state. **Verify after merge, in this order:** + re-run `node_baseline --tags vpn-ingress`; `kubectl get nodes -L + homelab.keithjackson.dev/vpn-ingress` (exactly one node); `kubectl -n kube-system get lease | grep + l2announce` (no holder is that node); `ip neigh show 192.168.2.20` from the prod-1 wireguard pod + (expect a `lladdr`, not `FAILED`); then a client. +* **That Cilium actually releases a lease when its holder stops matching the `nodeSelector`.** I + assumed re-election and did not test it. If Cilium keeps renewing, the affected Leases must be + deleted once by hand to force an election. Check this first if the fix appears not to work. +* **The brief interruption on apply.** Moving the Gateway and `pihole-dns-udp` leases off prod-1 will + blip `192.168.2.20` and `192.168.2.23` for as long as re-election plus gratuitous ARP takes. Not + measured, and I did not test whether in-flight LAN sessions survive it. +* **The Ansible tasks have never been executed.** They parse as YAML and follow + `roles/vpn_egress_gateway`'s shape, but neither the label task nor the label-removal task has run, + so the `changed_when`/`failed_when` conditions and the `k3s kubectl label ... -` removal path are + unproven. The removal task's `failed_when` tolerates `'not found'` in stderr; that string was + assumed from kubectl's behaviour, not observed. +* **The residual off-repo coupling is NOT closed and cannot be, as built.** The router's forward is + unreadable from the cluster. If someone re-points it without moving the label, the alert watches + the wrong node and stays silent while the VPN breaks identically against a different set of VIPs. + Only the runbook guards it. Closing it needs a WireGuard exporter publishing which node is + actually terminating peer traffic (`wg show` knows — only the forwarded node has a peer + `endpoint`); listed as "Not built" in the component README and **recommended follow-up, not done + here**. +* **Why Cilium behaves this way** — intended, a known upstream issue, or fixed in a later version. I + measured the behaviour and did not read Cilium's source or issue tracker, so I cannot say whether + an upgrade would remove the need for this exclusion. +* **`192.168.2.21:8443` (unifi) fails from all three nodes**, including ones where ARP resolves, so + it is a different, pre-existing fault. Not investigated, not addressed. +* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 offline + subset only; none of the above is covered by a test that runs in the gate. +* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b). diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index 2b865a6..d286c9d 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -420,3 +420,22 @@ flux_bootstrap_env_path: "clusters/prod" # rather than leaving an inert drop-in behind, so this group-wide `true` is still correct # for it. etcd_io_priority_enabled: true + +# --- The node terminating remote-access WireGuard — DECIDED (measured 2026-10-08) --------------- +# k3s-prod-1 (192.168.2.10) is what the SOHO router forwards WAN udp/51820 to, confirmed from the +# cluster rather than assumed: only prod-1's `wg show` had a peer with an `endpoint` and a recent +# handshake, the other two listeners were idle. +# +# node_baseline labels it `homelab.keithjackson.dev/vpn-ingress=true`, and +# platform-config/cilium/l2-announcement-policy.yaml excludes that label from announcing any +# LoadBalancer VIP -- because a node CANNOT reach a VIP whose Cilium L2 announcement lease it holds +# itself (the announcer does not answer its own ARP request; the neighbour entry goes to `FAILED`). +# prod-1 held the `shared` Gateway lease, and since vpn-ingress MASQUERADEs clients to the node's +# own address, all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved. +# +# [parity] MUST equal the node named by the router's forward (runbooks/vpn-remote-access.md step 3). +# Nothing in CI can read that forward, so re-pointing it means changing BOTH: this value plus a +# node_baseline re-run, and the router line. `VpnIngressNodeAnnouncesL2VIP` catches the case where +# the label lands on a node that then takes a lease; it CANNOT catch the forward being moved +# without this value following, which is why the runbook spells the two-step out. +node_baseline_vpn_ingress_node: k3s-prod-1 diff --git a/infra/ansible/roles/node_baseline/defaults/main.yml b/infra/ansible/roles/node_baseline/defaults/main.yml index b6a7f4b..ae7a493 100644 --- a/infra/ansible/roles/node_baseline/defaults/main.yml +++ b/infra/ansible/roles/node_baseline/defaults/main.yml @@ -259,3 +259,30 @@ node_baseline_dns_options: # Written as ONE marked block, so removing an entry here removes it from the node, and # an empty list removes the block entirely. node_baseline_hosts_pins: [] + +# --- The node that terminates remote-access WireGuard (tags: vpn-ingress) ---------------------- +# Hostname of the node the SOHO router forwards WAN udp/51820 to, i.e. the one platform/vpn-ingress +# actually serves remote clients from. Set it in group_vars for the environment; EMPTY by default, +# because dev has no such forward and because a wrong value is worse than none. +# +# WHAT IT IS FOR, AND WHY IT IS NOT COSMETIC. The matching node gets +# `homelab.keithjackson.dev/vpn-ingress=true`, and +# platform-config/cilium/l2-announcement-policy.yaml EXCLUDES that label from announcing any +# LoadBalancer VIP. MEASURED on prod 2026-10-08: a node cannot reach a VIP whose Cilium L2 +# announcement lease it holds itself (the announcer does not answer its own ARP request -- the +# neighbour entry goes to `FAILED`). Because vpn-ingress runs on `hostNetwork` and MASQUERADEs +# clients to the node's own address, a VPN client inherits that: with the forward on the node +# holding the `shared` Gateway lease, all 16 Gateway hostnames were dead over the VPN while PiHole +# DNS still resolved. Names worked, every app hung, nothing alerted. +# +# [parity] MUST name the node the router's forward actually points at. The forward is off-repo and +# no check can read it (runbooks/vpn-remote-access.md step 3). Changing the forward is therefore a +# two-part change: this value (then re-run the role), and the router line. +# +# Only ONE node should ever carry the label: it removes that node from VIP announcement candidacy, +# so labelling all three would leave nothing announcing any VIP at all. The task below removes the +# label from a node that no longer matches, so moving the value moves the label. +node_baseline_vpn_ingress_node: "" +node_baseline_vpn_ingress_label_key: homelab.keithjackson.dev/vpn-ingress +# k3s ships its kubeconfig here; every prod node is a k3s server, so `k3s kubectl` works locally. +node_baseline_vpn_ingress_kubeconfig: /etc/rancher/k3s/k3s.yaml diff --git a/infra/ansible/roles/node_baseline/tasks/main.yml b/infra/ansible/roles/node_baseline/tasks/main.yml index 1422573..de317f4 100644 --- a/infra/ansible/roles/node_baseline/tasks/main.yml +++ b/infra/ansible/roles/node_baseline/tasks/main.yml @@ -1109,3 +1109,45 @@ state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}" backup: true tags: [dns, hosts] + +# ── vpn-ingress node label — tag `vpn-ingress` ──────────────────────────────────────────────── +# Marks the node terminating remote-access WireGuard so Cilium's L2 announcement policy can EXCLUDE +# it from announcing LoadBalancer VIPs (a node cannot reach a VIP it announces itself -- see +# defaults/main.yml). Same shape as roles/vpn_egress_gateway's egress-gateway label, and for the +# same reason: the label is the one place BOTH a nodeSelector and a PromQL alert can read it from. +# +# Skipped entirely when node_baseline_vpn_ingress_node is empty (dev, or before the forward exists), +# so this adds nothing to an environment that has no WAN forward. +- name: Label this node as the vpn-ingress node + ansible.builtin.command: + cmd: >- + k3s kubectl label node {{ inventory_hostname }} + {{ node_baseline_vpn_ingress_label_key }}=true --overwrite + environment: + KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}" + register: node_baseline_vpn_ingress_label_apply + changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_apply.stdout" + when: + - node_baseline_vpn_ingress_node | length > 0 + - inventory_hostname == node_baseline_vpn_ingress_node + tags: [vpn-ingress] + +# The other half: moving the knob must MOVE the label, not add a second one. Two labelled nodes +# would remove two of three nodes from announcement candidacy; three would leave no announcer at +# all and take every VIP off the LAN. +- name: Remove the vpn-ingress label from a node that no longer holds the forward + ansible.builtin.command: + cmd: >- + k3s kubectl label node {{ inventory_hostname }} + {{ node_baseline_vpn_ingress_label_key }}- + environment: + KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}" + register: node_baseline_vpn_ingress_label_remove + changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_remove.stdout" + failed_when: + - node_baseline_vpn_ingress_label_remove.rc != 0 + - "'not found' not in node_baseline_vpn_ingress_label_remove.stderr" + when: + - node_baseline_vpn_ingress_node | length > 0 + - inventory_hostname != node_baseline_vpn_ingress_node + tags: [vpn-ingress] diff --git a/platform-config/cilium/l2-announcement-policy.yaml b/platform-config/cilium/l2-announcement-policy.yaml index d65e0ff..c9229f5 100644 --- a/platform-config/cilium/l2-announcement-policy.yaml +++ b/platform-config/cilium/l2-announcement-policy.yaml @@ -16,6 +16,71 @@ spec: externalIPs: false interfaces: - ^e(n|th) + # =============================================================================================== + # WHY THIS IS A matchExpressions LIST AND NOT `matchLabels: {kubernetes.io/os: linux}` + # =============================================================================================== + # The second expression excludes every node labelled `homelab.keithjackson.dev/vpn-ingress=true` + # from announcing ANY LoadBalancer VIP. That is the node the SOHO router forwards WAN udp/51820 to. + # It is not tuning; it is the fix for a measured, silent, total failure of remote access. + # + # MEASURED ON PROD 2026-10-08: A NODE CANNOT REACH A VIP IT ANNOUNCES ITSELF. Cilium's L2 + # announcer answers ARP for the VIP on behalf of OTHER LAN hosts, but not for the announcing node + # itself. On the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP + # is dropped before it leaves the node: + # + # k3s-prod-1, holding cilium-l2announce-gateway-cilium-gateway-shared: + # ip neigh show 192.168.2.20 -> 192.168.2.20 dev eno1 FAILED (its own VIP) + # nc -z 192.168.2.20 443 -> FAIL nc -z 192.168.2.20 80 -> FAIL + # k3s-prod-2 and k3s-prod-3, holding no lease for it: + # ip neigh show 192.168.2.20 -> lladdr bc:e9:2f:7e:c4:a6 (prod-1) nc 443/80 -> OK + # + # Symmetric across all 3 nodes x 5 VIPs: every node reached every VIP EXCEPT the ones it alone + # announced (prod-2 could not reach crafty 192.168.2.22 or unifi 192.168.2.21; prod-3 could not + # reach plex 192.168.2.24). So this is a property of L2 announcement, not of one Service. + # + # WHY IT ONLY HURTS THE VPN NODE. For an ordinary workload this is harmless: pods reach Services + # through the cluster datapath and never ARP for a VIP. platform/vpn-ingress is the exception -- + # it runs WireGuard with `hostNetwork: true` and MASQUERADEs remote clients to the node's own + # address, so a VPN client's packets take the node's LAN path and inherit exactly this ARP + # failure. On 2026-10-08 the forward pointed at k3s-prod-1, which held the Gateway lease, and all + # 16 Gateway hostnames were unreachable over the VPN while PiHole DNS still answered -- so names + # resolved and every app hung. Nothing in the cluster reported it; the alerts below now do. + # + # WHY A NODE LABEL AND NOT A cluster-vars KNOB. A knob would put the node's identity in git, which + # is more reviewable -- but it CANNOT BE USED BY THE ALERT that guards this, and an unguarded + # invariant here is worth less than a reviewable one. The alert harness + # (platform/monitoring/tests/extract_rules.py) REFUSES any alert whose `expr` contains a + # dollar-brace postBuild placeholder (spelled without braces here ON PURPOSE: this file is itself + # envsubst'd, and a literal dollar-brace-dot-dot-dot is UNPARSEABLE to envsubst, which fails the + # WHOLE Kustomization -- ci: flux-postbuild-substitution caught exactly this comment), because the + # rule it would unit-test is not the rule that gets + # deployed. A label is readable from BOTH sides: this selector matches on it, and + # `kube_node_labels` carries it into Prometheus so `VpnIngressNodeAnnouncesL2VIP` can join lease + # holders against it. One source of truth, no git-vs-cluster skew, and the policy stays + # env-agnostic as its header above claims. The label is set by + # infra/ansible/roles/node_baseline (node_baseline_vpn_ingress_node), the same shape + # roles/vpn_egress_gateway already uses for `egress-gateway`. + # + # THE COST, STATED: VIP announcement candidacy drops from 3 nodes to 2. That is still redundant, + # and the trade is deliberate -- the excluded node is the one whose *outbound* reach to every VIP + # matters most, because it carries every remote client. If BOTH remaining nodes are down the VIPs + # go dark where previously one node might have served them, but at that point the cluster has one + # node left and far worse problems. + # + # `NotIn` with an ABSENT key MATCHES (k8s apimachinery: a missing label is "not in the set"), so + # the other two nodes need no label and nothing has to be applied to them. If the label is never + # applied anywhere, this excludes nothing and the policy degrades to its pre-2026-10-08 behaviour + # -- the status quo, not an outage: it fails back, not down. That silent-degradation path is + # exactly why `VpnIngressNodeLabelMissing` exists as a watchdog: without it, the alert below + # would be structurally INERT (its join would match nothing) and would look identical to healthy. nodeSelector: - matchLabels: - kubernetes.io/os: linux + matchExpressions: + - key: kubernetes.io/os + operator: In + values: + - linux + # The node terminating remote-access WireGuard must announce nothing -- see above. + - key: homelab.keithjackson.dev/vpn-ingress + operator: NotIn + values: + - "true" diff --git a/platform/monitoring/alerts.yaml b/platform/monitoring/alerts.yaml index e007f2d..ef18519 100644 --- a/platform/monitoring/alerts.yaml +++ b/platform/monitoring/alerts.yaml @@ -1147,6 +1147,97 @@ spec: metricLabelsAllowlist. runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale" # ----------------------------------------------------------------------------------------- + # REMOTE-ACCESS VPN (platform/vpn-ingress) -- NOT the egress tunnel above. Different + # direction, different workload: this is inbound WireGuard for remote devices. + # ----------------------------------------------------------------------------------------- + - alert: VpnIngressNodeAnnouncesL2VIP + # THE FAILURE THIS CATCHES. MEASURED on prod 2026-10-08: a node CANNOT reach a + # LoadBalancer VIP whose Cilium L2 announcement lease it holds itself -- the announcer + # does not answer its own ARP request, so the neighbour entry settles at `FAILED` and + # packets die before leaving the node (`ip neigh show 192.168.2.20` -> FAILED on the + # lease holder, resolved on both others). platform/vpn-ingress runs WireGuard on + # `hostNetwork` and MASQUERADEs remote clients to the node's own address, so if the node + # the router forwards udp/51820 to is also a lease holder, VPN clients cannot reach the + # VIPs it announces. On 2026-10-08 that was the `shared` Gateway: all 16 app hostnames + # dead over the VPN, while PiHole DNS kept answering (its VIP has two Services and the + # other lease sat on prod-2) -- so the tunnel handshook, names resolved, and every app + # hung. Nothing reported it for ~2 days. + # + # platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled + # `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so this should be + # permanently unsatisfiable. The alert exists because the exclusion has ways to stop + # working that no static check on that file's TEXT can see: the label being removed or + # moved to the wrong node, a Cilium upgrade changing `NotIn` semantics or the + # nodeSelector field, and the selector silently failing to match. This watches the + # OUTCOME -- who actually holds the lease -- so all of them surface the same way. + # + # WHY THE LABEL JOIN RATHER THAN A NODE NAME. The node's identity is NOT written into + # this expr, because the harness (tests/extract_rules.py) refuses any alert whose expr + # contains a dollar-brace placeholder -- spelled in words because a literal one here is + # unparseable to envsubst and would fail this whole Kustomization. Rightly so: the rule it + # unit-tests would not be the + # rule deployed. Joining `kube_lease_owner` against `kube_node_labels` keeps ONE source + # of truth (the label itself, which is also what the policy selects on), so this alert + # cannot drift out of agreement with the exclusion it guards. `label_replace` is needed + # because kube_lease_owner names the holder `lease_holder` while kube_node_labels names + # it `node`; the join is then on a shared `node` label. The label is in + # kube-state-metrics' metricLabelsAllowlist (kube-prometheus-stack.yaml) -- without that + # line this rule is inert, which `VpnIngressNodeLabelMissing` below exists to catch. + # + # `for: 15m` because leases legitimately move during a node reboot or a Cilium roll and + # Cilium re-elects within seconds; a real collision persists, since the lease holder + # keeps renewing and it never self-heals. + expr: | + max by (lease) ( + label_replace( + kube_lease_owner{namespace="kube-system", lease=~"cilium-l2announce-.*"}, + "node", "$1", "lease_holder", "(.+)" + ) + * on (node) group_left () + kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"} + ) == 1 + for: 15m + labels: + severity: critical + annotations: + summary: "VPN ingress node is announcing a LoadBalancer VIP ({{ $labels.lease }})" + description: >- + The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access + WireGuard, and it now holds the Cilium L2 announcement lease `{{ $labels.lease }}`. A + node cannot reach a VIP it announces itself, so every remote VPN client has just lost + all access to that VIP's Service -- silently, with DNS still resolving. Check that + platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that + the label is still on the node the router forwards udp/51820 to. + runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" + - alert: VpnIngressNodeLabelMissing + # THE WATCHDOG for the rule above, and the reason it is not optional. The exclusion in + # l2-announcement-policy.yaml is a `NotIn` on this label, and `NotIn` with an ABSENT key + # MATCHES -- so if the label is on NO node, the policy excludes nothing, every node is an + # announcement candidate again, and the 2026-10-08 failure is fully re-armed. The alert + # above would not fire either: its join would match nothing and return no series. Present, + # evaluating, and incapable of firing -- the exact "inert critical alert" shape that + # VPNEgressUplinkLabelMissing exists for on the egress side, and that this repo has been + # burned by before. + # + # The label is applied by infra/ansible/roles/node_baseline + # (node_baseline_vpn_ingress_node); the most likely cause of it going absent is a node + # rebuild where that role has not been re-run. + expr: absent(kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"}) + for: 30m + labels: + severity: warning + annotations: + summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF" + description: >- + No node publishes homelab.keithjackson.dev/vpn-ingress=true, so + l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none: + any node may now announce a LoadBalancer VIP, including the one terminating + remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so + the warning matters more than its severity suggests. Re-run the node_baseline role on + the forwarded node, and confirm the label is in kube-state-metrics' + metricLabelsAllowlist. + runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" + # ----------------------------------------------------------------------------------------- # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown` diff --git a/platform/monitoring/kube-prometheus-stack.yaml b/platform/monitoring/kube-prometheus-stack.yaml index 1f75c1c..0daf563 100644 --- a/platform/monitoring/kube-prometheus-stack.yaml +++ b/platform/monitoring/kube-prometheus-stack.yaml @@ -281,8 +281,20 @@ spec: # without anyone having to remember to re-enable it later. Scoped to one label on one resource: # 3 extra series, no cardinality concern. Label name is sanitized by KSM to # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`). + # `vpn-ingress` (added 2026-10-17) marks the node the SOHO router forwards WAN udp/51820 to. + # platform-config/cilium/l2-announcement-policy.yaml excludes that node from announcing any + # LoadBalancer VIP, because a node cannot reach a VIP whose L2 announcement lease it holds + # itself and vpn-ingress MASQUERADEs remote clients to the node's own address (measured + # 2026-10-08: all 16 Gateway hostnames dead over the VPN while DNS still resolved). It is + # allowlisted HERE because that is the only way the invariant can be WATCHED: + # `VpnIngressNodeAnnouncesL2VIP` joins `kube_lease_owner` against this label, and the alert + # unit-test harness refuses a dollar-brace placeholder in an alert expr (written out in words: + # a literal one here is unparseable to envsubst and fails the whole Kustomization), so a knob + # could not have been used. Without this line BOTH that alert and its + # `VpnIngressNodeLabelMissing` watchdog are structurally inert. Same reasoning, and the same + # 3-extra-series cost, as the two egress labels below. metricLabelsAllowlist: - - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]" + - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink,homelab.keithjackson.dev/vpn-ingress]" # --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) --------------------------------- # Velero exports NOTHING that enumerates Schedule objects and nothing that exposes # `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that diff --git a/platform/monitoring/tests/vpn-ingress-l2.test.yaml b/platform/monitoring/tests/vpn-ingress-l2.test.yaml new file mode 100644 index 0000000..4033449 --- /dev/null +++ b/platform/monitoring/tests/vpn-ingress-l2.test.yaml @@ -0,0 +1,203 @@ +# promtool rule-unit tests for VpnIngressNodeAnnouncesL2VIP + VpnIngressNodeLabelMissing +# (platform/monitoring/alerts.yaml) +# +# Run with: platform/monitoring/tests/run.sh +# (the merge gate runs the same harness via +# conformance/tests/l0-static/alert-rules-unit-tested) +# +# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS +# MEASURED on prod 2026-10-08: a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement +# lease it holds ITSELF -- the announcer does not answer its own ARP request, so on the lease holder +# the neighbour entry settles at `FAILED` and packets to that VIP die before leaving the node. +# platform/vpn-ingress runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's +# own address, so a VPN client takes that node's LAN path and inherits the failure. With the +# router's udp/51820 forward on k3s-prod-1, which held the `shared` Gateway lease, all 16 Gateway +# hostnames were dead over the VPN -- while PiHole DNS kept answering, because its VIP has two +# Services and k3s-prod-2 held the other lease. The tunnel handshook, names resolved, every app +# hung, and nothing reported it for ~2 days. +# +# platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled +# `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so VpnIngressNodeAnnouncesL2VIP +# should be permanently unsatisfiable. THAT IS EXACTLY WHY IT NEEDS A TEST: an alert expected never +# to fire is indistinguishable, in production, from an alert that CANNOT fire. Nothing would ever +# reveal a mistyped label, a broken `label_replace`, or a join on the wrong label. Test 1 proves it +# fires on the real failure; test 2 proves it is quiet in the fixed state, so it is not a rule that +# fires on anything; test 3 pins the `for: 15m` against routine lease churn. +# +# THE JOIN IS THE FRAGILE PART AND IS TESTED DELIBERATELY. `kube_lease_owner` names the holder in +# `lease_holder`; `kube_node_labels` names it `node`. The rule bridges them with `label_replace` +# and joins `on (node) group_left ()`. Fixtures therefore carry BOTH series shapes with real label +# sets, because a join that silently matches nothing returns no series -- which looks exactly like +# "healthy" and is the failure mode this whole group exists to prevent. +# +# Fixture values are prod's real shape (measured 2026-10-08): leases +# `cilium-l2announce-gateway-cilium-gateway-shared` (k3s-prod-1) and +# `cilium-l2announce-pihole-pihole-dns-tcp` (k3s-prod-2), in namespace kube-system, with +# k3s-prod-1 the vpn-ingress node. + +rule_files: + - generated.rules.yaml + +evaluation_interval: 1m + +tests: + # ============================================================================================= + # 1. THE BUG, EXACTLY AS IT HAPPENED -- the vpn-ingress node holds the Gateway lease: FIRES. + # ============================================================================================= + - interval: 1m + name: vpn-ingress node holds the shared Gateway L2 lease - VpnIngressNodeAnnouncesL2VIP fires + input_series: + # The collision. A lease holder keeps renewing, so this series does not lapse on its own -- + # which is why the condition never self-heals and why a 15m `for:` is safe. + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}' + values: '1+0x40' + # A second lease held by a DIFFERENT node, present throughout. This is the discriminator that + # stops the rule from being "any l2announce lease exists anywhere": if the join were dropped, + # this series alone would fire the alert, and test 2 would fail. + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-pihole-pihole-dns-tcp",lease_holder="k3s-prod-2"}' + values: '1+0x40' + # k3s-prod-1 is the vpn-ingress node. Only this node carries the label. + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}' + values: '1+0x40' + # The other two nodes publish kube_node_labels WITHOUT the vpn-ingress label. Present so the + # join is exercised against non-matching nodes rather than against nothing at all. + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}' + values: '1+0x40' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}' + values: '1+0x40' + alert_rule_test: + # Held 15m, so nothing before then. + - eval_time: 14m + alertname: VpnIngressNodeAnnouncesL2VIP + exp_alerts: [] + - eval_time: 20m + alertname: VpnIngressNodeAnnouncesL2VIP + exp_alerts: + # EXACTLY ONE alert: the pihole lease on prod-2 must NOT produce a second one. + - exp_labels: + severity: critical + lease: cilium-l2announce-gateway-cilium-gateway-shared + exp_annotations: + summary: "VPN ingress node is announcing a LoadBalancer VIP (cilium-l2announce-gateway-cilium-gateway-shared)" + description: >- + The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access + WireGuard, and it now holds the Cilium L2 announcement lease `cilium-l2announce-gateway-cilium-gateway-shared`. A + node cannot reach a VIP it announces itself, so every remote VPN client has just lost + all access to that VIP's Service -- silently, with DNS still resolving. Check that + platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that + the label is still on the node the router forwards udp/51820 to. + runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" + # The label IS present, so the watchdog must stay quiet here. + - eval_time: 40m + alertname: VpnIngressNodeLabelMissing + exp_alerts: [] + + # ============================================================================================= + # 2. THE FIXED STATE -- every lease is held by a node that is NOT the vpn-ingress node: QUIET. + # This is what l2-announcement-policy.yaml's exclusion produces, and the assertion that keeps + # test 1 meaningful rather than an alert that fires on anything. + # ============================================================================================= + - interval: 1m + name: all l2 leases held by other nodes - VpnIngressNodeAnnouncesL2VIP stays quiet + input_series: + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-2"}' + values: '1+0x40' + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-plex-plex-l4-tcp",lease_holder="k3s-prod-3"}' + values: '1+0x40' + # A lease the vpn-ingress node DOES legitimately hold, in a different family -- proves the + # `cilium-l2announce-.*` matcher is doing work. Every node holds operator/controller leases; + # only the ANNOUNCEMENT ones matter here, and a matcher broad enough to catch this would page + # on a healthy cluster forever. + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-operator-resource-lock",lease_holder="k3s-prod-1"}' + values: '1+0x40' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}' + values: '1+0x40' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}' + values: '1+0x40' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}' + values: '1+0x40' + alert_rule_test: + - eval_time: 40m + alertname: VpnIngressNodeAnnouncesL2VIP + exp_alerts: [] + - eval_time: 40m + alertname: VpnIngressNodeLabelMissing + exp_alerts: [] + + # ============================================================================================= + # 3. A LEGITIMATE, TRANSIENT LEASE MOVE -- the vpn-ingress node holds it for 11m during a node + # reboot or a Cilium roll, then Cilium re-elects away: QUIET. This pins the `for: 15m` as + # deliberate. Without it, every rolling restart of the cilium DaemonSet would page, and a paging + # alert that cries wolf on routine maintenance gets muted -- which is how the real collision + # would go unseen again. + # ============================================================================================= + - interval: 1m + name: transient lease move shorter than for 15m - VpnIngressNodeAnnouncesL2VIP stays quiet + input_series: + # Held by the vpn-ingress node for 11 samples, then the series goes absent (the lease moved, + # so kube-state-metrics retires this label set and starts a new one). + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}' + values: '1+0x10 _x30' + # Re-elected onto prod-3 from t=11m. A label VALUE change retires one series and starts + # another, which is why this is a second series and not a changing value. + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-3"}' + values: '_x11 1+0x29' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}' + values: '1+0x40' + alert_rule_test: + # Inside the transient window: pending at most, never firing. + - eval_time: 10m + alertname: VpnIngressNodeAnnouncesL2VIP + exp_alerts: [] + # Well past it, with the lease settled on another node. + - eval_time: 30m + alertname: VpnIngressNodeAnnouncesL2VIP + exp_alerts: [] + + # ============================================================================================= + # 4. THE WATCHDOG -- no node carries the label at all. The policy's `NotIn` then excludes + # NOTHING (a missing key matches), so the 2026-10-08 failure is fully re-armed, AND + # VpnIngressNodeAnnouncesL2VIP is structurally INERT: its join matches nothing and returns no + # series, which is indistinguishable from healthy. Both halves are asserted here, because an + # inert critical alert is the exact shape this repo has been burned by before. + # ============================================================================================= + - interval: 1m + name: vpn-ingress label absent everywhere - watchdog fires and the critical alert is inert + input_series: + # A lease held by the node that IS (in reality) the forwarded one -- but with no label + # anywhere, nothing can tell. This is the dangerous state. + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}' + values: '1+0x60' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}' + values: '1+0x60' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}' + values: '1+0x60' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}' + values: '1+0x60' + alert_rule_test: + # The critical alert CANNOT fire -- this is the inertness being pinned, not an aside. + - eval_time: 45m + alertname: VpnIngressNodeAnnouncesL2VIP + exp_alerts: [] + # ... which is precisely why the watchdog must. + - eval_time: 45m + alertname: VpnIngressNodeLabelMissing + exp_alerts: + - exp_labels: + severity: warning + # absent() propagates the matcher's own labels onto the series it returns, so the + # alert inherits this one. Asserted rather than worked around: it is harmless, and + # pinning it means a future change to the matcher cannot alter the alert's identity + # (and therefore its Alertmanager grouping) unnoticed. + label_homelab_keithjackson_dev_vpn_ingress: "true" + exp_annotations: + summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF" + description: >- + No node publishes homelab.keithjackson.dev/vpn-ingress=true, so + l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none: + any node may now announce a LoadBalancer VIP, including the one terminating + remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so + the warning matters more than its severity suggests. Re-run the node_baseline role on + the forwarded node, and confirm the label is in kube-state-metrics' + metricLabelsAllowlist. + runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" diff --git a/platform/vpn-ingress/README.md b/platform/vpn-ingress/README.md index 3cfc916..add6522 100644 --- a/platform/vpn-ingress/README.md +++ b/platform/vpn-ingress/README.md @@ -117,8 +117,19 @@ boundary around this port is WireGuard's peer authentication, full stop. The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400 forward). A single-replica Deployment would float on every reschedule and silently invalidate it. A DaemonSet instead puts an identical listener on all three nodes — every node answers the same -keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no -re-issued client profiles. The two idle listeners cost ~15 MiB each. +keys — so losing the forwarded node never costs a re-issued client profile. The two idle listeners +cost ~15 MiB each. + +> **This section used to say "losing the forwarded node is a one-line router change, with no cluster +> work". It was wrong, and it cost two days of silently broken remote access on 2026-10-08.** The +> nodes are warm, but they are **not interchangeable**: the node the forward points at must be the +> one labelled `homelab.keithjackson.dev/vpn-ingress=true`, because that is the node +> `platform-config/cilium/l2-announcement-policy.yaml` excludes from announcing LoadBalancer VIPs. +> **A node cannot reach a VIP whose Cilium L2 announcement lease it holds itself** (see "Measured on +> prod, 2026-10-08" below), and since this pod MASQUERADEs clients to the node's own address, a VPN +> client inherits that. Failover is therefore **two** changes: move +> `node_baseline_vpn_ingress_node` and re-run that role, then re-point the router — +> `runbooks/vpn-remote-access.md` step 3. A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node @@ -186,6 +197,45 @@ right way; one was a defect in my own manifest. | Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. | | Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. | +## Measured on prod, 2026-10-08 — why the forwarded node cannot be any node + +The first device enrolled, handshook, resolved names, and could not open a single connection to any +cluster web app. The tunnel was not at fault — every part of it measured healthy: +`net.ipv4.ip_forward=1`, `wg0` up at `10.90.0.1/24` MTU 1420, the peer authorised with real +bidirectional transfer (346 KiB in / 200 KiB out), `MASQUERADE` present and counting 26 flows, +`FORWARD -i/-o wg0 ACCEPT` counting 108 / 171 packets, and PiHole DNS answering from the node. + +**The fault was that a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it +holds itself.** Cilium's announcer answers ARP for the VIP on behalf of other LAN hosts, but not for +the announcing node's own request, so the neighbour entry settles at `FAILED`: + +| from | `192.168.2.20` (Gateway) | `192.168.2.22` (crafty) | `192.168.2.24` (plex) | announces | +|---|---|---|---|---| +| `k3s-prod-1` | **FAILED** | — | — | `.20` | +| `k3s-prod-2` | ok | **INCOMPLETE** | ok | `.21`, `.22`, `.23-tcp` | +| `k3s-prod-3` | ok | ok | **INCOMPLETE** | `.24` | + +Symmetric across 3 nodes × 5 VIPs: each node reached every VIP **except** the ones it alone +announced. It is a property of L2 announcement, not of any one Service, and it is harmless for +ordinary pods — they reach Services through the cluster datapath and never ARP for a VIP. This +component is the exception, because `hostNetwork` + `MASQUERADE` put remote clients on the node's +own LAN path. + +The forward pointed at `k3s-prod-1`, which held the `shared` Gateway lease, so **all 16 Gateway +hostnames were dead over the VPN**. Two things hid it: + +* **PiHole survived by accident.** `192.168.2.23` is announced by *two* Services + (`pihole-dns-udp` on prod-1, `pihole-dns-tcp` on prod-2), so prod-2 answered the ARP prod-1 could + not. Names resolved perfectly and every app hung — the hardest shape to diagnose from a client. +* **Nothing reported it.** No probe in this repo compares the forwarded node against the lease + holders, and from outside WireGuard is silent to unauthenticated traffic anyway. + +Closed by labelling the forwarded node `homelab.keithjackson.dev/vpn-ingress=true` and excluding +that label in `platform-config/cilium/l2-announcement-policy.yaml` (the node then ARP-resolves every +VIP), with `VpnIngressNodeAnnouncesL2VIP` watching the lease holders so a recurrence is loud and +`VpnIngressNodeLabelMissing` watching for the label going absent, which would silently re-arm the +whole failure. Cost: VIP announcement candidacy drops from three nodes to two. + ## Prerequisites outside this directory 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules` diff --git a/platform/vpn-ingress/daemonset.yaml b/platform/vpn-ingress/daemonset.yaml index c1a203d..64a07e4 100644 --- a/platform/vpn-ingress/daemonset.yaml +++ b/platform/vpn-ingress/daemonset.yaml @@ -62,9 +62,39 @@ # as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to # whichever node the scheduler picked and silently invalidate that forward on every reschedule. A # DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of -# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no -# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own -# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each. +# them, and if that node is lost every other node is already warm and answers the same keys, so no +# client profile ever changes. Each node binds udp/51820 in its own netns, so there is no conflict, +# and the two idle listeners cost ~15 MiB each. +# +# BUT THE NODES ARE NOT INTERCHANGEABLE, AND THIS COMMENT USED TO CLAIM THEY WERE. +# It said "FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work". That was wrong, and the +# wording cost two days of silently broken remote access on 2026-10-08. MEASURED that day: a node +# cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds ITSELF — the announcer +# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and packets to +# that VIP die before leaving the node: +# +# k3s-prod-1, holding the shared Gateway lease: ip neigh show 192.168.2.20 -> FAILED +# k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr …(prod-1) +# +# Because this pod MASQUERADEs clients to the node's own address (see PostUp below), a remote client +# inherits exactly that failure. With the forward on the Gateway's announcing node, all 16 Gateway +# hostnames were unreachable over the VPN while PiHole DNS still resolved — a tunnel that +# handshakes, resolves names, and then hangs on every single app. +# +# THE FIX IS DECLARATIVE AND LIVES OUTSIDE THIS FILE: the forwarded node is labelled +# homelab.keithjackson.dev/vpn-ingress=true by infra/ansible/roles/node_baseline (from +# node_baseline_vpn_ingress_node), and platform-config/cilium/l2-announcement-policy.yaml excludes +# that label from announcing ANY VIP, so the node can ARP-resolve all of them. A label rather than a +# cluster-vars knob because the alert that guards this has to read it too, and the alert unit-test +# harness refuses a postBuild placeholder in an expr. Failover is therefore TWO steps — move +# node_baseline_vpn_ingress_node and re-run the role, THEN re-point the router +# (runbooks/vpn-remote-access.md step 3). +# `VpnIngressNodeAnnouncesL2VIP` (platform/monitoring/alerts.yaml) fires if the two collide again. +# +# This pod deliberately does NOT probe that itself. A readinessProbe on VIP reachability would wedge +# a DaemonSet rollout behind an unrelated Gateway outage (maxUnavailable: 1 below), and a liveness +# probe would tear down a tunnel that is carrying LAN traffic perfectly well. The condition is a +# property of the CLUSTER's announcement layout, so it is watched there. # # All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the # DaemonSet to cover the cluster; add them if a node is ever tainted. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index f01643a..7adc398 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -119,6 +119,11 @@ configMapGenerator: - update-rollback=update-rollback.md - vpn-down=vpn-down.md - vpn-egress-stale=vpn-egress-stale.md + # `VpnIngressNodeAnnouncesL2VIP` — the INBOUND remote-access VPN, not the egress tunnel the two + # pages above cover. A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease + # it holds itself, and vpn-ingress MASQUERADEs clients to the node's own address, so the + # forwarded node must announce nothing (measured 2026-10-08). + - vpn-ingress-l2-collision=vpn-ingress-l2-collision.md # Operator procedure, not an alert page. Served because the component it completes is INERT # until three off-repo manual steps are done (kernel module, server key, WAN port-forward) — # and because the one check that actually proves it cannot be run from inside the house. diff --git a/runbooks/vpn-ingress-l2-collision.md b/runbooks/vpn-ingress-l2-collision.md new file mode 100644 index 0000000..38bb489 --- /dev/null +++ b/runbooks/vpn-ingress-l2-collision.md @@ -0,0 +1,165 @@ +# vpn-ingress-l2-collision — the VPN ingress node is announcing a LoadBalancer VIP + +**Alerts:** `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) and `VpnIngressNodeLabelMissing` +(warning, 30m) +**File:** `platform/monitoring/alerts.yaml` +**Related:** `/runbooks/vpn-remote-access` is the setup and enrolment runbook for this component. +`/runbooks/vpn-egress-stale` and `/runbooks/vpn-down` are the **other** VPN — the outbound PIA +tunnel — and have nothing to do with this. + +## What this means here + +**A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The +announcer answers ARP for the VIP on behalf of other LAN hosts, but not for the announcing node's +own request. On the lease holder the neighbour entry settles at `FAILED`, and every packet to that +VIP is dropped before it leaves the node. Measured on prod 2026-10-08, symmetric across all three +nodes and five VIPs — each node reached every VIP **except** the ones it alone announced. + +For an ordinary workload this is harmless: pods reach Services through the cluster datapath and +never ARP for a VIP. `platform/vpn-ingress` is the exception. It runs WireGuard with +`hostNetwork: true` and MASQUERADEs remote clients to the node's own address, so a remote client's +packets take that node's LAN path and inherit its ARP failures. **If the node the router forwards +`udp/51820` to is also an L2 lease holder, every remote VPN client silently loses all access to the +Services behind the VIPs it announces.** + +The node terminating WireGuard is marked with `homelab.keithjackson.dev/vpn-ingress=true`, and +`platform-config/cilium/l2-announcement-policy.yaml` excludes that label from announcing anything, +so this should be unreachable. + +**The symptom your users report will not sound like a network fault.** On 2026-10-08 the forward sat +on `k3s-prod-1`, which held the `shared` Gateway lease, so all 16 Gateway hostnames were dead over +the VPN — while PiHole DNS kept answering, because `192.168.2.23` is announced by *two* Services and +`k3s-prod-2` held the other lease. So the tunnel handshakes, `wg show` looks perfect, names resolve, +and **every app hangs on connect**. Expect "the VPN connects but nothing loads", not "the VPN is +down". + +Which Services are affected depends on which lease moved — read `{{ $labels.lease }}` from the +alert. `cilium-l2announce-gateway-cilium-gateway-shared` is the worst case: it is every web app at +once. + +### If it is `VpnIngressNodeLabelMissing` that fired + +That is the watchdog, and it means the protection is **switched off**, not that something is merely +untidy. The exclusion is a `NotIn` on the label, and `NotIn` with an absent key **matches** — so +with the label on no node, the policy excludes nothing and every node is an announcement candidate +again. `VpnIngressNodeAnnouncesL2VIP` is simultaneously **inert**: its join matches nothing and +returns no series, which is indistinguishable from healthy. Treat a quiet critical alert as +meaningless until this warning is cleared. Usual cause: a node was rebuilt and `node_baseline` has +not been re-run on it. + +## First commands + +Run cluster commands through the pinned toolbox, which supplies the kube context: + +```bash +# 1. WHO HOLDS WHAT, and WHO IS THE VPN NODE — the whole question, in two lines. +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kube-system get lease | grep l2announce' +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/vpn-ingress' +# Exactly ONE node should carry vpn-ingress=true, and NO l2announce lease should name it. + +# 2. IS THE EXCLUSION STILL IN THE LIVE POLICY? Expect a matchExpressions list with a +# homelab.keithjackson.dev/vpn-ingress NotIn [true] entry — not a bare matchLabels. +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get ciliuml2announcementpolicy default-l2 -o jsonpath="{.spec.nodeSelector}"; echo' + +# 3. IS THE LABEL REACHING PROMETHEUS? If this is empty the alerts are inert even when +# the label is on the node (kube-state-metrics metricLabelsAllowlist). +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring exec deploy/kube-prometheus-stack-kube-state-metrics -- \ + wget -qO- localhost:8080/metrics | grep vpn_ingress | head' +``` + +**Confirm the actual damage**, which is the one measurement that is unambiguous. From the WireGuard +pod on the affected node (it shares the host netns, so this is the path a VPN client takes): + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress get pods -o wide' # pick the pod on that node + +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress exec -c wireguard -- \ + sh -c "ping -c1 -W1 192.168.2.20 >/dev/null 2>&1; ip neigh show 192.168.2.20"' +# BROKEN: 192.168.2.20 dev eno1 FAILED +# HEALTHY: 192.168.2.20 dev eno1 lladdr bc:e9:2f:7e:c4:a6 STALE|REACHABLE|DELAY +``` + +The `ping` is only there to trigger ARP — the VIP does not answer ICMP even when healthy, so +**ignore its 100% packet loss**; the neighbour state is the answer. Substitute the VIP that belongs +to `{{ $labels.lease }}` if the Gateway is not the one that moved. + +**Which node actually holds the forward** (the router cannot be queried, but the cluster knows): +only the forwarded node's `wg show` has a peer with an `endpoint` and a recent `latest handshake`. + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'for p in $(kubectl --context k3s-prod -n vpn-ingress get pods -o name); do + echo "== $p"; kubectl --context k3s-prod -n vpn-ingress exec ${p#pod/} -c wireguard -- wg show; done' +``` + +## Likely causes, most likely first + +1. **The label is on the wrong node** — someone re-pointed the router's forward during a node + failover and did not move `node_baseline_vpn_ingress_node`. This is the most likely cause by a + wide margin, because the router is off-repo and nothing can check it. Confirm with the `wg show` + loop above: the node with the live peer endpoint is the real forward target. + *Fix:* merge a PR setting `node_baseline_vpn_ingress_node` + (`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`) to that node, then re-run the role — + `ansible-playbook -i inventory/hosts.ini -e k3s_hosts=k3s_prod --tags vpn-ingress`. + The role labels the new node and **removes** the label from the old one. Then confirm Cilium + re-elected the lease off the newly-labelled node. + +2. **The exclusion was edited out of `platform-config/cilium/l2-announcement-policy.yaml`**, or the + file was reverted to a bare `matchLabels: {kubernetes.io/os: linux}`. Command 2 above shows this + directly. *Fix:* restore the `matchExpressions` form; the file's own header explains why it is + shaped that way. + +3. **The label is missing entirely** (`VpnIngressNodeLabelMissing` firing) — a node rebuild without + a `node_baseline` re-run. *Fix:* re-run the role as above. + +4. **The label is on the node but not in Prometheus** — `metricLabelsAllowlist` in + `platform/monitoring/kube-prometheus-stack.yaml` lost the `vpn-ingress` entry. Then the critical + alert is inert *and* the watchdog fires. Command 3 above distinguishes this from cause 3. + +5. **Cilium kept renewing a lease its holder is no longer eligible for.** The exclusion is correct + and in place, but the lease never re-elected. This was never verified when the fix was authored + (see `decisions/2026-10-17-vpn-ingress-l2-lease-collision.md`). *Fix:* delete that one Lease + object to force an election — `kubectl -n kube-system delete lease ` — and watch it come + back on a different node. + +6. **A Cilium upgrade changed `NotIn` semantics or the `nodeSelector` field.** Least likely, but it + is the reason this is a runtime alert rather than a static check on the policy text. + +## When to escalate or stop + +**Act now, don't wait for a window.** Remote access is fully broken for the affected VIPs while this +is firing, and it does **not** self-heal: a lease holder keeps renewing, so this condition persists +indefinitely once entered. That is also why the 15m `for:` is safe — a transient move during a node +reboot or a Cilium roll resolves on its own well inside it. + +**The fastest honest mitigation is to move the forward, not the leases.** Re-pointing the router's +`udp/51820` at a node that holds no lease for the affected VIP restores service in seconds, with no +cluster change and no new client profiles — all three nodes answer the same keys. Follow it with the +`node_baseline_vpn_ingress_node` PR and role re-run so the two agree again; until you do, this alert +is watching the wrong node. Note each node's own announcements when choosing: as of 2026-10-08, +prod-2 announced crafty `192.168.2.22` and unifi `192.168.2.21`, prod-3 announced plex +`192.168.2.24`, so neither is a cost-free target. + +**Do not** `kubectl edit` the `CiliumL2AnnouncementPolicy` to fix this — Flux owns it and will revert +you, and the drift is the kind that outlives the memory of making it. The two acceptable hand +actions are deleting a single Lease to force re-election (cause 5), and `kubectl label node` as a +*temporary* stopgap while the PR lands — note that a node rebuild or a `node_baseline` run will undo +a hand-applied label, so it is not a fix. + +**Stop and escalate to the operator if** the leases will not leave the excluded node after a forced +re-election, or if the intended forward target is unclear. Both need a decision about which node +carries remote access, which is an operator call, not a runbook step. + +**Known gap, so you are not misled:** if someone re-points the router without moving the label, +`VpnIngressNodeAnnouncesL2VIP` watches the wrong node and **stays silent** while the VPN breaks in +exactly the way described above. There is no in-cluster signal for that today; closing it needs a +WireGuard exporter publishing which node is actually terminating peer traffic +(`platform/vpn-ingress/README.md`, "Not built") — the `wg show` loop above is the manual substitute. +Until then, if remote access is broken and both alerts are quiet, **check the forward against the +label by hand before believing the alerts.** diff --git a/runbooks/vpn-remote-access.md b/runbooks/vpn-remote-access.md index 9f123ab..dfdd874 100644 --- a/runbooks/vpn-remote-access.md +++ b/runbooks/vpn-remote-access.md @@ -171,14 +171,57 @@ server without a `wg0` "ready". On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g. `192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check. -Pick any node — all three run an identical listener. Note which one you chose here: +> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node. +> +> This step used to say "pick any node — all three run an identical listener". **That sentence +> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a +> cluster-vars knob instead of a free choice here. +> +> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The +> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease +> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they +> leave the node. Measured on prod 2026-10-08: +> +> ```text +> on k3s-prod-1 (held the shared Gateway lease): ip neigh show 192.168.2.20 -> FAILED +> on k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr … (prod-1) +> ``` +> +> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN +> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway +> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so +> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported +> it. +> +`infra/ansible/roles/node_baseline` now labels the forwarded node +> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in +> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes +> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI +> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a +> labelled node taking a lease, but it CANNOT catch the forward being moved without the label. + +Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`). +Note it here: ``` # WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set +# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml ``` -**Node loss = failover is this one line.** Re-point the forward at another node address; every -device keeps working with the same profile, because all three nodes answer the same keys. +**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that +was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and +reintroduces the failure above against a different set of addresses (prod-2 announces crafty and +unifi; prod-3 announces plex). Do both, in this order: + +1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role + (`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium + then moves the leases off the newly-labelled node and onto its peers. +2. **Re-point the router** forward at the new node's address. + +Every device keeps working with the same profile either way; all three nodes answer the same keys, +and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that +handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because +that alert watches the node the knob names, not the node your router actually points at. Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is already silent to unauthenticated probes (README, "Why this is safe to expose").