diff --git a/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md b/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md new file mode 100644 index 0000000..b21be0e --- /dev/null +++ b/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md @@ -0,0 +1,137 @@ +--- +date: 2026-10-17 +branch: fix/vpn-ingress-l2-lease-collision +title: "A node cannot reach a VIP it announces — so the VPN ingress node must announce nothing" +--- + +## What changed + +The first device enrolled on the remote-access VPN handshook, resolved names, and could not open a +single connection to any cluster web app. The tunnel was not at fault. The fault is a property of +Cilium L2 announcement that nothing in this repo knew about: + +**A node cannot reach a LoadBalancer VIP whose `CiliumL2AnnouncementPolicy` lease it holds itself.** +The announcer answers ARP for the VIP on behalf of other LAN hosts but not for its own request, so +on the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP is dropped +before it leaves the node. + +`platform/vpn-ingress` is uniquely exposed because it runs WireGuard with `hostNetwork: true` and +MASQUERADEs remote clients to the node's own address — a VPN client's packets take the node's LAN +path and inherit its ARP failures. Ordinary pods never do. The router's `udp/51820` forward pointed +at `k3s-prod-1`, which held `cilium-l2announce-gateway-cilium-gateway-shared`, so **all 16 `shared` +Gateway hostnames were dead over the VPN** — while PiHole DNS kept answering, because +`192.168.2.23` is announced by *two* Services and `k3s-prod-2` held the other lease. Names resolved +and every app hung on connect. + +Seven files: + +* **`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`** — new + `node_baseline_vpn_ingress_node`; the matching node gets + `homelab.keithjackson.dev/vpn-ingress=true`, and a second task **removes** the label from any node + that no longer matches, so moving the value moves the label rather than adding a second one. Same + shape as `roles/vpn_egress_gateway`'s `egress-gateway` label. Tagged `vpn-ingress`; skipped + entirely when the value is empty. +* **`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`** — set to `k3s-prod-1`, with a + `[parity]` note that it must equal the router's forward. +* **`platform-config/cilium/l2-announcement-policy.yaml`** — `nodeSelector` becomes a + `matchExpressions` list keeping `kubernetes.io/os: linux` and adding + `homelab.keithjackson.dev/vpn-ingress NotIn ["true"]`. The VPN node then holds no lease, so it + ARP-resolves every VIP. This fixes **all** VIPs for VPN clients (Gateway, crafty, plex, pihole, + unifi), not just the one that happened to collide. +* **`platform/monitoring/kube-prometheus-stack.yaml`** — the label added to kube-state-metrics' + `metricLabelsAllowlist`, which is what lets the invariant be watched at all. +* **`platform/monitoring/alerts.yaml`** — `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) joins + `kube_lease_owner` against `kube_node_labels` via `label_replace`, plus + `VpnIngressNodeLabelMissing` (warning, 30m) as its watchdog. +* **`platform/monitoring/tests/vpn-ingress-l2.test.yaml`** — four promtool cases: fires on the real + collision, quiet in the fixed state, quiet across a transient 11-minute lease move (pinning the + `for: 15m`), and the label-absent case asserting the critical alert is inert *and* the watchdog + fires. +* **`runbooks/vpn-ingress-l2-collision.md`** (new, registered in `runbooks/kustomization.yaml`), + **`runbooks/vpn-remote-access.md`**, **`platform/vpn-ingress/{daemonset.yaml,README.md}`** — step 3 + of the setup runbook said *"pick any node — all three run an identical listener"* and *"node loss = + failover is this one line"*. Both were false and the first caused this. Failover is now documented + as two ordered changes. + +**Design note — why a node label and not a `cluster-vars` knob.** The first attempt added +`WG_INGRESS_NODE` to `clusters/base/cluster-vars.yaml` and substituted it into both the policy and +the alert. The alert half is **impossible**: `platform/monitoring/tests/extract_rules.py` rejects any +alert whose `expr` contains a `${...}` placeholder, because the rule it would unit-test is not the +rule deployed. Rather than duplicate the node name into the alert as a literal (two sources that can +skew), the label is readable from both sides — the selector matches on it and `kube_node_labels` +carries it into Prometheus. One source of truth, and as a side effect the change touches no +sensitive path, so it merges on a green gate rather than needing a human review. + +The DaemonSet deliberately gained **no** probe for this. A readinessProbe on VIP reachability would +wedge a rollout behind an unrelated Gateway outage (`maxUnavailable: 1`); a liveness probe would tear +down a tunnel carrying LAN traffic correctly. + +## What I verified + +Measured live against prod, `--context k3s-prod`, on 2026-10-08: + +* **The whole tunnel is healthy**, which is what made this hard: `wg show` on `k3s-prod-1` shows the + peer with a real endpoint and bidirectional transfer (346 KiB in / 200 KiB out); + `net.ipv4.ip_forward=1`; `wg0` up at `10.90.0.1/24` MTU 1420; `iptables -t nat -S POSTROUTING` + carries the `MASQUERADE` rule with 26 packets; `FORWARD -i/-o wg0` ACCEPT with 108 / 171 packets; + DaemonSet generation 2, 3/3 ready on the content-hashed `wg-peers-m9kcckcb5t`. +* **The ARP failure and its symmetry.** On `k3s-prod-1`: `ip neigh show 192.168.2.20` → `FAILED`, + while `.21`/`.23` resolve to prod-2's MAC `04:0e:3c:96:b0:94`. On prod-2/prod-3, `.20` resolves to + prod-1's `bc:e9:2f:7e:c4:a6`. TCP agrees: `.20:443` and `.20:80` FAIL from prod-1, OK from the + other two. Repeated across 3 nodes × 5 VIPs — each node reached every VIP except the ones it alone + announced (prod-2 ✗ `.22`/`.21`; prod-3 ✗ `.24`). +* **Not an artefact of probing from the host netns.** `curl --interface wg0` from prod-1 fails the + same way, and a neighbour entry is node-global state, so the forwarded path cannot differ in ARP + resolution. +* **The Gateway itself was never broken:** from the workstation (192.168.2.133) + `home.keithjackson.dev` → `200`, `plex.home.keithjackson.dev` → `401`, both at `192.168.2.20`. + From prod-1 the router (`.1`) and TrueNAS (`.30`) are `REACHABLE`. +* **`NotIn` semantics, empirically** — the basis for the fail-back claim in the policy's comment: + `kubectl get nodes -l 'kubernetes.io/hostname notin (k3s-prod-1)'` returns prod-2 and prod-3, and + `NotIn` on a key no node carries returns **all three**. So an unapplied label excludes nothing and + degrades to the old behaviour rather than taking every VIP off the LAN. +* **The alert metric is really scraped:** `kube_lease_owner` is served by kube-state-metrics (55 + series, every `cilium-l2announce-*` lease with its holder in `lease_holder`), and `kube_node_labels` + already carries the two `egress-*` labels through the same allowlist (`k3s-prod-3` shows + `egress-gateway=true`), so the third entry follows a proven path. +* **The four promtool cases pass** (`platform/monitoring/tests/run.sh` → `SUCCESS`, 55 rules + checked), including that `VpnIngressNodeAnnouncesL2VIP` is inert when the label is absent. +* All edited YAML parses; the rendered `nodeSelector` is the intended two-expression list. + +## What I did NOT verify + +* **The fix is UNAPPLIED and therefore UNPROVEN end to end.** Applying it needs a mutating + `kubectl`/`flux reconcile` plus an Ansible run, which are operator actions (CLAUDE.md §5). I have + **not** observed `k3s-prod-1` resolve `192.168.2.20` after the exclusion lands, nor a VPN client + reaching an app. The prediction rests on the measured symmetry (non-announcing nodes resolve every + VIP, 15/15 data points), not on a test of the changed state. **Verify after merge, in this order:** + re-run `node_baseline --tags vpn-ingress`; `kubectl get nodes -L + homelab.keithjackson.dev/vpn-ingress` (exactly one node); `kubectl -n kube-system get lease | grep + l2announce` (no holder is that node); `ip neigh show 192.168.2.20` from the prod-1 wireguard pod + (expect a `lladdr`, not `FAILED`); then a client. +* **That Cilium actually releases a lease when its holder stops matching the `nodeSelector`.** I + assumed re-election and did not test it. If Cilium keeps renewing, the affected Leases must be + deleted once by hand to force an election. Check this first if the fix appears not to work. +* **The brief interruption on apply.** Moving the Gateway and `pihole-dns-udp` leases off prod-1 will + blip `192.168.2.20` and `192.168.2.23` for as long as re-election plus gratuitous ARP takes. Not + measured, and I did not test whether in-flight LAN sessions survive it. +* **The Ansible tasks have never been executed.** They parse as YAML and follow + `roles/vpn_egress_gateway`'s shape, but neither the label task nor the label-removal task has run, + so the `changed_when`/`failed_when` conditions and the `k3s kubectl label ... -` removal path are + unproven. The removal task's `failed_when` tolerates `'not found'` in stderr; that string was + assumed from kubectl's behaviour, not observed. +* **The residual off-repo coupling is NOT closed and cannot be, as built.** The router's forward is + unreadable from the cluster. If someone re-points it without moving the label, the alert watches + the wrong node and stays silent while the VPN breaks identically against a different set of VIPs. + Only the runbook guards it. Closing it needs a WireGuard exporter publishing which node is + actually terminating peer traffic (`wg show` knows — only the forwarded node has a peer + `endpoint`); listed as "Not built" in the component README and **recommended follow-up, not done + here**. +* **Why Cilium behaves this way** — intended, a known upstream issue, or fixed in a later version. I + measured the behaviour and did not read Cilium's source or issue tracker, so I cannot say whether + an upgrade would remove the need for this exclusion. +* **`192.168.2.21:8443` (unifi) fails from all three nodes**, including ones where ARP resolves, so + it is a different, pre-existing fault. Not investigated, not addressed. +* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 offline + subset only; none of the above is covered by a test that runs in the gate. +* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b). diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index 2b865a6..d286c9d 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -420,3 +420,22 @@ flux_bootstrap_env_path: "clusters/prod" # rather than leaving an inert drop-in behind, so this group-wide `true` is still correct # for it. etcd_io_priority_enabled: true + +# --- The node terminating remote-access WireGuard — DECIDED (measured 2026-10-08) --------------- +# k3s-prod-1 (192.168.2.10) is what the SOHO router forwards WAN udp/51820 to, confirmed from the +# cluster rather than assumed: only prod-1's `wg show` had a peer with an `endpoint` and a recent +# handshake, the other two listeners were idle. +# +# node_baseline labels it `homelab.keithjackson.dev/vpn-ingress=true`, and +# platform-config/cilium/l2-announcement-policy.yaml excludes that label from announcing any +# LoadBalancer VIP -- because a node CANNOT reach a VIP whose Cilium L2 announcement lease it holds +# itself (the announcer does not answer its own ARP request; the neighbour entry goes to `FAILED`). +# prod-1 held the `shared` Gateway lease, and since vpn-ingress MASQUERADEs clients to the node's +# own address, all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved. +# +# [parity] MUST equal the node named by the router's forward (runbooks/vpn-remote-access.md step 3). +# Nothing in CI can read that forward, so re-pointing it means changing BOTH: this value plus a +# node_baseline re-run, and the router line. `VpnIngressNodeAnnouncesL2VIP` catches the case where +# the label lands on a node that then takes a lease; it CANNOT catch the forward being moved +# without this value following, which is why the runbook spells the two-step out. +node_baseline_vpn_ingress_node: k3s-prod-1 diff --git a/infra/ansible/roles/node_baseline/defaults/main.yml b/infra/ansible/roles/node_baseline/defaults/main.yml index b6a7f4b..ae7a493 100644 --- a/infra/ansible/roles/node_baseline/defaults/main.yml +++ b/infra/ansible/roles/node_baseline/defaults/main.yml @@ -259,3 +259,30 @@ node_baseline_dns_options: # Written as ONE marked block, so removing an entry here removes it from the node, and # an empty list removes the block entirely. node_baseline_hosts_pins: [] + +# --- The node that terminates remote-access WireGuard (tags: vpn-ingress) ---------------------- +# Hostname of the node the SOHO router forwards WAN udp/51820 to, i.e. the one platform/vpn-ingress +# actually serves remote clients from. Set it in group_vars for the environment; EMPTY by default, +# because dev has no such forward and because a wrong value is worse than none. +# +# WHAT IT IS FOR, AND WHY IT IS NOT COSMETIC. The matching node gets +# `homelab.keithjackson.dev/vpn-ingress=true`, and +# platform-config/cilium/l2-announcement-policy.yaml EXCLUDES that label from announcing any +# LoadBalancer VIP. MEASURED on prod 2026-10-08: a node cannot reach a VIP whose Cilium L2 +# announcement lease it holds itself (the announcer does not answer its own ARP request -- the +# neighbour entry goes to `FAILED`). Because vpn-ingress runs on `hostNetwork` and MASQUERADEs +# clients to the node's own address, a VPN client inherits that: with the forward on the node +# holding the `shared` Gateway lease, all 16 Gateway hostnames were dead over the VPN while PiHole +# DNS still resolved. Names worked, every app hung, nothing alerted. +# +# [parity] MUST name the node the router's forward actually points at. The forward is off-repo and +# no check can read it (runbooks/vpn-remote-access.md step 3). Changing the forward is therefore a +# two-part change: this value (then re-run the role), and the router line. +# +# Only ONE node should ever carry the label: it removes that node from VIP announcement candidacy, +# so labelling all three would leave nothing announcing any VIP at all. The task below removes the +# label from a node that no longer matches, so moving the value moves the label. +node_baseline_vpn_ingress_node: "" +node_baseline_vpn_ingress_label_key: homelab.keithjackson.dev/vpn-ingress +# k3s ships its kubeconfig here; every prod node is a k3s server, so `k3s kubectl` works locally. +node_baseline_vpn_ingress_kubeconfig: /etc/rancher/k3s/k3s.yaml diff --git a/infra/ansible/roles/node_baseline/tasks/main.yml b/infra/ansible/roles/node_baseline/tasks/main.yml index 1422573..de317f4 100644 --- a/infra/ansible/roles/node_baseline/tasks/main.yml +++ b/infra/ansible/roles/node_baseline/tasks/main.yml @@ -1109,3 +1109,45 @@ state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}" backup: true tags: [dns, hosts] + +# ── vpn-ingress node label — tag `vpn-ingress` ──────────────────────────────────────────────── +# Marks the node terminating remote-access WireGuard so Cilium's L2 announcement policy can EXCLUDE +# it from announcing LoadBalancer VIPs (a node cannot reach a VIP it announces itself -- see +# defaults/main.yml). Same shape as roles/vpn_egress_gateway's egress-gateway label, and for the +# same reason: the label is the one place BOTH a nodeSelector and a PromQL alert can read it from. +# +# Skipped entirely when node_baseline_vpn_ingress_node is empty (dev, or before the forward exists), +# so this adds nothing to an environment that has no WAN forward. +- name: Label this node as the vpn-ingress node + ansible.builtin.command: + cmd: >- + k3s kubectl label node {{ inventory_hostname }} + {{ node_baseline_vpn_ingress_label_key }}=true --overwrite + environment: + KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}" + register: node_baseline_vpn_ingress_label_apply + changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_apply.stdout" + when: + - node_baseline_vpn_ingress_node | length > 0 + - inventory_hostname == node_baseline_vpn_ingress_node + tags: [vpn-ingress] + +# The other half: moving the knob must MOVE the label, not add a second one. Two labelled nodes +# would remove two of three nodes from announcement candidacy; three would leave no announcer at +# all and take every VIP off the LAN. +- name: Remove the vpn-ingress label from a node that no longer holds the forward + ansible.builtin.command: + cmd: >- + k3s kubectl label node {{ inventory_hostname }} + {{ node_baseline_vpn_ingress_label_key }}- + environment: + KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}" + register: node_baseline_vpn_ingress_label_remove + changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_remove.stdout" + failed_when: + - node_baseline_vpn_ingress_label_remove.rc != 0 + - "'not found' not in node_baseline_vpn_ingress_label_remove.stderr" + when: + - node_baseline_vpn_ingress_node | length > 0 + - inventory_hostname != node_baseline_vpn_ingress_node + tags: [vpn-ingress] diff --git a/platform-config/cilium/l2-announcement-policy.yaml b/platform-config/cilium/l2-announcement-policy.yaml index d65e0ff..c9229f5 100644 --- a/platform-config/cilium/l2-announcement-policy.yaml +++ b/platform-config/cilium/l2-announcement-policy.yaml @@ -16,6 +16,71 @@ spec: externalIPs: false interfaces: - ^e(n|th) + # =============================================================================================== + # WHY THIS IS A matchExpressions LIST AND NOT `matchLabels: {kubernetes.io/os: linux}` + # =============================================================================================== + # The second expression excludes every node labelled `homelab.keithjackson.dev/vpn-ingress=true` + # from announcing ANY LoadBalancer VIP. That is the node the SOHO router forwards WAN udp/51820 to. + # It is not tuning; it is the fix for a measured, silent, total failure of remote access. + # + # MEASURED ON PROD 2026-10-08: A NODE CANNOT REACH A VIP IT ANNOUNCES ITSELF. Cilium's L2 + # announcer answers ARP for the VIP on behalf of OTHER LAN hosts, but not for the announcing node + # itself. On the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP + # is dropped before it leaves the node: + # + # k3s-prod-1, holding cilium-l2announce-gateway-cilium-gateway-shared: + # ip neigh show 192.168.2.20 -> 192.168.2.20 dev eno1 FAILED (its own VIP) + # nc -z 192.168.2.20 443 -> FAIL nc -z 192.168.2.20 80 -> FAIL + # k3s-prod-2 and k3s-prod-3, holding no lease for it: + # ip neigh show 192.168.2.20 -> lladdr bc:e9:2f:7e:c4:a6 (prod-1) nc 443/80 -> OK + # + # Symmetric across all 3 nodes x 5 VIPs: every node reached every VIP EXCEPT the ones it alone + # announced (prod-2 could not reach crafty 192.168.2.22 or unifi 192.168.2.21; prod-3 could not + # reach plex 192.168.2.24). So this is a property of L2 announcement, not of one Service. + # + # WHY IT ONLY HURTS THE VPN NODE. For an ordinary workload this is harmless: pods reach Services + # through the cluster datapath and never ARP for a VIP. platform/vpn-ingress is the exception -- + # it runs WireGuard with `hostNetwork: true` and MASQUERADEs remote clients to the node's own + # address, so a VPN client's packets take the node's LAN path and inherit exactly this ARP + # failure. On 2026-10-08 the forward pointed at k3s-prod-1, which held the Gateway lease, and all + # 16 Gateway hostnames were unreachable over the VPN while PiHole DNS still answered -- so names + # resolved and every app hung. Nothing in the cluster reported it; the alerts below now do. + # + # WHY A NODE LABEL AND NOT A cluster-vars KNOB. A knob would put the node's identity in git, which + # is more reviewable -- but it CANNOT BE USED BY THE ALERT that guards this, and an unguarded + # invariant here is worth less than a reviewable one. The alert harness + # (platform/monitoring/tests/extract_rules.py) REFUSES any alert whose `expr` contains a + # dollar-brace postBuild placeholder (spelled without braces here ON PURPOSE: this file is itself + # envsubst'd, and a literal dollar-brace-dot-dot-dot is UNPARSEABLE to envsubst, which fails the + # WHOLE Kustomization -- ci: flux-postbuild-substitution caught exactly this comment), because the + # rule it would unit-test is not the rule that gets + # deployed. A label is readable from BOTH sides: this selector matches on it, and + # `kube_node_labels` carries it into Prometheus so `VpnIngressNodeAnnouncesL2VIP` can join lease + # holders against it. One source of truth, no git-vs-cluster skew, and the policy stays + # env-agnostic as its header above claims. The label is set by + # infra/ansible/roles/node_baseline (node_baseline_vpn_ingress_node), the same shape + # roles/vpn_egress_gateway already uses for `egress-gateway`. + # + # THE COST, STATED: VIP announcement candidacy drops from 3 nodes to 2. That is still redundant, + # and the trade is deliberate -- the excluded node is the one whose *outbound* reach to every VIP + # matters most, because it carries every remote client. If BOTH remaining nodes are down the VIPs + # go dark where previously one node might have served them, but at that point the cluster has one + # node left and far worse problems. + # + # `NotIn` with an ABSENT key MATCHES (k8s apimachinery: a missing label is "not in the set"), so + # the other two nodes need no label and nothing has to be applied to them. If the label is never + # applied anywhere, this excludes nothing and the policy degrades to its pre-2026-10-08 behaviour + # -- the status quo, not an outage: it fails back, not down. That silent-degradation path is + # exactly why `VpnIngressNodeLabelMissing` exists as a watchdog: without it, the alert below + # would be structurally INERT (its join would match nothing) and would look identical to healthy. nodeSelector: - matchLabels: - kubernetes.io/os: linux + matchExpressions: + - key: kubernetes.io/os + operator: In + values: + - linux + # The node terminating remote-access WireGuard must announce nothing -- see above. + - key: homelab.keithjackson.dev/vpn-ingress + operator: NotIn + values: + - "true" diff --git a/platform/monitoring/alerts.yaml b/platform/monitoring/alerts.yaml index e007f2d..ef18519 100644 --- a/platform/monitoring/alerts.yaml +++ b/platform/monitoring/alerts.yaml @@ -1147,6 +1147,97 @@ spec: metricLabelsAllowlist. runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale" # ----------------------------------------------------------------------------------------- + # REMOTE-ACCESS VPN (platform/vpn-ingress) -- NOT the egress tunnel above. Different + # direction, different workload: this is inbound WireGuard for remote devices. + # ----------------------------------------------------------------------------------------- + - alert: VpnIngressNodeAnnouncesL2VIP + # THE FAILURE THIS CATCHES. MEASURED on prod 2026-10-08: a node CANNOT reach a + # LoadBalancer VIP whose Cilium L2 announcement lease it holds itself -- the announcer + # does not answer its own ARP request, so the neighbour entry settles at `FAILED` and + # packets die before leaving the node (`ip neigh show 192.168.2.20` -> FAILED on the + # lease holder, resolved on both others). platform/vpn-ingress runs WireGuard on + # `hostNetwork` and MASQUERADEs remote clients to the node's own address, so if the node + # the router forwards udp/51820 to is also a lease holder, VPN clients cannot reach the + # VIPs it announces. On 2026-10-08 that was the `shared` Gateway: all 16 app hostnames + # dead over the VPN, while PiHole DNS kept answering (its VIP has two Services and the + # other lease sat on prod-2) -- so the tunnel handshook, names resolved, and every app + # hung. Nothing reported it for ~2 days. + # + # platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled + # `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so this should be + # permanently unsatisfiable. The alert exists because the exclusion has ways to stop + # working that no static check on that file's TEXT can see: the label being removed or + # moved to the wrong node, a Cilium upgrade changing `NotIn` semantics or the + # nodeSelector field, and the selector silently failing to match. This watches the + # OUTCOME -- who actually holds the lease -- so all of them surface the same way. + # + # WHY THE LABEL JOIN RATHER THAN A NODE NAME. The node's identity is NOT written into + # this expr, because the harness (tests/extract_rules.py) refuses any alert whose expr + # contains a dollar-brace placeholder -- spelled in words because a literal one here is + # unparseable to envsubst and would fail this whole Kustomization. Rightly so: the rule it + # unit-tests would not be the + # rule deployed. Joining `kube_lease_owner` against `kube_node_labels` keeps ONE source + # of truth (the label itself, which is also what the policy selects on), so this alert + # cannot drift out of agreement with the exclusion it guards. `label_replace` is needed + # because kube_lease_owner names the holder `lease_holder` while kube_node_labels names + # it `node`; the join is then on a shared `node` label. The label is in + # kube-state-metrics' metricLabelsAllowlist (kube-prometheus-stack.yaml) -- without that + # line this rule is inert, which `VpnIngressNodeLabelMissing` below exists to catch. + # + # `for: 15m` because leases legitimately move during a node reboot or a Cilium roll and + # Cilium re-elects within seconds; a real collision persists, since the lease holder + # keeps renewing and it never self-heals. + expr: | + max by (lease) ( + label_replace( + kube_lease_owner{namespace="kube-system", lease=~"cilium-l2announce-.*"}, + "node", "$1", "lease_holder", "(.+)" + ) + * on (node) group_left () + kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"} + ) == 1 + for: 15m + labels: + severity: critical + annotations: + summary: "VPN ingress node is announcing a LoadBalancer VIP ({{ $labels.lease }})" + description: >- + The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access + WireGuard, and it now holds the Cilium L2 announcement lease `{{ $labels.lease }}`. A + node cannot reach a VIP it announces itself, so every remote VPN client has just lost + all access to that VIP's Service -- silently, with DNS still resolving. Check that + platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that + the label is still on the node the router forwards udp/51820 to. + runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" + - alert: VpnIngressNodeLabelMissing + # THE WATCHDOG for the rule above, and the reason it is not optional. The exclusion in + # l2-announcement-policy.yaml is a `NotIn` on this label, and `NotIn` with an ABSENT key + # MATCHES -- so if the label is on NO node, the policy excludes nothing, every node is an + # announcement candidate again, and the 2026-10-08 failure is fully re-armed. The alert + # above would not fire either: its join would match nothing and return no series. Present, + # evaluating, and incapable of firing -- the exact "inert critical alert" shape that + # VPNEgressUplinkLabelMissing exists for on the egress side, and that this repo has been + # burned by before. + # + # The label is applied by infra/ansible/roles/node_baseline + # (node_baseline_vpn_ingress_node); the most likely cause of it going absent is a node + # rebuild where that role has not been re-run. + expr: absent(kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"}) + for: 30m + labels: + severity: warning + annotations: + summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF" + description: >- + No node publishes homelab.keithjackson.dev/vpn-ingress=true, so + l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none: + any node may now announce a LoadBalancer VIP, including the one terminating + remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so + the warning matters more than its severity suggests. Re-run the node_baseline role on + the forwarded node, and confirm the label is in kube-state-metrics' + metricLabelsAllowlist. + runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" + # ----------------------------------------------------------------------------------------- # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown` diff --git a/platform/monitoring/kube-prometheus-stack.yaml b/platform/monitoring/kube-prometheus-stack.yaml index 1f75c1c..0daf563 100644 --- a/platform/monitoring/kube-prometheus-stack.yaml +++ b/platform/monitoring/kube-prometheus-stack.yaml @@ -281,8 +281,20 @@ spec: # without anyone having to remember to re-enable it later. Scoped to one label on one resource: # 3 extra series, no cardinality concern. Label name is sanitized by KSM to # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`). + # `vpn-ingress` (added 2026-10-17) marks the node the SOHO router forwards WAN udp/51820 to. + # platform-config/cilium/l2-announcement-policy.yaml excludes that node from announcing any + # LoadBalancer VIP, because a node cannot reach a VIP whose L2 announcement lease it holds + # itself and vpn-ingress MASQUERADEs remote clients to the node's own address (measured + # 2026-10-08: all 16 Gateway hostnames dead over the VPN while DNS still resolved). It is + # allowlisted HERE because that is the only way the invariant can be WATCHED: + # `VpnIngressNodeAnnouncesL2VIP` joins `kube_lease_owner` against this label, and the alert + # unit-test harness refuses a dollar-brace placeholder in an alert expr (written out in words: + # a literal one here is unparseable to envsubst and fails the whole Kustomization), so a knob + # could not have been used. Without this line BOTH that alert and its + # `VpnIngressNodeLabelMissing` watchdog are structurally inert. Same reasoning, and the same + # 3-extra-series cost, as the two egress labels below. metricLabelsAllowlist: - - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]" + - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink,homelab.keithjackson.dev/vpn-ingress]" # --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) --------------------------------- # Velero exports NOTHING that enumerates Schedule objects and nothing that exposes # `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that diff --git a/platform/monitoring/tests/vpn-ingress-l2.test.yaml b/platform/monitoring/tests/vpn-ingress-l2.test.yaml new file mode 100644 index 0000000..4033449 --- /dev/null +++ b/platform/monitoring/tests/vpn-ingress-l2.test.yaml @@ -0,0 +1,203 @@ +# promtool rule-unit tests for VpnIngressNodeAnnouncesL2VIP + VpnIngressNodeLabelMissing +# (platform/monitoring/alerts.yaml) +# +# Run with: platform/monitoring/tests/run.sh +# (the merge gate runs the same harness via +# conformance/tests/l0-static/alert-rules-unit-tested) +# +# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS +# MEASURED on prod 2026-10-08: a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement +# lease it holds ITSELF -- the announcer does not answer its own ARP request, so on the lease holder +# the neighbour entry settles at `FAILED` and packets to that VIP die before leaving the node. +# platform/vpn-ingress runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's +# own address, so a VPN client takes that node's LAN path and inherits the failure. With the +# router's udp/51820 forward on k3s-prod-1, which held the `shared` Gateway lease, all 16 Gateway +# hostnames were dead over the VPN -- while PiHole DNS kept answering, because its VIP has two +# Services and k3s-prod-2 held the other lease. The tunnel handshook, names resolved, every app +# hung, and nothing reported it for ~2 days. +# +# platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled +# `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so VpnIngressNodeAnnouncesL2VIP +# should be permanently unsatisfiable. THAT IS EXACTLY WHY IT NEEDS A TEST: an alert expected never +# to fire is indistinguishable, in production, from an alert that CANNOT fire. Nothing would ever +# reveal a mistyped label, a broken `label_replace`, or a join on the wrong label. Test 1 proves it +# fires on the real failure; test 2 proves it is quiet in the fixed state, so it is not a rule that +# fires on anything; test 3 pins the `for: 15m` against routine lease churn. +# +# THE JOIN IS THE FRAGILE PART AND IS TESTED DELIBERATELY. `kube_lease_owner` names the holder in +# `lease_holder`; `kube_node_labels` names it `node`. The rule bridges them with `label_replace` +# and joins `on (node) group_left ()`. Fixtures therefore carry BOTH series shapes with real label +# sets, because a join that silently matches nothing returns no series -- which looks exactly like +# "healthy" and is the failure mode this whole group exists to prevent. +# +# Fixture values are prod's real shape (measured 2026-10-08): leases +# `cilium-l2announce-gateway-cilium-gateway-shared` (k3s-prod-1) and +# `cilium-l2announce-pihole-pihole-dns-tcp` (k3s-prod-2), in namespace kube-system, with +# k3s-prod-1 the vpn-ingress node. + +rule_files: + - generated.rules.yaml + +evaluation_interval: 1m + +tests: + # ============================================================================================= + # 1. THE BUG, EXACTLY AS IT HAPPENED -- the vpn-ingress node holds the Gateway lease: FIRES. + # ============================================================================================= + - interval: 1m + name: vpn-ingress node holds the shared Gateway L2 lease - VpnIngressNodeAnnouncesL2VIP fires + input_series: + # The collision. A lease holder keeps renewing, so this series does not lapse on its own -- + # which is why the condition never self-heals and why a 15m `for:` is safe. + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}' + values: '1+0x40' + # A second lease held by a DIFFERENT node, present throughout. This is the discriminator that + # stops the rule from being "any l2announce lease exists anywhere": if the join were dropped, + # this series alone would fire the alert, and test 2 would fail. + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-pihole-pihole-dns-tcp",lease_holder="k3s-prod-2"}' + values: '1+0x40' + # k3s-prod-1 is the vpn-ingress node. Only this node carries the label. + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}' + values: '1+0x40' + # The other two nodes publish kube_node_labels WITHOUT the vpn-ingress label. Present so the + # join is exercised against non-matching nodes rather than against nothing at all. + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}' + values: '1+0x40' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}' + values: '1+0x40' + alert_rule_test: + # Held 15m, so nothing before then. + - eval_time: 14m + alertname: VpnIngressNodeAnnouncesL2VIP + exp_alerts: [] + - eval_time: 20m + alertname: VpnIngressNodeAnnouncesL2VIP + exp_alerts: + # EXACTLY ONE alert: the pihole lease on prod-2 must NOT produce a second one. + - exp_labels: + severity: critical + lease: cilium-l2announce-gateway-cilium-gateway-shared + exp_annotations: + summary: "VPN ingress node is announcing a LoadBalancer VIP (cilium-l2announce-gateway-cilium-gateway-shared)" + description: >- + The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access + WireGuard, and it now holds the Cilium L2 announcement lease `cilium-l2announce-gateway-cilium-gateway-shared`. A + node cannot reach a VIP it announces itself, so every remote VPN client has just lost + all access to that VIP's Service -- silently, with DNS still resolving. Check that + platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that + the label is still on the node the router forwards udp/51820 to. + runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" + # The label IS present, so the watchdog must stay quiet here. + - eval_time: 40m + alertname: VpnIngressNodeLabelMissing + exp_alerts: [] + + # ============================================================================================= + # 2. THE FIXED STATE -- every lease is held by a node that is NOT the vpn-ingress node: QUIET. + # This is what l2-announcement-policy.yaml's exclusion produces, and the assertion that keeps + # test 1 meaningful rather than an alert that fires on anything. + # ============================================================================================= + - interval: 1m + name: all l2 leases held by other nodes - VpnIngressNodeAnnouncesL2VIP stays quiet + input_series: + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-2"}' + values: '1+0x40' + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-plex-plex-l4-tcp",lease_holder="k3s-prod-3"}' + values: '1+0x40' + # A lease the vpn-ingress node DOES legitimately hold, in a different family -- proves the + # `cilium-l2announce-.*` matcher is doing work. Every node holds operator/controller leases; + # only the ANNOUNCEMENT ones matter here, and a matcher broad enough to catch this would page + # on a healthy cluster forever. + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-operator-resource-lock",lease_holder="k3s-prod-1"}' + values: '1+0x40' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}' + values: '1+0x40' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}' + values: '1+0x40' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}' + values: '1+0x40' + alert_rule_test: + - eval_time: 40m + alertname: VpnIngressNodeAnnouncesL2VIP + exp_alerts: [] + - eval_time: 40m + alertname: VpnIngressNodeLabelMissing + exp_alerts: [] + + # ============================================================================================= + # 3. A LEGITIMATE, TRANSIENT LEASE MOVE -- the vpn-ingress node holds it for 11m during a node + # reboot or a Cilium roll, then Cilium re-elects away: QUIET. This pins the `for: 15m` as + # deliberate. Without it, every rolling restart of the cilium DaemonSet would page, and a paging + # alert that cries wolf on routine maintenance gets muted -- which is how the real collision + # would go unseen again. + # ============================================================================================= + - interval: 1m + name: transient lease move shorter than for 15m - VpnIngressNodeAnnouncesL2VIP stays quiet + input_series: + # Held by the vpn-ingress node for 11 samples, then the series goes absent (the lease moved, + # so kube-state-metrics retires this label set and starts a new one). + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}' + values: '1+0x10 _x30' + # Re-elected onto prod-3 from t=11m. A label VALUE change retires one series and starts + # another, which is why this is a second series and not a changing value. + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-3"}' + values: '_x11 1+0x29' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}' + values: '1+0x40' + alert_rule_test: + # Inside the transient window: pending at most, never firing. + - eval_time: 10m + alertname: VpnIngressNodeAnnouncesL2VIP + exp_alerts: [] + # Well past it, with the lease settled on another node. + - eval_time: 30m + alertname: VpnIngressNodeAnnouncesL2VIP + exp_alerts: [] + + # ============================================================================================= + # 4. THE WATCHDOG -- no node carries the label at all. The policy's `NotIn` then excludes + # NOTHING (a missing key matches), so the 2026-10-08 failure is fully re-armed, AND + # VpnIngressNodeAnnouncesL2VIP is structurally INERT: its join matches nothing and returns no + # series, which is indistinguishable from healthy. Both halves are asserted here, because an + # inert critical alert is the exact shape this repo has been burned by before. + # ============================================================================================= + - interval: 1m + name: vpn-ingress label absent everywhere - watchdog fires and the critical alert is inert + input_series: + # A lease held by the node that IS (in reality) the forwarded one -- but with no label + # anywhere, nothing can tell. This is the dangerous state. + - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}' + values: '1+0x60' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}' + values: '1+0x60' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}' + values: '1+0x60' + - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}' + values: '1+0x60' + alert_rule_test: + # The critical alert CANNOT fire -- this is the inertness being pinned, not an aside. + - eval_time: 45m + alertname: VpnIngressNodeAnnouncesL2VIP + exp_alerts: [] + # ... which is precisely why the watchdog must. + - eval_time: 45m + alertname: VpnIngressNodeLabelMissing + exp_alerts: + - exp_labels: + severity: warning + # absent() propagates the matcher's own labels onto the series it returns, so the + # alert inherits this one. Asserted rather than worked around: it is harmless, and + # pinning it means a future change to the matcher cannot alter the alert's identity + # (and therefore its Alertmanager grouping) unnoticed. + label_homelab_keithjackson_dev_vpn_ingress: "true" + exp_annotations: + summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF" + description: >- + No node publishes homelab.keithjackson.dev/vpn-ingress=true, so + l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none: + any node may now announce a LoadBalancer VIP, including the one terminating + remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so + the warning matters more than its severity suggests. Re-run the node_baseline role on + the forwarded node, and confirm the label is in kube-state-metrics' + metricLabelsAllowlist. + runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision" diff --git a/platform/vpn-ingress/README.md b/platform/vpn-ingress/README.md index 3cfc916..add6522 100644 --- a/platform/vpn-ingress/README.md +++ b/platform/vpn-ingress/README.md @@ -117,8 +117,19 @@ boundary around this port is WireGuard's peer authentication, full stop. The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400 forward). A single-replica Deployment would float on every reschedule and silently invalidate it. A DaemonSet instead puts an identical listener on all three nodes — every node answers the same -keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no -re-issued client profiles. The two idle listeners cost ~15 MiB each. +keys — so losing the forwarded node never costs a re-issued client profile. The two idle listeners +cost ~15 MiB each. + +> **This section used to say "losing the forwarded node is a one-line router change, with no cluster +> work". It was wrong, and it cost two days of silently broken remote access on 2026-10-08.** The +> nodes are warm, but they are **not interchangeable**: the node the forward points at must be the +> one labelled `homelab.keithjackson.dev/vpn-ingress=true`, because that is the node +> `platform-config/cilium/l2-announcement-policy.yaml` excludes from announcing LoadBalancer VIPs. +> **A node cannot reach a VIP whose Cilium L2 announcement lease it holds itself** (see "Measured on +> prod, 2026-10-08" below), and since this pod MASQUERADEs clients to the node's own address, a VPN +> client inherits that. Failover is therefore **two** changes: move +> `node_baseline_vpn_ingress_node` and re-run that role, then re-point the router — +> `runbooks/vpn-remote-access.md` step 3. A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node @@ -186,6 +197,45 @@ right way; one was a defect in my own manifest. | Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. | | Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. | +## Measured on prod, 2026-10-08 — why the forwarded node cannot be any node + +The first device enrolled, handshook, resolved names, and could not open a single connection to any +cluster web app. The tunnel was not at fault — every part of it measured healthy: +`net.ipv4.ip_forward=1`, `wg0` up at `10.90.0.1/24` MTU 1420, the peer authorised with real +bidirectional transfer (346 KiB in / 200 KiB out), `MASQUERADE` present and counting 26 flows, +`FORWARD -i/-o wg0 ACCEPT` counting 108 / 171 packets, and PiHole DNS answering from the node. + +**The fault was that a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it +holds itself.** Cilium's announcer answers ARP for the VIP on behalf of other LAN hosts, but not for +the announcing node's own request, so the neighbour entry settles at `FAILED`: + +| from | `192.168.2.20` (Gateway) | `192.168.2.22` (crafty) | `192.168.2.24` (plex) | announces | +|---|---|---|---|---| +| `k3s-prod-1` | **FAILED** | — | — | `.20` | +| `k3s-prod-2` | ok | **INCOMPLETE** | ok | `.21`, `.22`, `.23-tcp` | +| `k3s-prod-3` | ok | ok | **INCOMPLETE** | `.24` | + +Symmetric across 3 nodes × 5 VIPs: each node reached every VIP **except** the ones it alone +announced. It is a property of L2 announcement, not of any one Service, and it is harmless for +ordinary pods — they reach Services through the cluster datapath and never ARP for a VIP. This +component is the exception, because `hostNetwork` + `MASQUERADE` put remote clients on the node's +own LAN path. + +The forward pointed at `k3s-prod-1`, which held the `shared` Gateway lease, so **all 16 Gateway +hostnames were dead over the VPN**. Two things hid it: + +* **PiHole survived by accident.** `192.168.2.23` is announced by *two* Services + (`pihole-dns-udp` on prod-1, `pihole-dns-tcp` on prod-2), so prod-2 answered the ARP prod-1 could + not. Names resolved perfectly and every app hung — the hardest shape to diagnose from a client. +* **Nothing reported it.** No probe in this repo compares the forwarded node against the lease + holders, and from outside WireGuard is silent to unauthenticated traffic anyway. + +Closed by labelling the forwarded node `homelab.keithjackson.dev/vpn-ingress=true` and excluding +that label in `platform-config/cilium/l2-announcement-policy.yaml` (the node then ARP-resolves every +VIP), with `VpnIngressNodeAnnouncesL2VIP` watching the lease holders so a recurrence is loud and +`VpnIngressNodeLabelMissing` watching for the label going absent, which would silently re-arm the +whole failure. Cost: VIP announcement candidacy drops from three nodes to two. + ## Prerequisites outside this directory 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules` diff --git a/platform/vpn-ingress/daemonset.yaml b/platform/vpn-ingress/daemonset.yaml index c1a203d..64a07e4 100644 --- a/platform/vpn-ingress/daemonset.yaml +++ b/platform/vpn-ingress/daemonset.yaml @@ -62,9 +62,39 @@ # as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to # whichever node the scheduler picked and silently invalidate that forward on every reschedule. A # DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of -# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no -# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own -# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each. +# them, and if that node is lost every other node is already warm and answers the same keys, so no +# client profile ever changes. Each node binds udp/51820 in its own netns, so there is no conflict, +# and the two idle listeners cost ~15 MiB each. +# +# BUT THE NODES ARE NOT INTERCHANGEABLE, AND THIS COMMENT USED TO CLAIM THEY WERE. +# It said "FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work". That was wrong, and the +# wording cost two days of silently broken remote access on 2026-10-08. MEASURED that day: a node +# cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds ITSELF — the announcer +# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and packets to +# that VIP die before leaving the node: +# +# k3s-prod-1, holding the shared Gateway lease: ip neigh show 192.168.2.20 -> FAILED +# k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr …(prod-1) +# +# Because this pod MASQUERADEs clients to the node's own address (see PostUp below), a remote client +# inherits exactly that failure. With the forward on the Gateway's announcing node, all 16 Gateway +# hostnames were unreachable over the VPN while PiHole DNS still resolved — a tunnel that +# handshakes, resolves names, and then hangs on every single app. +# +# THE FIX IS DECLARATIVE AND LIVES OUTSIDE THIS FILE: the forwarded node is labelled +# homelab.keithjackson.dev/vpn-ingress=true by infra/ansible/roles/node_baseline (from +# node_baseline_vpn_ingress_node), and platform-config/cilium/l2-announcement-policy.yaml excludes +# that label from announcing ANY VIP, so the node can ARP-resolve all of them. A label rather than a +# cluster-vars knob because the alert that guards this has to read it too, and the alert unit-test +# harness refuses a postBuild placeholder in an expr. Failover is therefore TWO steps — move +# node_baseline_vpn_ingress_node and re-run the role, THEN re-point the router +# (runbooks/vpn-remote-access.md step 3). +# `VpnIngressNodeAnnouncesL2VIP` (platform/monitoring/alerts.yaml) fires if the two collide again. +# +# This pod deliberately does NOT probe that itself. A readinessProbe on VIP reachability would wedge +# a DaemonSet rollout behind an unrelated Gateway outage (maxUnavailable: 1 below), and a liveness +# probe would tear down a tunnel that is carrying LAN traffic perfectly well. The condition is a +# property of the CLUSTER's announcement layout, so it is watched there. # # All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the # DaemonSet to cover the cluster; add them if a node is ever tainted. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index f01643a..7adc398 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -119,6 +119,11 @@ configMapGenerator: - update-rollback=update-rollback.md - vpn-down=vpn-down.md - vpn-egress-stale=vpn-egress-stale.md + # `VpnIngressNodeAnnouncesL2VIP` — the INBOUND remote-access VPN, not the egress tunnel the two + # pages above cover. A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease + # it holds itself, and vpn-ingress MASQUERADEs clients to the node's own address, so the + # forwarded node must announce nothing (measured 2026-10-08). + - vpn-ingress-l2-collision=vpn-ingress-l2-collision.md # Operator procedure, not an alert page. Served because the component it completes is INERT # until three off-repo manual steps are done (kernel module, server key, WAN port-forward) — # and because the one check that actually proves it cannot be run from inside the house. diff --git a/runbooks/vpn-ingress-l2-collision.md b/runbooks/vpn-ingress-l2-collision.md new file mode 100644 index 0000000..38bb489 --- /dev/null +++ b/runbooks/vpn-ingress-l2-collision.md @@ -0,0 +1,165 @@ +# vpn-ingress-l2-collision — the VPN ingress node is announcing a LoadBalancer VIP + +**Alerts:** `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) and `VpnIngressNodeLabelMissing` +(warning, 30m) +**File:** `platform/monitoring/alerts.yaml` +**Related:** `/runbooks/vpn-remote-access` is the setup and enrolment runbook for this component. +`/runbooks/vpn-egress-stale` and `/runbooks/vpn-down` are the **other** VPN — the outbound PIA +tunnel — and have nothing to do with this. + +## What this means here + +**A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The +announcer answers ARP for the VIP on behalf of other LAN hosts, but not for the announcing node's +own request. On the lease holder the neighbour entry settles at `FAILED`, and every packet to that +VIP is dropped before it leaves the node. Measured on prod 2026-10-08, symmetric across all three +nodes and five VIPs — each node reached every VIP **except** the ones it alone announced. + +For an ordinary workload this is harmless: pods reach Services through the cluster datapath and +never ARP for a VIP. `platform/vpn-ingress` is the exception. It runs WireGuard with +`hostNetwork: true` and MASQUERADEs remote clients to the node's own address, so a remote client's +packets take that node's LAN path and inherit its ARP failures. **If the node the router forwards +`udp/51820` to is also an L2 lease holder, every remote VPN client silently loses all access to the +Services behind the VIPs it announces.** + +The node terminating WireGuard is marked with `homelab.keithjackson.dev/vpn-ingress=true`, and +`platform-config/cilium/l2-announcement-policy.yaml` excludes that label from announcing anything, +so this should be unreachable. + +**The symptom your users report will not sound like a network fault.** On 2026-10-08 the forward sat +on `k3s-prod-1`, which held the `shared` Gateway lease, so all 16 Gateway hostnames were dead over +the VPN — while PiHole DNS kept answering, because `192.168.2.23` is announced by *two* Services and +`k3s-prod-2` held the other lease. So the tunnel handshakes, `wg show` looks perfect, names resolve, +and **every app hangs on connect**. Expect "the VPN connects but nothing loads", not "the VPN is +down". + +Which Services are affected depends on which lease moved — read `{{ $labels.lease }}` from the +alert. `cilium-l2announce-gateway-cilium-gateway-shared` is the worst case: it is every web app at +once. + +### If it is `VpnIngressNodeLabelMissing` that fired + +That is the watchdog, and it means the protection is **switched off**, not that something is merely +untidy. The exclusion is a `NotIn` on the label, and `NotIn` with an absent key **matches** — so +with the label on no node, the policy excludes nothing and every node is an announcement candidate +again. `VpnIngressNodeAnnouncesL2VIP` is simultaneously **inert**: its join matches nothing and +returns no series, which is indistinguishable from healthy. Treat a quiet critical alert as +meaningless until this warning is cleared. Usual cause: a node was rebuilt and `node_baseline` has +not been re-run on it. + +## First commands + +Run cluster commands through the pinned toolbox, which supplies the kube context: + +```bash +# 1. WHO HOLDS WHAT, and WHO IS THE VPN NODE — the whole question, in two lines. +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kube-system get lease | grep l2announce' +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/vpn-ingress' +# Exactly ONE node should carry vpn-ingress=true, and NO l2announce lease should name it. + +# 2. IS THE EXCLUSION STILL IN THE LIVE POLICY? Expect a matchExpressions list with a +# homelab.keithjackson.dev/vpn-ingress NotIn [true] entry — not a bare matchLabels. +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod get ciliuml2announcementpolicy default-l2 -o jsonpath="{.spec.nodeSelector}"; echo' + +# 3. IS THE LABEL REACHING PROMETHEUS? If this is empty the alerts are inert even when +# the label is on the node (kube-state-metrics metricLabelsAllowlist). +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n monitoring exec deploy/kube-prometheus-stack-kube-state-metrics -- \ + wget -qO- localhost:8080/metrics | grep vpn_ingress | head' +``` + +**Confirm the actual damage**, which is the one measurement that is unambiguous. From the WireGuard +pod on the affected node (it shares the host netns, so this is the path a VPN client takes): + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress get pods -o wide' # pick the pod on that node + +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress exec -c wireguard -- \ + sh -c "ping -c1 -W1 192.168.2.20 >/dev/null 2>&1; ip neigh show 192.168.2.20"' +# BROKEN: 192.168.2.20 dev eno1 FAILED +# HEALTHY: 192.168.2.20 dev eno1 lladdr bc:e9:2f:7e:c4:a6 STALE|REACHABLE|DELAY +``` + +The `ping` is only there to trigger ARP — the VIP does not answer ICMP even when healthy, so +**ignore its 100% packet loss**; the neighbour state is the answer. Substitute the VIP that belongs +to `{{ $labels.lease }}` if the Gateway is not the one that moved. + +**Which node actually holds the forward** (the router cannot be queried, but the cluster knows): +only the forwarded node's `wg show` has a peer with an `endpoint` and a recent `latest handshake`. + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'for p in $(kubectl --context k3s-prod -n vpn-ingress get pods -o name); do + echo "== $p"; kubectl --context k3s-prod -n vpn-ingress exec ${p#pod/} -c wireguard -- wg show; done' +``` + +## Likely causes, most likely first + +1. **The label is on the wrong node** — someone re-pointed the router's forward during a node + failover and did not move `node_baseline_vpn_ingress_node`. This is the most likely cause by a + wide margin, because the router is off-repo and nothing can check it. Confirm with the `wg show` + loop above: the node with the live peer endpoint is the real forward target. + *Fix:* merge a PR setting `node_baseline_vpn_ingress_node` + (`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`) to that node, then re-run the role — + `ansible-playbook -i inventory/hosts.ini -e k3s_hosts=k3s_prod --tags vpn-ingress`. + The role labels the new node and **removes** the label from the old one. Then confirm Cilium + re-elected the lease off the newly-labelled node. + +2. **The exclusion was edited out of `platform-config/cilium/l2-announcement-policy.yaml`**, or the + file was reverted to a bare `matchLabels: {kubernetes.io/os: linux}`. Command 2 above shows this + directly. *Fix:* restore the `matchExpressions` form; the file's own header explains why it is + shaped that way. + +3. **The label is missing entirely** (`VpnIngressNodeLabelMissing` firing) — a node rebuild without + a `node_baseline` re-run. *Fix:* re-run the role as above. + +4. **The label is on the node but not in Prometheus** — `metricLabelsAllowlist` in + `platform/monitoring/kube-prometheus-stack.yaml` lost the `vpn-ingress` entry. Then the critical + alert is inert *and* the watchdog fires. Command 3 above distinguishes this from cause 3. + +5. **Cilium kept renewing a lease its holder is no longer eligible for.** The exclusion is correct + and in place, but the lease never re-elected. This was never verified when the fix was authored + (see `decisions/2026-10-17-vpn-ingress-l2-lease-collision.md`). *Fix:* delete that one Lease + object to force an election — `kubectl -n kube-system delete lease ` — and watch it come + back on a different node. + +6. **A Cilium upgrade changed `NotIn` semantics or the `nodeSelector` field.** Least likely, but it + is the reason this is a runtime alert rather than a static check on the policy text. + +## When to escalate or stop + +**Act now, don't wait for a window.** Remote access is fully broken for the affected VIPs while this +is firing, and it does **not** self-heal: a lease holder keeps renewing, so this condition persists +indefinitely once entered. That is also why the 15m `for:` is safe — a transient move during a node +reboot or a Cilium roll resolves on its own well inside it. + +**The fastest honest mitigation is to move the forward, not the leases.** Re-pointing the router's +`udp/51820` at a node that holds no lease for the affected VIP restores service in seconds, with no +cluster change and no new client profiles — all three nodes answer the same keys. Follow it with the +`node_baseline_vpn_ingress_node` PR and role re-run so the two agree again; until you do, this alert +is watching the wrong node. Note each node's own announcements when choosing: as of 2026-10-08, +prod-2 announced crafty `192.168.2.22` and unifi `192.168.2.21`, prod-3 announced plex +`192.168.2.24`, so neither is a cost-free target. + +**Do not** `kubectl edit` the `CiliumL2AnnouncementPolicy` to fix this — Flux owns it and will revert +you, and the drift is the kind that outlives the memory of making it. The two acceptable hand +actions are deleting a single Lease to force re-election (cause 5), and `kubectl label node` as a +*temporary* stopgap while the PR lands — note that a node rebuild or a `node_baseline` run will undo +a hand-applied label, so it is not a fix. + +**Stop and escalate to the operator if** the leases will not leave the excluded node after a forced +re-election, or if the intended forward target is unclear. Both need a decision about which node +carries remote access, which is an operator call, not a runbook step. + +**Known gap, so you are not misled:** if someone re-points the router without moving the label, +`VpnIngressNodeAnnouncesL2VIP` watches the wrong node and **stays silent** while the VPN breaks in +exactly the way described above. There is no in-cluster signal for that today; closing it needs a +WireGuard exporter publishing which node is actually terminating peer traffic +(`platform/vpn-ingress/README.md`, "Not built") — the `wg show` loop above is the manual substitute. +Until then, if remote access is broken and both alerts are quiet, **check the forward against the +label by hand before believing the alerts.** diff --git a/runbooks/vpn-remote-access.md b/runbooks/vpn-remote-access.md index 9f123ab..dfdd874 100644 --- a/runbooks/vpn-remote-access.md +++ b/runbooks/vpn-remote-access.md @@ -171,14 +171,57 @@ server without a `wg0` "ready". On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g. `192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check. -Pick any node — all three run an identical listener. Note which one you chose here: +> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node. +> +> This step used to say "pick any node — all three run an identical listener". **That sentence +> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a +> cluster-vars knob instead of a free choice here. +> +> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The +> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease +> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they +> leave the node. Measured on prod 2026-10-08: +> +> ```text +> on k3s-prod-1 (held the shared Gateway lease): ip neigh show 192.168.2.20 -> FAILED +> on k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr … (prod-1) +> ``` +> +> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN +> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway +> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so +> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported +> it. +> +`infra/ansible/roles/node_baseline` now labels the forwarded node +> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in +> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes +> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI +> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a +> labelled node taking a lease, but it CANNOT catch the forward being moved without the label. + +Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`). +Note it here: ``` # WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set +# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml ``` -**Node loss = failover is this one line.** Re-point the forward at another node address; every -device keeps working with the same profile, because all three nodes answer the same keys. +**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that +was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and +reintroduces the failure above against a different set of addresses (prod-2 announces crafty and +unifi; prod-3 announces plex). Do both, in this order: + +1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role + (`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium + then moves the leases off the newly-labelled node and onto its peers. +2. **Re-point the router** forward at the new node's address. + +Every device keeps working with the same profile either way; all three nodes answer the same keys, +and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that +handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because +that alert watches the node the knob names, not the node your router actually points at. Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is already silent to unauthenticated probes (README, "Why this is safe to expose").