Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

revert(vpn-ingress): withdraw the L2 lease-collision fix — a node CAN reach a VIP it announces
gate / gate (pull_request) Successful in 1m52s
Details
gate / conformance (pull_request) Has been skipped
Details

Browse Source 
PR #69 claimed a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
lease it holds, and excluded the VPN ingress node from announcing on that basis.
The claim makes a sharp prediction; today's power-cycle put ALL EIGHT
cilium-l2announce-* leases on k3s-prod-1, and the prediction failed.

Measured 2026-10-17 from the wireguard pod's host netns on each node: the holder
reaches 192.168.2.20/.22/.23/.24 normally (curl 200 in 18ms on .20) while its
ip neigh for every one of them sits at FAILED. The asserted cause is present
without the asserted effect, which falsifies it. .21 (unifi) fails from all three
nodes and is a separate pre-existing fault.

It is inert because no path resolves a VIP by ARP: cilium-dbg reports
KubeProxyReplacement on [eno1, wg0] and programs 192.168.2.20:443 =>
127.0.0.1:13606, so VIPs are DNAT'd in BPF before any neighbour lookup, on the
forwarded path as well as the host one.

Both probes behind the original diagnosis were bad. `ip neigh` FAILED is
permanent and harmless, and a ping artefact. `curl --interface wg0` — the control
used to rule out a host-netns artefact — fails on all three nodes, including ones
whose ARP resolves, so it discriminates nothing. The deeper error: a MASQUERADEd
client is FORWARDED while every pod-runnable probe is socket-originated. Sharing
a netns is not sharing a hook.

Reverted the policy nodeSelector, the Ansible node label, both alerts, their
promtool tests and the kube-state-metrics allowlist entry. Kept documentation
only: the measurements, which probes lie, and the fact that the forwarded path
remains unmeasured without a connected client.

Also recorded, for the reported "LAN and cluster work, internet does not":
new-peer.sh has emitted a FULL tunnel (0.0.0.0/0) by default since f92707f, so
there is no full-tunnel option to add and no code change is made. The server side
is capable — ip_forward=1, the 10.90.0.0/24 MASQUERADE present with an empty
CILIUM_POST_nat ahead of it, FORWARD accepts, default route intact, nodes reach
1.1.1.1. A full tunnel egresses the HOUSE WAN, not PIA: the egress policy selects
pods by egress-class=vpn, this pod is labelled internet, and tun0 exists on
k3s-prod-3 only. Root cause NOT found; no client has connected since the
power-cycle (0 forwarded packets, no handshake), so it is not reproducible here.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-17 23:58:40 -05:00
1 parent 4656b07178
commit e82dab5df5
13 files changed
+338 -871
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

runbooks/kustomization.yaml
-5
View File
@@ -119,11 +119,6 @@ configMapGenerator:
- update-rollback=update-rollback.md
- vpn-down=vpn-down.md
- vpn-egress-stale=vpn-egress-stale.md
# `VpnIngressNodeAnnouncesL2VIP` — the INBOUND remote-access VPN, not the egress tunnel the two
# pages above cover. A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease
# it holds itself, and vpn-ingress MASQUERADEs clients to the node's own address, so the
# forwarded node must announce nothing (measured 2026-10-08).
- vpn-ingress-l2-collision=vpn-ingress-l2-collision.md
# Operator procedure, not an alert page. Served because the component it completes is INERT
# until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
# and because the one check that actually proves it cannot be run from inside the house.
runbooks/vpn-ingress-l2-collision.md
-165
View File
@@ -1,165 +0,0 @@
# vpn-ingress-l2-collision — the VPN ingress node is announcing a LoadBalancer VIP
**Alerts:** `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) and `VpnIngressNodeLabelMissing`
(warning, 30m)
**File:** `platform/monitoring/alerts.yaml`
**Related:** `/runbooks/vpn-remote-access` is the setup and enrolment runbook for this component.
`/runbooks/vpn-egress-stale` and `/runbooks/vpn-down` are the **other** VPN — the outbound PIA
tunnel — and have nothing to do with this.
## What this means here
**A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
announcer answers ARP for the VIP on behalf of other LAN hosts, but not for the announcing node's
own request. On the lease holder the neighbour entry settles at `FAILED`, and every packet to that
VIP is dropped before it leaves the node. Measured on prod 2026-10-08, symmetric across all three
nodes and five VIPs — each node reached every VIP **except** the ones it alone announced.
For an ordinary workload this is harmless: pods reach Services through the cluster datapath and
never ARP for a VIP. `platform/vpn-ingress` is the exception. It runs WireGuard with
`hostNetwork: true` and MASQUERADEs remote clients to the node's own address, so a remote client's
packets take that node's LAN path and inherit its ARP failures. **If the node the router forwards
`udp/51820` to is also an L2 lease holder, every remote VPN client silently loses all access to the
Services behind the VIPs it announces.**
The node terminating WireGuard is marked with `homelab.keithjackson.dev/vpn-ingress=true`, and
`platform-config/cilium/l2-announcement-policy.yaml` excludes that label from announcing anything,
so this should be unreachable.
**The symptom your users report will not sound like a network fault.** On 2026-10-08 the forward sat
on `k3s-prod-1`, which held the `shared` Gateway lease, so all 16 Gateway hostnames were dead over
the VPN — while PiHole DNS kept answering, because `192.168.2.23` is announced by *two* Services and
`k3s-prod-2` held the other lease. So the tunnel handshakes, `wg show` looks perfect, names resolve,
and **every app hangs on connect**. Expect "the VPN connects but nothing loads", not "the VPN is
down".
Which Services are affected depends on which lease moved — read `{{ $labels.lease }}` from the
alert. `cilium-l2announce-gateway-cilium-gateway-shared` is the worst case: it is every web app at
once.
### If it is `VpnIngressNodeLabelMissing` that fired
That is the watchdog, and it means the protection is **switched off**, not that something is merely
untidy. The exclusion is a `NotIn` on the label, and `NotIn` with an absent key **matches** — so
with the label on no node, the policy excludes nothing and every node is an announcement candidate
again. `VpnIngressNodeAnnouncesL2VIP` is simultaneously **inert**: its join matches nothing and
returns no series, which is indistinguishable from healthy. Treat a quiet critical alert as
meaningless until this warning is cleared. Usual cause: a node was rebuilt and `node_baseline` has
not been re-run on it.
## First commands
Run cluster commands through the pinned toolbox, which supplies the kube context:
```bash
# 1. WHO HOLDS WHAT, and WHO IS THE VPN NODE — the whole question, in two lines.
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n kube-system get lease | grep l2announce'
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/vpn-ingress'
# Exactly ONE node should carry vpn-ingress=true, and NO l2announce lease should name it.
# 2. IS THE EXCLUSION STILL IN THE LIVE POLICY? Expect a matchExpressions list with a
# homelab.keithjackson.dev/vpn-ingress NotIn [true] entry — not a bare matchLabels.
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get ciliuml2announcementpolicy default-l2 -o jsonpath="{.spec.nodeSelector}"; echo'
# 3. IS THE LABEL REACHING PROMETHEUS? If this is empty the alerts are inert even when
# the label is on the node (kube-state-metrics metricLabelsAllowlist).
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring exec deploy/kube-prometheus-stack-kube-state-metrics -- \
wget -qO- localhost:8080/metrics | grep vpn_ingress | head'
```
**Confirm the actual damage**, which is the one measurement that is unambiguous. From the WireGuard
pod on the affected node (it shares the host netns, so this is the path a VPN client takes):
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress get pods -o wide' # pick the pod on that node
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- \
sh -c "ping -c1 -W1 192.168.2.20 >/dev/null 2>&1; ip neigh show 192.168.2.20"'
# BROKEN: 192.168.2.20 dev eno1 FAILED
# HEALTHY: 192.168.2.20 dev eno1 lladdr bc:e9:2f:7e:c4:a6 STALE|REACHABLE|DELAY
```
The `ping` is only there to trigger ARP — the VIP does not answer ICMP even when healthy, so
**ignore its 100% packet loss**; the neighbour state is the answer. Substitute the VIP that belongs
to `{{ $labels.lease }}` if the Gateway is not the one that moved.
**Which node actually holds the forward** (the router cannot be queried, but the cluster knows):
only the forwarded node's `wg show` has a peer with an `endpoint` and a recent `latest handshake`.
```bash
./infra/devcontainer/dev.sh run bash -c \
'for p in $(kubectl --context k3s-prod -n vpn-ingress get pods -o name); do
echo "== $p"; kubectl --context k3s-prod -n vpn-ingress exec ${p#pod/} -c wireguard -- wg show; done'
```
## Likely causes, most likely first
1. **The label is on the wrong node** — someone re-pointed the router's forward during a node
failover and did not move `node_baseline_vpn_ingress_node`. This is the most likely cause by a
wide margin, because the router is off-repo and nothing can check it. Confirm with the `wg show`
loop above: the node with the live peer endpoint is the real forward target.
*Fix:* merge a PR setting `node_baseline_vpn_ingress_node`
(`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`) to that node, then re-run the role —
`ansible-playbook -i inventory/hosts.ini <playbook> -e k3s_hosts=k3s_prod --tags vpn-ingress`.
The role labels the new node and **removes** the label from the old one. Then confirm Cilium
re-elected the lease off the newly-labelled node.
2. **The exclusion was edited out of `platform-config/cilium/l2-announcement-policy.yaml`**, or the
file was reverted to a bare `matchLabels: {kubernetes.io/os: linux}`. Command 2 above shows this
directly. *Fix:* restore the `matchExpressions` form; the file's own header explains why it is
shaped that way.
3. **The label is missing entirely** (`VpnIngressNodeLabelMissing` firing) — a node rebuild without
a `node_baseline` re-run. *Fix:* re-run the role as above.
4. **The label is on the node but not in Prometheus** — `metricLabelsAllowlist` in
`platform/monitoring/kube-prometheus-stack.yaml` lost the `vpn-ingress` entry. Then the critical
alert is inert *and* the watchdog fires. Command 3 above distinguishes this from cause 3.
5. **Cilium kept renewing a lease its holder is no longer eligible for.** The exclusion is correct
and in place, but the lease never re-elected. This was never verified when the fix was authored
(see `decisions/2026-10-17-vpn-ingress-l2-lease-collision.md`). *Fix:* delete that one Lease
object to force an election — `kubectl -n kube-system delete lease <name>` — and watch it come
back on a different node.
6. **A Cilium upgrade changed `NotIn` semantics or the `nodeSelector` field.** Least likely, but it
is the reason this is a runtime alert rather than a static check on the policy text.
## When to escalate or stop
**Act now, don't wait for a window.** Remote access is fully broken for the affected VIPs while this
is firing, and it does **not** self-heal: a lease holder keeps renewing, so this condition persists
indefinitely once entered. That is also why the 15m `for:` is safe — a transient move during a node
reboot or a Cilium roll resolves on its own well inside it.
**The fastest honest mitigation is to move the forward, not the leases.** Re-pointing the router's
`udp/51820` at a node that holds no lease for the affected VIP restores service in seconds, with no
cluster change and no new client profiles — all three nodes answer the same keys. Follow it with the
`node_baseline_vpn_ingress_node` PR and role re-run so the two agree again; until you do, this alert
is watching the wrong node. Note each node's own announcements when choosing: as of 2026-10-08,
prod-2 announced crafty `192.168.2.22` and unifi `192.168.2.21`, prod-3 announced plex
`192.168.2.24`, so neither is a cost-free target.
**Do not** `kubectl edit` the `CiliumL2AnnouncementPolicy` to fix this — Flux owns it and will revert
you, and the drift is the kind that outlives the memory of making it. The two acceptable hand
actions are deleting a single Lease to force re-election (cause 5), and `kubectl label node` as a
*temporary* stopgap while the PR lands — note that a node rebuild or a `node_baseline` run will undo
a hand-applied label, so it is not a fix.
**Stop and escalate to the operator if** the leases will not leave the excluded node after a forced
re-election, or if the intended forward target is unclear. Both need a decision about which node
carries remote access, which is an operator call, not a runbook step.
**Known gap, so you are not misled:** if someone re-points the router without moving the label,
`VpnIngressNodeAnnouncesL2VIP` watches the wrong node and **stays silent** while the VPN breaks in
exactly the way described above. There is no in-cluster signal for that today; closing it needs a
WireGuard exporter publishing which node is actually terminating peer traffic
(`platform/vpn-ingress/README.md`, "Not built") — the `wg show` loop above is the manual substitute.
Until then, if remote access is broken and both alerts are quiet, **check the forward against the
label by hand before believing the alerts.**
runbooks/vpn-remote-access.md
+81 -46
View File
@@ -171,57 +171,14 @@ server without a `wg0` "ready".
On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
`192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.
> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node.
>
> This step used to say "pick any node — all three run an identical listener". **That sentence
> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a
> cluster-vars knob instead of a free choice here.
>
> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease
> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they
> leave the node. Measured on prod 2026-10-08:
>
> ```text
> on k3s-prod-1 (held the shared Gateway lease): ip neigh show 192.168.2.20 -> FAILED
> on k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr … (prod-1)
> ```
>
> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN
> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway
> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so
> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported
> it.
>
`infra/ansible/roles/node_baseline` now labels the forwarded node
> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in
> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes
> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI
> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a
> labelled node taking a lease, but it CANNOT catch the forward being moved without the label.
Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`).
Note it here:
Pick any node — all three run an identical listener. Note which one you chose here:
```
# WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set <date>
# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml
```
**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that
was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and
reintroduces the failure above against a different set of addresses (prod-2 announces crafty and
unifi; prod-3 announces plex). Do both, in this order:
1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role
(`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium
then moves the leases off the newly-labelled node and onto its peers.
2. **Re-point the router** forward at the new node's address.
Every device keeps working with the same profile either way; all three nodes answer the same keys,
and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that
handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because
that alert watches the node the knob names, not the node your router actually points at.
**Node loss = failover is this one line.** Re-point the forward at another node address; every
device keeps working with the same profile, because all three nodes answer the same keys.
Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
already silent to unauthenticated probes (README, "Why this is safe to expose").
@@ -316,6 +273,84 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
## Triage — "the client reaches the LAN and cluster apps, but not the internet"
Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom
almost always lives, and because the server side has been measured and is not the cause.
**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.**
| the profile says | meaning |
|---|---|
| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. |
| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. |
`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit —
`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client
profile only**, so an already-issued profile keeps whatever it was rendered with until it is
re-rendered and re-imported on the device. The two forms differ by one flag:
```bash
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet>' # full tunnel
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet> --split' # LAN only
```
Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into
`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is
editing `AllowedIPs` in the profile **on the device** — the server does not care.
**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the
client connected, then watch the two counters that only forwarded traffic moves:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c "
wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5;
iptables -t nat -L POSTROUTING -n -v | head -5"'
```
* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which
node the router's `udp/51820` forward points at (the router is unreadable from the cluster).
* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is
upstream of the cluster (router forward, client, or carrier), not here.
* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side
failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there,
which breaks internet while node-local cluster VIPs still work. It has not been observed.
**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly:
* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack
client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel —
so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4:
`curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it,
and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is
blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it).
* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified
2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a
blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on
a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all.
**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the
`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain
**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route
`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the
PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled
`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the
**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6).
## Triage — which L2 announcement lease a node holds does NOT matter
Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node
could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`:
that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh`
for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer
VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend
is a node-local socket. Full measurements and the two misleading probes
(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`.
`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a
separate, pre-existing fault in that service and has nothing to do with the VPN.
## Revoking a device
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the