Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

docs(vpn): stop crediting the nftables kill-switch for a drop it never makes

Browse Source 
Four places in the tree told the reader that the host nftables table is the
authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress
README, egress-steering.yaml, the role README, and the vpn-killswitch test's own
GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway
datapath bpf_redirect()s those skbs to the egress device without invoking
netfilter POSTROUTING. Corrected with the measurement in each place rather than
softened, because a guard the repo trusts and that does not fire is worse than no
guard.

What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops
the traffic when it cannot use its cached egress interface, verified by tcpdump on
both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out
either). The nft table keeps a narrower, now-proven job: node-originated traffic
sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on
k3s-prod-3 when such a packet was routed at the WAN).

Also recorded:

* egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the
  gateway node is what tells Cilium it changed (the egress-uplink node label). A
  future reader's instinct will be to have the node write to that object; the
  comment says not to, because it is Flux-managed, and points at the label and
  `cilium-dbg bpf egress list` as the debugging entry points instead.
* egress-steering.yaml — why the new NFS media mounts are not affected by
  pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns,
  so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server
  too, but that is the second reason, not the first.
* vpn-killswitch test — its recovery step depends on the re-steer path, and could
  not have passed before it existed. Its no-leak assertion was always right; only
  the explanation was wrong. Also states the prerequisite plainly: the test cannot
  run at all without a labelled gateway node.
* platform README — prod is live on k3s-prod-3 with the before/after egress IPs,
  and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no
  matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway
  restored egress rather than moving it off the WAN.
* role README — the standing "add the shared fwmark to the interface contract"
  escalation is closed as not implementable on Cilium OSS, rather than left open
  for someone to attempt.

DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 /
docs-03 amendment for the architect (specs are immutable to agents) and the five
things left for the operator — chief among them that a silent gateway outage still
has no alert, which is what let the dev gateway sit dead for 80 minutes.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-12 17:57:52 -05:00
1 parent 60e3f09c67
commit 04606d48b2
5 files changed
+479 -53
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

platform/vpn-egress/README.md
+101 -5
View File
@@ -8,8 +8,12 @@ Egress is realized per **[ADR-0015](../../../home-network-k3s/decisions/adr-0015
by a **Cilium Egress Gateway** policy, with a **host nftables kill-switch**. There is **no in-cluster
gateway pod** — the tunnel is host infrastructure, not a Flux workload.
> **STATUS: live + conformance-green.** The node tunnel was validated end-to-end; the `egress-class`
> (R-NET-05/07) and `vpn-killswitch` (R-NET-06) conformance tests pass against the live cluster.
> **STATUS: live on BOTH clusters** — `k3s-dev-3` and, since 2026-09-12, `k3s-prod-3`.
> The `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) tests pass on dev.
> **Caveat on that "conformance-green":** the `vpn-killswitch` test stops and then restarts the
> gateway tunnel, and until 2026-09-12 a restarted tunnel never recovered (Cilium kept a stale
> egress ifindex — see below), so the test's recovery step could not have passed on the
> implementation as it stood. Whatever produced the earlier green result, re-run it.
## How it works (ADR-0015)
Three layers, two owners:
@@ -24,9 +28,101 @@ Three layers, two owners:
(Kyverno-enforced) and SNATs them out the gateway node's `tun0`. Native Cilium eBPF — no client-pod
changes, no sidecar, no overlay. `excludedCIDRs` keeps RFC1918 / cluster / LAN traffic on the local
datapath; only public destinations leave via the tunnel.
- **Kill-switch — host nftables (Ansible).** A host nftables rule on the gateway node DROPS the steered
`vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so
`internet`-class and node traffic are unaffected.
- **Kill-switch — Cilium's own drop, NOT host nftables (corrected 2026-09-12).** vpn-class fails
closed because Cilium drops the traffic when it cannot use its cached egress interface. The host
nftables table is retained for a narrower job. See "The kill-switch does not do what this file
used to claim" below — that section is the measurement, not an opinion.
## LIVE ON PROD since 2026-09-12 — `k3s-prod-3`
| | dev | prod |
|---|---|---|
| gateway node | `k3s-dev-3` | `k3s-prod-3` |
| inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) |
| vpn-class egress IP | `158.173.241.36` (PIA Stockholm) | `158.173.241.36` (PIA Stockholm) |
| internet-class egress IP | home WAN | `136.51.115.252` (home WAN) |
Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three
nodes — all egress the PIA address; plex and nextcloud (internet-class) still egress the home
WAN. **Before the gateway existed, prod's vpn-class pods had no internet at all**: the CEGP was
reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress list` showed
`Gateway IP: Not Found` for their `0.0.0.0/0` entry and the traffic was dropped. Arming the
gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also
means nothing had been leaking in the meantime.
Both environments currently share one PIA account and, as it happens, one public egress IP.
That is inside the account's connection limit, but it means the two environments are not
distinguishable from the outside; a separate prod PIA account is the clean fix if that ever
matters (see `infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`).
## The kill-switch does not do what this file used to claim (measured 2026-09-12)
This README, `egress-steering.yaml`, and the role all used to say the **host nftables
kill-switch** is the authoritative fail-closed backstop for `vpn`-class traffic. **It cannot be.**
With `bpf.masquerade: true` — which Egress Gateway requires — the SNAT and the egress happen in
eBPF, which `bpf_redirect()`s the skb straight to the egress device. netfilter POSTROUTING is
never invoked for those packets.
A counters-only nft table installed alongside the real one on the gateway node, while a vpn-class
pod egressed:
| counter | packets |
|---|---|
| `forward`, saddr = pod CIDR | **0** |
| `forward`, saddr = PIA tunnel range | **0** |
| `postrouting`, oif tun0, saddr = tunnel range | **0** |
| `postrouting`, oif WAN, saddr = pod CIDR | **0** |
| `postrouting`, oif WAN, any (control: the hook IS live) | 45166 |
| `postrouting`, oif tun0, saddr = tunnel (control: a node-originated curl) | 16 |
**What actually fails closed is Cilium itself.** When it cannot use its cached egress
interface/IP it drops the packet. Verified with tcpdump on *both* the WAN and tun0, during a
tunnel stop and during a tunnel restart: **0 packets left either device.** There is no leak — the
guarantee is real — but the mechanism is Cilium's, and the existing `vpn-killswitch` conformance
test passes because of it, not because of the nft rules.
The nft table is **kept**, with a `counter` on every rule, for the narrower thing it genuinely
does cover: **node-originated** traffic sourced from a tunnel address. That was proven on
`k3s-prod-3` in both directions — a source-bound ping with the tunnel ip-rule present got 3/3
replies and moved `killswitch_accept`; with the ip-rule removed (so the routing table sent it out
`eno1`) `killswitch_drop` moved **0 -> 3 packets / 252 bytes** and the ping got nothing back.
If `killswitch_drop` ever starts counting in normal operation, the datapath has changed — read
that as news, not reassurance.
Also removed: the `meta mark 0x1c8 oifname != tun0 drop` "contract hook". Nothing stamps that
mark (the only Cilium mark on the gateway node is `0x200/0xf00`, the L7-proxy rule to table
2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is
a hazard to whatever component picks that value next. The role's `vpn_egress_gateway_fwmark`
variable is gone with it, and the "add the fwmark to the interface contract" escalation in the
role README is **closed as not-implementable** rather than left open.
## Cilium never re-resolves `tun0` — the gateway dies on every reconnect without help
Cilium resolves `interface: tun0` once into a cached (egress IP, egress ifindex) and does not
revisit it. Every tunnel restart re-creates tun0 with a **new ifindex**, so the cached one goes
stale and vpn-class egress is silently blackholed. Found the hard way: the dev gateway had been
dead for ~80 minutes when this work started, with no alert and nothing in the ledger.
`egress-gateway-reconciliation-trigger-interval: 1s` does **not** help; both clusters carry that
setting and dev still did not self-heal 20s+ after the tunnel was back. It coalesces
reconciliations that k8s object events already triggered — netlink link events are not among them.
The Ansible role now publishes `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on
the gateway node on every connect, plus a 2-minute convergence timer. That node update makes
Cilium re-process the policy and re-resolve the interface (verified: cached ifindex 1986 -> 1987
within 10s, egress recovered immediately). A node label rather than an annotation on this
Flux-managed policy, so GitOps state is never written to from a node.
**Debugging a dead gateway starts here:**
```bash
kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink
kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
cilium-dbg bpf egress list # Egress Ifindex must match the node's live `ip link show tun0`
```
If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
gateway node is the next stop.
## Resilience — there is NO gateway failover (spec correction, escalated)
platform/vpn-egress/egress-steering.yaml
+44 -6
View File
@@ -59,16 +59,38 @@ spec:
# Excluded flows fall through to Cilium's normal local datapath (invariant 1).
destinationCIDRs:
- "0.0.0.0/0"
#
# BULK NFS (added to the record 2026-09-12, when plex/sonarr/radarr/nextcloud gained media
# mounts): the export is 192.168.2.102:/mnt/Storage/plex on prod (192.168.1.9 on dev), both
# inside 192.168.0.0/16 below, so NFS is excluded from the tunnel twice over. The FIRST reason
# is the one that actually matters and needs no CIDR at all: an NFS PersistentVolume is mounted
# by the NODE KERNEL in the host netns, not by the pod, so `selectors.podSelector` never
# matches it and no pod-scoped egress steering can touch it. Verified after arming the prod
# gateway: sonarr (k3s-prod-1) and radarr (k3s-prod-3, the gateway node itself) both still list
# AND write the export, and the node's own mounts are intact. No RFC1918 destination needs
# adding here for the media path.
excludedCIDRs:
- "10.0.0.0/8" # RFC1918 + Cilium cluster-pool pod CIDR (CLUSTER_POD_CIDR) + k3s service net
- "172.16.0.0/12" # RFC1918
- "192.168.0.0/16" # RFC1918 (home LAN / node network family)
- "192.168.0.0/16" # RFC1918 (home LAN / node network family / the bulk NFS server)
# --- Where it egresses --------------------------------------------------
# The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`. When
# tun0 is absent/down, Cilium has no egress interface here; the host nftables kill-switch (Ansible,
# ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to
# vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED,
# internet fails OPEN).
# The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`.
#
# CORRECTION 2026-09-12 — this comment used to say the host nftables kill-switch "is the
# authoritative fail-closed backstop that DROPS this vpn-class traffic". It is not, and cannot
# be. With `bpf.masquerade: true` the SNAT and egress happen in eBPF, which bpf_redirect()s the
# skb to the egress device without invoking netfilter POSTROUTING, so the host nft rules never
# see this traffic. Measured with a counters-only nft table on the gateway node while a
# vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and
# the PIA range, versus 45166 packets on that same hook for WAN traffic (so the hook is live)
# and 16 for a node-originated curl out tun0 (so the matching works).
# WHAT ACTUALLY FAILS CLOSED: Cilium itself. When the cached egress interface/IP cannot be used
# it drops the packet — tcpdump on both the WAN and tun0 during a tunnel stop AND during a
# tunnel restart caught 0 packets leaving either device. vpn fails CLOSED (verified, no leak);
# internet-class and node traffic are never selected and fail OPEN (invariants 3 & 4 hold).
# The host nft table is retained, with counters, for the narrower case it CAN see:
# node-originated traffic sourced from a tunnel address (proven on k3s-prod-3 — the drop rule's
# counter moved 0 -> 3 packets when such a packet was routed at the WAN).
#
# SINGLE GATEWAY ONLY — there is NO failover, and a second labelled node is NOT a standby.
# `egressGateway` (singular) resolves the nodeSelector to ONE node. Verbatim from the CRD
@@ -89,6 +111,22 @@ spec:
# ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)"
# is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see
# platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06.
#
# `interface: tun0` IS RESOLVED ONCE — the gateway node has to tell Cilium when tun0 changes.
# Measured on k3s-dev-3 2026-09-12: Cilium caches an (egress IP, egress ifindex) pair from the
# first resolution and does NOT re-resolve on a netlink change. `systemctl restart
# vpn-egress-tunnel` re-creates tun0 with a NEW ifindex; the map kept the old one and vpn-class
# egress was silently blackholed (tcpdump: 0 packets out the WAN, 0 out tun0 — fail-closed, but
# dead). `egress-gateway-reconciliation-trigger-interval: 1s` does not cover link events.
# Upstream: "it does not automatically respond to a change in the gateway node's network
# configuration... You can force a fresh selection by re-applying the Egress Gateway policy."
# The Ansible role closes this: roles/vpn_egress_gateway publishes
# `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node on every
# connect (plus a 2-minute convergence timer). That node update makes Cilium re-process this
# policy and re-resolve the interface — verified: cached ifindex 1986 -> 1987 within 10s.
# DO NOT "fix" this by having the node write to THIS object: it is Flux-managed, and a node
# writing to GitOps state is a drift fight. Check the label, not this file, when debugging:
# kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink
egressGateway:
nodeSelector:
matchLabels: