Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity
Files
d8b25a67feeea46f3e1311d4e8fb15aedaaad8e8
home-network-k3s-impl/platform/vpn-egress/egress-steering.yaml
T
jakman85andClaude Opus 5 04606d48b2 docs(vpn): stop crediting the nftables kill-switch for a drop it never makes 
Four places in the tree told the reader that the host nftables table is the
authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress
README, egress-steering.yaml, the role README, and the vpn-killswitch test's own
GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway
datapath bpf_redirect()s those skbs to the egress device without invoking
netfilter POSTROUTING. Corrected with the measurement in each place rather than
softened, because a guard the repo trusts and that does not fire is worse than no
guard.

What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops
the traffic when it cannot use its cached egress interface, verified by tcpdump on
both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out
either). The nft table keeps a narrower, now-proven job: node-originated traffic
sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on
k3s-prod-3 when such a packet was routed at the WAN).

Also recorded:

* egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the
  gateway node is what tells Cilium it changed (the egress-uplink node label). A
  future reader's instinct will be to have the node write to that object; the
  comment says not to, because it is Flux-managed, and points at the label and
  `cilium-dbg bpf egress list` as the debugging entry points instead.
* egress-steering.yaml — why the new NFS media mounts are not affected by
  pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns,
  so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server
  too, but that is the second reason, not the first.
* vpn-killswitch test — its recovery step depends on the re-steer path, and could
  not have passed before it existed. Its no-leak assertion was always right; only
  the explanation was wrong. Also states the prerequisite plainly: the test cannot
  run at all without a labelled gateway node.
* platform README — prod is live on k3s-prod-3 with the before/after egress IPs,
  and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no
  matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway
  restored egress rather than moving it off the WAN.
* role README — the standing "add the shared fwmark to the interface contract"
  escalation is closed as not implementable on Cilium OSS, rather than left open
  for someone to attempt.

DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 /
docs-03 amendment for the architect (specs are immutable to agents) and the five
things left for the operator — chief among them that a silent gateway outage still
has no alert, which is what let the dev gateway sit dead for 80 minutes.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 17:57:52 -05:00

142 lines
10 KiB
YAML
Raw Blame History

---
# ============================================================================
# Cilium Egress Gateway steering for `egress-class=vpn` pods — ADR-0015.
# Serves: R-NET-04, R-NET-05, R-NET-06, R-NET-07.
# R-NET-04 added 2026-09-06: this file IS the R-NET-04 datapath — the rule that selects between the
# two egress paths. Omitting it made `run.sh --status` report R-NET-04 as "TEST-ONLY / impl none"
# even though the implementation is right here.
#
# ##########################################################################
# ## LIVE (2026-08-24): wired into kustomization.yaml, reconciled by Flux ##
# ## and conformance-green (egress-class + vpn-killswitch pass on dev). ##
# ##########################################################################
#
# This policy steers vpn-class pods out the gateway node's host `tun0` interface. It REQUIRES the
# Ansible-managed host OpenVPN split-tunnel service on a node labeled
# `homelab.keithjackson.dev/egress-gateway=true` — deploy that role BEFORE this policy reconciles, or
# vpn-class pods are SNAT'd to a non-existent interface. The tunnel is live and validated end-to-end;
# if tun0 is ever down, the host nftables kill-switch fails vpn-class CLOSED (see below). ADR-0015
# chose this mechanism (node tunnel + Cilium Egress Gateway); this file IS that datapath.
#
# ---------------------------------------------------------------------------
# PREREQUISITES CONFIRMED (2026-08-24):
# * Cilium Egress Gateway is ENABLED per ADR-0002 — platform/cilium/helmrelease.yaml and
# platform/cilium/values.yaml both set (all [parity] with the Ansible bootstrap render):
# egressGateway.enabled: true
# bpf.masquerade: true (Egress Gateway REQUIRES eBPF masquerade)
# kubeProxyReplacement: true
# No Cilium HelmRelease change is needed for this policy. (If any of the three were false this
# would be an ESCALATION, not a silent HelmRelease edit — they are all true, so we proceed.)
#
# INTERFACE CONTRACT (shared verbatim with the parallel Ansible host-tunnel role — ADR-0015):
# * Gateway node label ...... homelab.keithjackson.dev/egress-gateway=true (applied by Ansible)
# * Host tunnel interface .... tun0 (split-tunnel OpenVPN)
# * Client selection ......... homelab.keithjackson.dev/egress-class=vpn (Kyverno-enforced)
# ---------------------------------------------------------------------------
apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
name: vpn-egress-steering
annotations:
homelab.keithjackson.dev/requirements: R-NET-04,R-NET-05,R-NET-06,R-NET-07
homelab.keithjackson.dev/adr: ADR-0015
spec:
# --- Client selection (INVARIANTS 1 & 2) --------------------------------
# ONLY pods explicitly opted into the vpn class are matched. Default/internet-class pods and node
# traffic are never selected, so they stay on the local/private-WAN path (invariant 1: default is
# the local path). Because ONLY these flows are steered, tun0 never becomes a default route for the
# node or for any other pod (invariant 2). The selector is intentionally cluster-wide (no namespace
# scope) so the class label alone is the contract — every namespace's vpn-class pods route here.
selectors:
- podSelector:
matchLabels:
homelab.keithjackson.dev/egress-class: vpn
# --- What gets tunneled (INVARIANT 1: only PUBLIC internet leaves via the VPN) ------------------
# destinationCIDRs is the full internet; excludedCIDRs carves out everything that must stay LOCAL:
# all RFC1918 private ranges (which also cover the Cilium cluster-pool pod CIDR 10.0.0.0/8, the k3s
# service/pod ranges, and the LAN 192.168.x/10.73.x node network). So intra-cluster east-west,
# pod->service, and LAN traffic are NOT SNAT'd out tun0 — only genuinely public destinations are.
# Excluded flows fall through to Cilium's normal local datapath (invariant 1).
destinationCIDRs:
- "0.0.0.0/0"
#
# BULK NFS (added to the record 2026-09-12, when plex/sonarr/radarr/nextcloud gained media
# mounts): the export is 192.168.2.102:/mnt/Storage/plex on prod (192.168.1.9 on dev), both
# inside 192.168.0.0/16 below, so NFS is excluded from the tunnel twice over. The FIRST reason
# is the one that actually matters and needs no CIDR at all: an NFS PersistentVolume is mounted
# by the NODE KERNEL in the host netns, not by the pod, so `selectors.podSelector` never
# matches it and no pod-scoped egress steering can touch it. Verified after arming the prod
# gateway: sonarr (k3s-prod-1) and radarr (k3s-prod-3, the gateway node itself) both still list
# AND write the export, and the node's own mounts are intact. No RFC1918 destination needs
# adding here for the media path.
excludedCIDRs:
- "10.0.0.0/8" # RFC1918 + Cilium cluster-pool pod CIDR (CLUSTER_POD_CIDR) + k3s service net
- "172.16.0.0/12" # RFC1918
- "192.168.0.0/16" # RFC1918 (home LAN / node network family / the bulk NFS server)
# --- Where it egresses --------------------------------------------------
# The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`.
#
# CORRECTION 2026-09-12 — this comment used to say the host nftables kill-switch "is the
# authoritative fail-closed backstop that DROPS this vpn-class traffic". It is not, and cannot
# be. With `bpf.masquerade: true` the SNAT and egress happen in eBPF, which bpf_redirect()s the
# skb to the egress device without invoking netfilter POSTROUTING, so the host nft rules never
# see this traffic. Measured with a counters-only nft table on the gateway node while a
# vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and
# the PIA range, versus 45166 packets on that same hook for WAN traffic (so the hook is live)
# and 16 for a node-originated curl out tun0 (so the matching works).
# WHAT ACTUALLY FAILS CLOSED: Cilium itself. When the cached egress interface/IP cannot be used
# it drops the packet — tcpdump on both the WAN and tun0 during a tunnel stop AND during a
# tunnel restart caught 0 packets leaving either device. vpn fails CLOSED (verified, no leak);
# internet-class and node traffic are never selected and fail OPEN (invariants 3 & 4 hold).
# The host nft table is retained, with counters, for the narrower case it CAN see:
# node-originated traffic sourced from a tunnel address (proven on k3s-prod-3 — the drop rule's
# counter moved 0 -> 3 packets when such a packet was routed at the WAN).
#
# SINGLE GATEWAY ONLY — there is NO failover, and a second labelled node is NOT a standby.
# `egressGateway` (singular) resolves the nodeSelector to ONE node. Verbatim from the CRD
# deployed on this cluster (`ciliumegressgatewaypolicies.cilium.io`, v2, Cilium 1.20.1 —
# `kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`):
#
# "EgressGateway is the gateway node responsible for SNATing traffic.
# In case multiple nodes are a match for the given set of labels, the first node
# in lexical ordering based on their name will be selected."
#
# So labelling a second node does not create an active/standby pair — it just hands the role to
# whichever node sorts FIRST BY NAME. A labelled `k3s-prod-1` would silently take the role from a
# labelled `k3s-prod-3`, moving the tunnel (and the public egress IP) with no event and no alert.
# LABEL EXACTLY ONE NODE. The plural `egressGateways` field (also present in 1.20) is NOT a fix:
# it SHARDS endpoints across entries by CiliumEndpoint UID, so losing one entry's node blackholes
# that shard and re-shards the rest (breaking live connections), and each shard exits from a
# DIFFERENT public IP. Upstream: cilium/cilium#30157, cilium/cilium#18230.
# ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)"
# is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see
# platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06.
#
# `interface: tun0` IS RESOLVED ONCE — the gateway node has to tell Cilium when tun0 changes.
# Measured on k3s-dev-3 2026-09-12: Cilium caches an (egress IP, egress ifindex) pair from the
# first resolution and does NOT re-resolve on a netlink change. `systemctl restart
# vpn-egress-tunnel` re-creates tun0 with a NEW ifindex; the map kept the old one and vpn-class
# egress was silently blackholed (tcpdump: 0 packets out the WAN, 0 out tun0 — fail-closed, but
# dead). `egress-gateway-reconciliation-trigger-interval: 1s` does not cover link events.
# Upstream: "it does not automatically respond to a change in the gateway node's network
# configuration... You can force a fresh selection by re-applying the Egress Gateway policy."
# The Ansible role closes this: roles/vpn_egress_gateway publishes
# `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node on every
# connect (plus a 2-minute convergence timer). That node update makes Cilium re-process this
# policy and re-resolve the interface — verified: cached ifindex 1986 -> 1987 within 10s.
# DO NOT "fix" this by having the node write to THIS object: it is Flux-managed, and a node
# writing to GitOps state is a drift fight. Check the label, not this file, when debugging:
# kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink
egressGateway:
nodeSelector:
matchLabels:
homelab.keithjackson.dev/egress-gateway: "true"
interface: tun0
# ---------------------------------------------------------------------------
# DONE (2026-08-24): enabled + validated + Gluetun retired.
# * Host tun0 split-tunnel is up on the egress-gateway node; host nftables kill-switch in place.
# * This file is wired into platform/vpn-egress/kustomization.yaml and reconciled by Flux.
# * `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) conformance tests PASS live.
# * The Gluetun Deployment was retired per ADR-0015 ("Gluetun is dropped as the gateway").
# ---------------------------------------------------------------------------
Reference in new issue View Git Blame Copy Permalink