Files
home-network-k3s-impl/platform/vpn-ingress/README.md
T
jakman85andClaude Opus 5 b258e47aa6
gate / gate (pull_request) Successful in 1m55s
gate / conformance (pull_request) Has been skipped
docs(vpn-ingress): withdraw the L2-lease diagnosis — it was falsified by measurement
The claim this branch originally shipped (a node cannot reach a LoadBalancer VIP
whose Cilium L2 announcement lease it holds, so VPN clients inherit that via
hostNetwork MASQUERADE) is FALSE. Re-measured with all 8 l2announce leases on
k3s-prod-1:

  ip neigh show 192.168.2.20   -> INCOMPLETE   (cause present)
  nc -z 192.168.2.20 443       -> OK
  curl --resolve ... :192.168.2.20  -> 200 in 25ms   (effect absent)

Cause present, effect absent. prod-2 and prod-3 return the same 200.

Why it was never going to bite: cilium-dbg status shows
`KubeProxyReplacement: True [eno1 ..., wg0 10.90.0.1]` and
`Masquerading: BPF [eno1, wg0]` -- Cilium's LB datapath is attached to wg0, so a
client packet for a VIP is DNAT'd in BPF BEFORE any neighbour lookup, and the
Gateway backend is a node-local Envoy socket anyway.

The methodological error, which is the part worth keeping: every probe was run
from the wireguard pod on the grounds that sharing the host netns made it "the
path a MASQUERADEd client takes". False. A MASQUERADEd client is FORWARDED
(netfilter FORWARD + routing); a probe started in the pod is SOCKET-ORIGINATED
(Cilium socket-LB). Different hooks -- sharing a namespace is not sharing a hook.
The `curl --interface wg0` control cited to rule out a host-netns artefact fails
on ALL THREE nodes including ones whose ARP resolves, so it discriminated
nothing. A raw-IP `curl -k https://192.168.2.20/` returning rc=35 is missing SNI,
not unreachability.

Also corrected: new-peer.sh emits AllowedIPs = 0.0.0.0/0 (FULL tunnel) by
default; `--split` is the opt-in. An earlier reading had the branches backwards.

All code is withdrawn -- the nodeSelector exclusion, the node_baseline label
tasks, both alerts, the promtool tests and the KSM allowlist entry are gone.
Net change is three documentation files, purely additive (356 insertions, 0
deletions), recording what was measured so the next person does not redo it.

The reported internet-routing problem is NOT diagnosed here: no client had
connected since the power-cycle (no peer endpoint, 0 packets through FORWARD and
MASQUERADE), so the forwarded path was unmeasurable.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 03:14:28 -05:00

305 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# platform/vpn-ingress — WireGuard remote access (north-south INGRESS)
Lets the operator's own devices — Android, Linux, Windows, macOS, iOS — dial in from anywhere on
the internet and behave as though plugged into the home LAN: full reach to `192.168.2.0/24` and
egress out the house WAN.
> **This is not `platform/vpn-egress`.** They are opposite directions and share nothing but the
> word "VPN". `vpn-egress` pushes *pods out* to the internet through a PIA tunnel (ADR-0015,
> R-NET-05/06/07). This component lets *remote clients in*. Changing one never affects the other.
| | |
|---|---|
| Transport | WireGuard, **udp/51820**, kernel implementation |
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
| Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
## Why WireGuard
The requirement names four client platforms. WireGuard is the only option that has a *first-party*
client on all of them (official apps for Android, iOS, Windows and macOS; in-kernel on Linux since
5.6, so a Linux client needs no userspace daemon at all). It is also the smallest thing to expose:
one UDP port, no cipher negotiation, no PKI, no user database, no TLS stack.
OpenVPN would have needed a CA, per-device certificates and a revocation list — three pieces of
long-lived state this repo would then have to protect and rotate. IKEv2/IPsec has native clients but
is painful to run in a container and brittle behind double NAT. Neither buys anything here.
### Why this is safe to expose to the whole internet
The honest answer is not "because it is encrypted" — it is the handshake's failure mode:
**An unauthenticated packet gets no reply at all.** WireGuard's first handshake message carries a
MAC keyed with the *responder's own public key*. If the sender does not already know it, the packet
is dropped silently — no error, no reset, no banner, nothing a scanner can fingerprint. From the
outside, udp/51820 is indistinguishable from a closed port. There is no login prompt to brute-force,
no version string to match against a CVE, and no amplification to abuse.
That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
would also have made roaming clients unusable — a phone's address changes with every cell tower).
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a
resolver answers anybody, so it must be scoped by address. WireGuard answers nobody.
What it does **not** protect against is a stolen device — anyone holding a device's private key
*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).
## Architecture, and the one decision that drives everything
```
phone on 5G SOHO router k3s-prod-N (host netns)
┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐
│ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │
│ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │
└──────────────┘ internet └───────────────┘ │ SNAT to the node's addr │
└─────────────┬──────────────┘
│ as `host`/`remote-node`
┌─────────────────────────────────┼──────────────────────┐
▼ ▼ ▼
LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet
(NAS, router, printers) PiHole DNS VIP .23
```
### `hostNetwork: true` is required, not convenient
This is the only genuinely interesting decision in the component, and it was forced by a *measured*
property of this cluster rather than chosen for convenience.
In Cilium, what a packet may reach is decided by the **security identity of its source**. A
WireGuard server running as an ordinary pod must SNAT client traffic to its own pod IP (Cilium drops
a packet whose source is not the endpoint's own address), so every remote client would have arrived
everywhere wearing *that pod's* identity. On 2026-10-04 this cluster measured what that costs
(`decisions/2026-10-04-plex-advertise-plexdirect.md`): a request to the shared Gateway VIP **from
inside a pod returns `403 Access denied` from Envoy for all 16 Gateway hostnames**, because Cilium's
Gateway preserves the original source identity and enforces the *destination's* ingress policy —
and every app admits `reserved:ingress` plus named peers, never an arbitrary pod. LAN browsers work
only because they arrive as `reserved:world` / `reserved:host`.
So the pod-networked version of this component would have produced a tunnel that came up healthy,
reported connected peers, routed packets — and 403'd **every single cluster web UI**: plex, gitea,
keycloak, nextcloud, grafana, headlamp, the apex. `*.home.keithjackson.dev` would not even have
resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately omits the `cluster`
entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
manifests would have hinted at it.
In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the
node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
byte-for-byte what a LAN client looks like. Three further problems vanish with it:
* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
`/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
pod-netns build would have needed `--allowed-unsafe-sysctls` on all three kubelets — a
node-restart change to the k3s service, to run a VPN.
* **No LoadBalancer VIP**, so no `CiliumL2AnnouncementPolicy` lease lottery (the backend-blind
announcer that cost this LAN an 8-day Minecraft outage — see
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
`externalTrafficPolicy` trade-off to make at all.
* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct.
`hostNetwork` is *permitted* here, and not by oversight:
`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
app-namespace=true` only, and names this exact carve-out in its own comment — "platform components
that do … e.g. Cilium, node-exporter … live in platform namespaces that do NOT carry the
app-namespace label". [`namespace.yaml`](./namespace.yaml) is deliberately unlabelled.
**The cost, stated rather than buried:** a `hostNetwork` pod has no Cilium endpoint, so **no
NetworkPolicy constrains this workload** — not [`networkpolicy.yaml`](./networkpolicy.yaml), not
anything. That file says so at length instead of implying a protection it cannot provide. The
boundary around this port is WireGuard's peer authentication, full stop.
### Why a DaemonSet
The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
re-issued client profiles. The two idle listeners cost ~15 MiB each.
A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
hop into the one path whose failure mode is *"I am away from home and cannot get in"*. Fewer
translations win here.
### The split that keeps authorisation reviewable
| half | where | why |
|---|---|---|
| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
so the assembled config holding the private key never touches any node's disk and dies with the pod.
Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with
everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in,
corrected after review — would have put the one credential every enrolled device depends on *outside*
the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this
key's public half.
Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
new line granting a new device access to the whole LAN.
## What a client can reach — and how to narrow it
By design, **everything a LAN machine can reach**: all of `192.168.2.0/24`, the shared Gateway's 16
hostnames, PiHole, the NAS, the router's admin UI, and the internet via the house WAN.
**That includes the three k3s nodes themselves** — `192.168.2.10/.11/.12`, and therefore the
kube-apiserver on 6443, kubelet on 10250 and SSH on 22. An enrolled device is as privileged as a
machine sitting in the house, which is what "reach the entire local network" asks for; it is
recorded here because it is the largest consequence of saying yes to that, and because nobody should
discover it later. **It is also why a lost device must be revoked promptly** (delete its stanza,
merge; ~1 min to take effect).
To grant less, narrow it at the **client** profile first — `new-peer.sh --split` emits a
split-tunnel profile routing only the LAN and the tunnel subnet, so the device keeps its own
internet path. To enforce less *server-side*, add filtering to the `PostUp` chain in
[`daemonset.yaml`](./daemonset.yaml) (e.g. `iptables -A FORWARD -i wg0 -d 192.168.2.10 -j DROP`
before the accept rules) — the FORWARD chain there is the only enforcement point, since no
NetworkPolicy applies.
## Operating it
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
stops working as soon as the pods roll (the generator's content hash guarantees they do).
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
(`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
* **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
## Measured on prod, 2026-10-06 — what is now known rather than assumed
The first real reconcile settled four things this component's design rested on. Three went the
right way; one was a defect in my own manifest.
| question | answer |
|---|---|
| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach
A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose
Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from
announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is
real but inert, and the distinction matters enough to record, because the wrong version is very
easy to re-derive from `ip neigh` alone.
Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case,
after the UPS power-cycle — from the host netns of the `wireguard` pod on each node:
| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
|---|---|---|
| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms |
| `nc -z 192.168.2.20 443` | **OK** | OK |
| plex `192.168.2.24:32400` | **`200`** | `200` |
So the announcer really does **not** answer its own ARP request — that part of the observation is
reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**:
* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
`Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**,
so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before
any route or neighbour lookup for the VIP happens**.
* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the
shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without
a packet crossing the LAN at all.
* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real**
service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes —
that is service-aware DNAT, not wire traffic.
**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis:
* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an
artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not
anything is wrong.
* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the
two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it
discriminates nothing — and it is not the client path either (see below).
**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are
**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is
**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the
forwarded path in either direction: it reports success where the forwarded path might fail, and
`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with
a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE`
counters move.
## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses
`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the
component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature
to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel
client cannot reach the internet, nothing in the list below is the reason:
| checked | result on all three nodes |
|---|---|
| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` |
| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change |
| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE |
| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present |
| default route | `default via 192.168.2.1 dev eno1 onlink`, intact |
| node's own internet reachability | `http://1.1.1.1/` returns `301` |
**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is
closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by
`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled
`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity
traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway
node. A remote client's apparent public address is therefore the **home WAN IP**, which is the
intended behaviour and the thing to state when enrolling a device.
**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one
enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and
the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the
power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the
forwarded node shows a peer with an `endpoint` and a recent `latest handshake`.
## Prerequisites outside this directory
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
(`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
`NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a
`REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring
is complete and the only outstanding act is replacing the value (runbook step 2). Until then the
DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the
value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by
mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is
the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves
every Flux condition green.
3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
and the half CI can never check.
## Not built (deliberate, with reasons)
* **No metrics / no peer-level alerting.** A WireGuard exporter would expose
`wireguard_latest_handshake_seconds` per peer — genuinely useful for spotting unexpected
activity — but it needs a second `NET_ADMIN` container in the pod that terminates the
internet-facing tunnel, and that trade was not worth making unverified. Pod-down is already
covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
kube-prometheus-stack. **This is the top follow-up.**
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
`wg-server-key` Secret.
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point
was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's
`AllowedIPs`; not done because nothing here needs it.
* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
into the tunnel.