jakman85andClaude Opus 5.5
8f60e6103a
new-peer.sh generated MTU 1420, which stalled traffic on mobile data despite a healthy handshake (one phone, one carrier, 2026-11-09). Default to 1280, overridable through WG_MTU. Correct stale claims in the runbook and the DaemonSet header against live BPF conntrack: LoadBalancer VIPs see 10.90.0.x as world, not a SNATed host identity. Add a troubleshooting section. The DaemonSet render is unchanged, so WireGuard pods do not roll. Co-Authored-By: Claude Opus 5.5 <[email protected]>
267 lines
19 KiB
Markdown
267 lines
19 KiB
Markdown
# platform/vpn-ingress — WireGuard remote access (north-south INGRESS)
|
||
|
||
Lets the operator's own devices — Android, Linux, Windows, macOS, iOS — dial in from anywhere on
|
||
the internet and behave as though plugged into the home LAN: full reach to `192.168.2.0/24` and
|
||
egress out the house WAN.
|
||
|
||
> **This is not `platform/vpn-egress`.** They are opposite directions and share nothing but the
|
||
> word "VPN". `vpn-egress` pushes *pods out* to the internet through a PIA tunnel (ADR-0015,
|
||
> R-NET-05/06/07). This component lets *remote clients in*. Changing one never affects the other.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Transport | WireGuard, **udp/51820**, kernel implementation |
|
||
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
|
||
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
|
||
| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
|
||
| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
|
||
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
|
||
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
|
||
| Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
|
||
|
||
## Why WireGuard
|
||
|
||
The requirement names four client platforms. WireGuard is the only option that has a *first-party*
|
||
client on all of them (official apps for Android, iOS, Windows and macOS; in-kernel on Linux since
|
||
5.6, so a Linux client needs no userspace daemon at all). It is also the smallest thing to expose:
|
||
one UDP port, no cipher negotiation, no PKI, no user database, no TLS stack.
|
||
|
||
OpenVPN would have needed a CA, per-device certificates and a revocation list — three pieces of
|
||
long-lived state this repo would then have to protect and rotate. IKEv2/IPsec has native clients but
|
||
is painful to run in a container and brittle behind double NAT. Neither buys anything here.
|
||
|
||
### Why this is safe to expose to the whole internet
|
||
|
||
The honest answer is not "because it is encrypted" — it is the handshake's failure mode:
|
||
|
||
**An unauthenticated packet gets no reply at all.** WireGuard's first handshake message carries a
|
||
MAC keyed with the *responder's own public key*. If the sender does not already know it, the packet
|
||
is dropped silently — no error, no reset, no banner, nothing a scanner can fingerprint. From the
|
||
outside, udp/51820 is indistinguishable from a closed port. There is no login prompt to brute-force,
|
||
no version string to match against a CVE, and no amplification to abuse.
|
||
|
||
That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
|
||
would also have made roaming clients unusable — a phone's address changes with every cell tower).
|
||
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN and
|
||
this tunnel's client subnet: a resolver answers anybody, so it must be scoped by address. WireGuard
|
||
answers nobody.
|
||
|
||
What it does **not** protect against is a stolen device — anyone holding a device's private key
|
||
*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).
|
||
|
||
## Architecture, and the one decision that drives everything
|
||
|
||
```
|
||
phone on 5G SOHO router k3s-prod-N (host netns)
|
||
┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐
|
||
│ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │
|
||
│ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │
|
||
└──────────────┘ internet └───────────────┘ │ SNAT (LAN), eBPF LB (VIPs)│
|
||
└─────────────┬──────────────┘
|
||
│ translated per destination (below)
|
||
┌─────────────────────────────────┼──────────────────────┐
|
||
▼ ▼ ▼
|
||
LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet
|
||
(NAS, router, printers) PiHole DNS VIP .23
|
||
```
|
||
|
||
### `hostNetwork: true` is required, not convenient
|
||
|
||
This is the only genuinely interesting decision in the component, and it was forced by a *measured*
|
||
property of this cluster rather than chosen for convenience.
|
||
|
||
In Cilium, what a packet may reach is decided by the **security identity of its source**. A
|
||
WireGuard server running as an ordinary pod must SNAT client traffic to its own pod IP (Cilium drops
|
||
a packet whose source is not the endpoint's own address), so every remote client would have arrived
|
||
everywhere wearing *that pod's* identity. On 2026-10-04 this cluster measured what that costs
|
||
(`decisions/2026-10-04-plex-advertise-plexdirect.md`): a request to the shared Gateway VIP **from
|
||
inside a pod returns `403 Access denied` from Envoy for all 16 Gateway hostnames**, because Cilium's
|
||
Gateway preserves the original source identity and enforces the *destination's* ingress policy —
|
||
and every app admits `reserved:ingress` plus named peers, never an arbitrary pod. LAN browsers work
|
||
only because they arrive as `reserved:world` / `reserved:host`.
|
||
|
||
So the pod-networked version of this component would have produced a tunnel that came up healthy,
|
||
reported connected peers, routed packets — and 403'd **every single cluster web UI**: plex, gitea,
|
||
keycloak, nextcloud, grafana, headlamp, the apex. `*.home.keithjackson.dev` would not even have
|
||
resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately omits the `cluster`
|
||
entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
|
||
manifests would have hinted at it.
|
||
|
||
In the **host** netns there is no pod identity to carry, which is what avoids the 403s. Clients do
|
||
**not**, however, all arrive SNATed to the node. Measured 2026-11-08: Cilium counts `wg0` among its
|
||
kube-proxy-replacement devices (`cilium-dbg status`: `eno1`, `wg0`, `tun0`), so what a packet looks
|
||
like depends on where it is going:
|
||
|
||
| destination | handled by | source the destination sees |
|
||
|---|---|---|
|
||
| LAN hosts, the internet | kernel FORWARD, then the iptables MASQUERADE in [`daemonset.yaml`](./daemonset.yaml) | the node's address |
|
||
| LoadBalancer VIPs (Gateway `.20`, PiHole `.23`, ...) | Cilium's eBPF load balancer, as the packet enters from `wg0`, **before** MASQUERADE | `10.90.0.x` at the VIP's source-range check, identity `world` |
|
||
|
||
The second row is why PiHole's `loadBalancerSourceRanges` has to list `${WG_CLIENT_CIDR}` as well as
|
||
the LAN: left LAN-only, tunnel DNS is predicted to die with `DROP_NOT_IN_SRC_RANGE`. That prediction
|
||
and the row's source column come from reading Cilium's datapath source; **no peer has ever completed
|
||
a handshake, so neither has been observed live.** The Gateway VIP carries no source ranges and needs
|
||
no change. Three further problems vanish with the host netns:
|
||
|
||
* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
|
||
`/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
|
||
pod-netns build would have needed `--allowed-unsafe-sysctls` on all three kubelets — a
|
||
node-restart change to the k3s service, to run a VPN.
|
||
* **No LoadBalancer VIP**, so no `CiliumL2AnnouncementPolicy` lease lottery (the backend-blind
|
||
announcer that cost this LAN an 8-day Minecraft outage — see
|
||
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
|
||
`externalTrafficPolicy` trade-off to make at all.
|
||
* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
|
||
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` correct on the
|
||
server's `wg0`. It says nothing about a client's own path: `new-peer.sh` defaults client profiles
|
||
to `1280` (see "Troubleshooting" in `runbooks/vpn-remote-access.md`).
|
||
|
||
`hostNetwork` is *permitted* here, and not by oversight:
|
||
`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
|
||
app-namespace=true` only, and names this exact carve-out in its own comment — "platform components
|
||
that do … e.g. Cilium, node-exporter … live in platform namespaces that do NOT carry the
|
||
app-namespace label". [`namespace.yaml`](./namespace.yaml) is deliberately unlabelled.
|
||
|
||
**The cost, stated rather than buried:** a `hostNetwork` pod has no Cilium endpoint, so **no
|
||
NetworkPolicy constrains this workload** — not [`networkpolicy.yaml`](./networkpolicy.yaml), not
|
||
anything. That file says so at length instead of implying a protection it cannot provide. The
|
||
boundary around this port is WireGuard's peer authentication, full stop.
|
||
|
||
### Why a DaemonSet
|
||
|
||
The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
|
||
forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
|
||
A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
|
||
keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
|
||
re-issued client profiles. The two idle listeners cost ~15 MiB each.
|
||
|
||
A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
|
||
replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
|
||
hop into the one path whose failure mode is *"I am away from home and cannot get in"*. Fewer
|
||
translations win here.
|
||
|
||
### The split that keeps authorisation reviewable
|
||
|
||
| half | where | why |
|
||
|---|---|---|
|
||
| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
|
||
| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
|
||
|
||
The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
|
||
so the assembled config holding the private key never touches any node's disk and dies with the pod.
|
||
|
||
Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with
|
||
everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in,
|
||
corrected after review — would have put the one credential every enrolled device depends on *outside*
|
||
the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this
|
||
key's public half.
|
||
|
||
Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
|
||
new line granting a new device access to the whole LAN.
|
||
|
||
## What a client can reach — and how to narrow it
|
||
|
||
By design, **everything a LAN machine can reach**: all of `192.168.2.0/24`, the shared Gateway's 16
|
||
hostnames, PiHole, the NAS, the router's admin UI, and the internet via the house WAN.
|
||
|
||
**That includes the three k3s nodes themselves** — `192.168.2.10/.11/.12`, and therefore the
|
||
kube-apiserver on 6443, kubelet on 10250 and SSH on 22. An enrolled device is as privileged as a
|
||
machine sitting in the house, which is what "reach the entire local network" asks for; it is
|
||
recorded here because it is the largest consequence of saying yes to that, and because nobody should
|
||
discover it later. **It is also why a lost device must be revoked promptly** (delete its stanza,
|
||
merge; ~1 min to take effect).
|
||
|
||
To grant less, narrow it at the **client** profile first — `new-peer.sh --split` emits a
|
||
split-tunnel profile routing only the LAN and the tunnel subnet, so the device keeps its own
|
||
internet path. To enforce less *server-side*, add filtering to the `PostUp` chain in
|
||
[`daemonset.yaml`](./daemonset.yaml) (e.g. `iptables -A FORWARD -i wg0 -d 192.168.2.10 -j DROP`
|
||
before the accept rules) — the FORWARD chain there is the only enforcement point, since no
|
||
NetworkPolicy applies.
|
||
|
||
## Phone profiles: one tunnel for home, one for away
|
||
|
||
A phone that needs the LAN-only apps runs full tunnel (`AllowedIPs = 0.0.0.0/0`, `DNS = 192.168.2.23`)
|
||
everywhere, so it needs **two tunnels**, each its own peer with its own key and address:
|
||
|
||
| tunnel | peer | address | Endpoint | use when |
|
||
|---|---|---|---|---|
|
||
| `away` | `keith-phone` | `10.90.0.2` | `wan.home.keithjackson.dev:51820` | off the home network |
|
||
| `home` | `keith-phone-home` | `10.90.0.3` | `192.168.2.10:51820` | on the home Wi-Fi |
|
||
|
||
**`wan.` cannot work from inside the house.** It is an unproxied A record holding the house's WAN
|
||
address (`platform/externaldns/README.md`), and the router has no hairpin NAT, so a LAN client that
|
||
sends to it never comes back in through the port-forward. The Cloudflare Tunnel is no way round: it
|
||
carries HTTP(S) to the Gateway and WireGuard is UDP to a node, so `wan.` has to stay DNS-only and
|
||
`away` has to keep using it.
|
||
|
||
**`home` depends on `k3s-prod-1`.** Its Endpoint is that node's address, with no failover: if
|
||
prod-1 is down, `home` cannot handshake even though the other two listeners are healthy. The fix
|
||
is client-side: edit the tunnel's Endpoint to `192.168.2.11` or `.12`, since every node answers the
|
||
same keys. No server change and no new key is needed. The same dependency applies to `away` if the
|
||
WAN forward points at prod-1.
|
||
|
||
Android allows one active VPN at a time, so switch tunnels by hand when leaving or arriving.
|
||
|
||
## Operating it
|
||
|
||
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
|
||
stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
|
||
* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
|
||
stops working as soon as the pods roll (the generator's content hash guarantees they do).
|
||
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
|
||
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
|
||
(`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
|
||
* **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
|
||
|
||
## Measured on prod, 2026-10-06 — what is now known rather than assumed
|
||
|
||
The first real reconcile settled four things this component's design rested on. Three went the
|
||
right way; one was a defect in my own manifest.
|
||
|
||
| question | answer |
|
||
|---|---|
|
||
| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
|
||
| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
|
||
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
|
||
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
|
||
|
||
## Prerequisites outside this directory
|
||
|
||
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
|
||
(`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
|
||
`NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
|
||
CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
|
||
2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a
|
||
`REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring
|
||
is complete and the only outstanding act is replacing the value (runbook step 2). Until then the
|
||
DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the
|
||
value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by
|
||
mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is
|
||
the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves
|
||
every Flux condition green.
|
||
3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
|
||
and the half CI can never check.
|
||
|
||
## Not built (deliberate, with reasons)
|
||
|
||
* **No metrics / no peer-level alerting.** A WireGuard exporter would expose
|
||
`wireguard_latest_handshake_seconds` per peer — genuinely useful for spotting unexpected
|
||
activity — but it needs a second `NET_ADMIN` container in the pod that terminates the
|
||
internet-facing tunnel, and that trade was not worth making unverified. Pod-down is already
|
||
covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
|
||
kube-prometheus-stack. **This is the top follow-up.**
|
||
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
|
||
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
|
||
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
|
||
`wg-server-key` Secret.
|
||
* **No IPv6.** The server offers no IPv6 tunnel address and `new-peer.sh` emits only `0.0.0.0/0`.
|
||
What happens to a client's IPv6 then depends on the client. The **Android app does not leak it**:
|
||
its `GoBackend` skips `allowFamily()` when the config is one peer with a default route, so Android
|
||
blocks IPv6 for the VPN's duration and apps fall back to IPv4. Other clients are not covered by
|
||
that code; expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside the tunnel. Fix is
|
||
an `fd..` ULA on `wg0` plus `::/0` in the client's `AllowedIPs`; not done because nothing here
|
||
needs it.
|
||
* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
|
||
browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
|
||
into the tunnel.
|