jakman85andClaude Opus 5
5c43fbac65
The tunnel never came up on its first reconcile with a real key. One line of my
own manifest was the whole cause. Measured on prod 2026-10-06:
[#] ip link add dev wg0 type wireguard <- fine
[#] wg addconf wg0 /dev/fd/63 <- fine
[#] ip -4 address add 10.90.0.1/24 dev wg0 <- fine
[#] ip link set mtu 1420 up dev wg0 <- fine
[#] sysctl -q -w net.ipv4.ip_forward=1
sysctl: permission denied on key "net.ipv4.ip_forward"
[#] ip link delete dev wg0
**** Tunnel failed, will stop all others! ****
Not a missing capability: with hostNetwork the runtime mounts /proc/sys
READ-ONLY (confirmed in the pod's /proc/mounts), because those sysctls are the
HOST's globals. CAP_NET_ADMIN cannot lift a read-only mount, so the write can
never succeed here -- and wg-quick treats a failed PostUp as fatal, so it
deleted the interface it had just built.
The line was never needed. node_baseline already sets ip_forward=1, and reading
it from the pod returns 1. My header comment said exactly that ("so no pod
sysctl is needed") and then emitted the sysctl anyway, justified as "re-asserted
so this component does not silently depend on another role's sysctl file". That
defensive re-assertion is the entire outage. In a hostNetwork pod: VERIFY host
state, never write it.
Replaced with a read in the init container -- `grep -qx 1
/proc/sys/net/ipv4/ip_forward` -- failing closed with a message naming the
node_baseline role. Still fail-closed on purpose: a tunnel that comes up unable
to forward satisfies every probe here and silently drops all client traffic.
Also documents, in the manifest and the runbook, that the image's `chown:
Operation not permitted` run and its "we will not provide support for it"
warning are expected and harmless -- capabilities are dropped to NET_ADMIN so
its s6 init has no CAP_CHOWN, /config is a writable tmpfs, everything runs as
root, and wg-quick proceeds (it did; the failure was 30 lines later).
Three other unknowns closed by the same reconcile, now recorded in the README:
the kernel module reaches the pod (ip link add type wireguard works, no
CAP_SYS_MODULE needed); allowPrivilegeEscalation:false does NOT break the s6
init; and the image's iptables is nf_tables (xtables-nft-multi) matching Debian
13, which closes the worst documented fear -- rules landing in a table the
kernel never consults.
No sensitive paths; merges on a green gate with no override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
224 lines
16 KiB
Markdown
224 lines
16 KiB
Markdown
# platform/vpn-ingress — WireGuard remote access (north-south INGRESS)
|
||
|
||
Lets the operator's own devices — Android, Linux, Windows, macOS, iOS — dial in from anywhere on
|
||
the internet and behave as though plugged into the home LAN: full reach to `192.168.2.0/24` and
|
||
egress out the house WAN.
|
||
|
||
> **This is not `platform/vpn-egress`.** They are opposite directions and share nothing but the
|
||
> word "VPN". `vpn-egress` pushes *pods out* to the internet through a PIA tunnel (ADR-0015,
|
||
> R-NET-05/06/07). This component lets *remote clients in*. Changing one never affects the other.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Transport | WireGuard, **udp/51820**, kernel implementation |
|
||
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
|
||
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
|
||
| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
|
||
| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
|
||
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
|
||
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
|
||
| Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
|
||
|
||
## Why WireGuard
|
||
|
||
The requirement names four client platforms. WireGuard is the only option that has a *first-party*
|
||
client on all of them (official apps for Android, iOS, Windows and macOS; in-kernel on Linux since
|
||
5.6, so a Linux client needs no userspace daemon at all). It is also the smallest thing to expose:
|
||
one UDP port, no cipher negotiation, no PKI, no user database, no TLS stack.
|
||
|
||
OpenVPN would have needed a CA, per-device certificates and a revocation list — three pieces of
|
||
long-lived state this repo would then have to protect and rotate. IKEv2/IPsec has native clients but
|
||
is painful to run in a container and brittle behind double NAT. Neither buys anything here.
|
||
|
||
### Why this is safe to expose to the whole internet
|
||
|
||
The honest answer is not "because it is encrypted" — it is the handshake's failure mode:
|
||
|
||
**An unauthenticated packet gets no reply at all.** WireGuard's first handshake message carries a
|
||
MAC keyed with the *responder's own public key*. If the sender does not already know it, the packet
|
||
is dropped silently — no error, no reset, no banner, nothing a scanner can fingerprint. From the
|
||
outside, udp/51820 is indistinguishable from a closed port. There is no login prompt to brute-force,
|
||
no version string to match against a CVE, and no amplification to abuse.
|
||
|
||
That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
|
||
would also have made roaming clients unusable — a phone's address changes with every cell tower).
|
||
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a
|
||
resolver answers anybody, so it must be scoped by address. WireGuard answers nobody.
|
||
|
||
What it does **not** protect against is a stolen device — anyone holding a device's private key
|
||
*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).
|
||
|
||
## Architecture, and the one decision that drives everything
|
||
|
||
```
|
||
phone on 5G SOHO router k3s-prod-N (host netns)
|
||
┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐
|
||
│ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │
|
||
│ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │
|
||
└──────────────┘ internet └───────────────┘ │ SNAT to the node's addr │
|
||
└─────────────┬──────────────┘
|
||
│ as `host`/`remote-node`
|
||
┌─────────────────────────────────┼──────────────────────┐
|
||
▼ ▼ ▼
|
||
LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet
|
||
(NAS, router, printers) PiHole DNS VIP .23
|
||
```
|
||
|
||
### `hostNetwork: true` is required, not convenient
|
||
|
||
This is the only genuinely interesting decision in the component, and it was forced by a *measured*
|
||
property of this cluster rather than chosen for convenience.
|
||
|
||
In Cilium, what a packet may reach is decided by the **security identity of its source**. A
|
||
WireGuard server running as an ordinary pod must SNAT client traffic to its own pod IP (Cilium drops
|
||
a packet whose source is not the endpoint's own address), so every remote client would have arrived
|
||
everywhere wearing *that pod's* identity. On 2026-10-04 this cluster measured what that costs
|
||
(`decisions/2026-10-04-plex-advertise-plexdirect.md`): a request to the shared Gateway VIP **from
|
||
inside a pod returns `403 Access denied` from Envoy for all 16 Gateway hostnames**, because Cilium's
|
||
Gateway preserves the original source identity and enforces the *destination's* ingress policy —
|
||
and every app admits `reserved:ingress` plus named peers, never an arbitrary pod. LAN browsers work
|
||
only because they arrive as `reserved:world` / `reserved:host`.
|
||
|
||
So the pod-networked version of this component would have produced a tunnel that came up healthy,
|
||
reported connected peers, routed packets — and 403'd **every single cluster web UI**: plex, gitea,
|
||
keycloak, nextcloud, grafana, headlamp, the apex. `*.home.keithjackson.dev` would not even have
|
||
resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately omits the `cluster`
|
||
entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
|
||
manifests would have hinted at it.
|
||
|
||
In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the
|
||
node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
|
||
byte-for-byte what a LAN client looks like. Three further problems vanish with it:
|
||
|
||
* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
|
||
`/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
|
||
pod-netns build would have needed `--allowed-unsafe-sysctls` on all three kubelets — a
|
||
node-restart change to the k3s service, to run a VPN.
|
||
* **No LoadBalancer VIP**, so no `CiliumL2AnnouncementPolicy` lease lottery (the backend-blind
|
||
announcer that cost this LAN an 8-day Minecraft outage — see
|
||
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
|
||
`externalTrafficPolicy` trade-off to make at all.
|
||
* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
|
||
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct.
|
||
|
||
`hostNetwork` is *permitted* here, and not by oversight:
|
||
`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
|
||
app-namespace=true` only, and names this exact carve-out in its own comment — "platform components
|
||
that do … e.g. Cilium, node-exporter … live in platform namespaces that do NOT carry the
|
||
app-namespace label". [`namespace.yaml`](./namespace.yaml) is deliberately unlabelled.
|
||
|
||
**The cost, stated rather than buried:** a `hostNetwork` pod has no Cilium endpoint, so **no
|
||
NetworkPolicy constrains this workload** — not [`networkpolicy.yaml`](./networkpolicy.yaml), not
|
||
anything. That file says so at length instead of implying a protection it cannot provide. The
|
||
boundary around this port is WireGuard's peer authentication, full stop.
|
||
|
||
### Why a DaemonSet
|
||
|
||
The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
|
||
forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
|
||
A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
|
||
keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
|
||
re-issued client profiles. The two idle listeners cost ~15 MiB each.
|
||
|
||
A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
|
||
replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
|
||
hop into the one path whose failure mode is *"I am away from home and cannot get in"*. Fewer
|
||
translations win here.
|
||
|
||
### The split that keeps authorisation reviewable
|
||
|
||
| half | where | why |
|
||
|---|---|---|
|
||
| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
|
||
| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
|
||
|
||
The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
|
||
so the assembled config holding the private key never touches any node's disk and dies with the pod.
|
||
|
||
Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with
|
||
everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in,
|
||
corrected after review — would have put the one credential every enrolled device depends on *outside*
|
||
the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this
|
||
key's public half.
|
||
|
||
Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
|
||
new line granting a new device access to the whole LAN.
|
||
|
||
## What a client can reach — and how to narrow it
|
||
|
||
By design, **everything a LAN machine can reach**: all of `192.168.2.0/24`, the shared Gateway's 16
|
||
hostnames, PiHole, the NAS, the router's admin UI, and the internet via the house WAN.
|
||
|
||
**That includes the three k3s nodes themselves** — `192.168.2.10/.11/.12`, and therefore the
|
||
kube-apiserver on 6443, kubelet on 10250 and SSH on 22. An enrolled device is as privileged as a
|
||
machine sitting in the house, which is what "reach the entire local network" asks for; it is
|
||
recorded here because it is the largest consequence of saying yes to that, and because nobody should
|
||
discover it later. **It is also why a lost device must be revoked promptly** (delete its stanza,
|
||
merge; ~1 min to take effect).
|
||
|
||
To grant less, narrow it at the **client** profile first — `new-peer.sh --split` emits a
|
||
split-tunnel profile routing only the LAN and the tunnel subnet, so the device keeps its own
|
||
internet path. To enforce less *server-side*, add filtering to the `PostUp` chain in
|
||
[`daemonset.yaml`](./daemonset.yaml) (e.g. `iptables -A FORWARD -i wg0 -d 192.168.2.10 -j DROP`
|
||
before the accept rules) — the FORWARD chain there is the only enforcement point, since no
|
||
NetworkPolicy applies.
|
||
|
||
## Operating it
|
||
|
||
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
|
||
stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
|
||
* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
|
||
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
|
||
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
|
||
(`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
|
||
* **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
|
||
|
||
## Measured on prod, 2026-10-06 — what is now known rather than assumed
|
||
|
||
The first real reconcile settled four things this component's design rested on. Three went the
|
||
right way; one was a defect in my own manifest.
|
||
|
||
| question | answer |
|
||
|---|---|
|
||
| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
|
||
| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
|
||
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
|
||
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
|
||
|
||
## Prerequisites outside this directory
|
||
|
||
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
|
||
(`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
|
||
`NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
|
||
CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
|
||
2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a
|
||
`REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring
|
||
is complete and the only outstanding act is replacing the value (runbook step 2). Until then the
|
||
DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the
|
||
value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by
|
||
mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is
|
||
the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves
|
||
every Flux condition green.
|
||
3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
|
||
and the half CI can never check.
|
||
|
||
## Not built (deliberate, with reasons)
|
||
|
||
* **No metrics / no peer-level alerting.** A WireGuard exporter would expose
|
||
`wireguard_latest_handshake_seconds` per peer — genuinely useful for spotting unexpected
|
||
activity — but it needs a second `NET_ADMIN` container in the pod that terminates the
|
||
internet-facing tunnel, and that trade was not worth making unverified. Pod-down is already
|
||
covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
|
||
kube-prometheus-stack. **This is the top follow-up.**
|
||
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
|
||
handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
|
||
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
|
||
`wg-server-key` Secret.
|
||
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
|
||
still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point
|
||
was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's
|
||
`AllowedIPs`; not done because nothing here needs it.
|
||
* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
|
||
browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
|
||
into the tunnel.
|