feat(vpn-ingress): WireGuard remote access on udp/51820 (hostNetwork, declarative peers)
Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus internet egress out the house WAN. Opposite direction to platform/vpn-egress, which sends pods OUT through PIA; the two share nothing but the word "VPN". WireGuard because it is the only option with a first-party client on all four named platforms, and because its handshake MAC is keyed with the responder's own public key: an unauthenticated packet gets NO reply, so udp/51820 is indistinguishable from a closed port. That, not "it is encrypted", is what makes an unfiltered WAN port defensible, and why there is no source allowlist. hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT clients to its own pod IP, so every remote client would carry that pod's Cilium identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. The obvious build would have shipped a healthy-looking tunnel on which every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve. In the host netns traffic is SNATed to the node and arrives as host/remote-node, exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the L2-announcement lease lottery, and double encapsulation (hence MTU 1420). Permitted by restrict-host-namespaces' own carve-out for platform namespaces. The cost is stated in networkpolicy.yaml rather than papered over: no NetworkPolicy constrains a hostNetwork pod. A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule; all three nodes answer the same keys, so failover is a one-line router change. Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my house" a reviewable PR diff. The only secret -- the server's private key -- is an out-of-band Secret: an agent must not mint a long-lived private key, and the two halves are joined only at pod start into a tmpfs that dies with the pod. Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain (kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md. Touches clusters/** , so ci/approval-policy.sh will block pending the operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
22c94effff
commit
1a67fc1b3c
16 files changed
+1391
-1
No files matched your search
@@ -120,6 +120,9 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
- **`vpn-egress-stale`** — the tunnel is UP but `egress-class=vpn` pods have no egress: Cilium's cached
|
||||
ifindex outlived a reconnect. Also covers the watchdog for that rule going inert.
|
||||
- **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.
|
||||
- **`vpn-remote-access`** — the INBOUND remote-access VPN (WireGuard, udp/51820): standing it up,
|
||||
enrolling and revoking devices, and the off-LAN checks that are the only honest proof it works.
|
||||
The opposite direction to the two pages above — do not confuse them.
|
||||
|
||||
### Backups and DR
|
||||
- **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here.
|
||||
|
||||
@@ -113,3 +113,8 @@ configMapGenerator:
|
||||
- update-rollback=update-rollback.md
|
||||
- vpn-down=vpn-down.md
|
||||
- vpn-egress-stale=vpn-egress-stale.md
|
||||
# Operator procedure, not an alert page. Served because the component it completes is INERT
|
||||
# until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
|
||||
# and because the one check that actually proves it cannot be run from inside the house.
|
||||
# Peer-level alerting does not exist yet (it needs an exporter); this page is the substitute.
|
||||
- vpn-remote-access=vpn-remote-access.md
|
||||
@@ -0,0 +1,203 @@
|
||||
# vpn-remote-access — standing up the WireGuard remote-access VPN, and proving it
|
||||
|
||||
**Status: the in-cluster half is shipped; the VPN is INERT until three manual steps are done.**
|
||||
All three are off-repo by necessity — a kernel module, a private key, and a router port-forward.
|
||||
**Alert:** pod-down is covered generically (`KubePodNotReady` / `KubeDeploymentReplicasMismatch`).
|
||||
There is **no peer-level alerting** — see "Why there is no connection alert". This is an **operator
|
||||
procedure**, not an incident page.
|
||||
**Files:** `platform/vpn-ingress/` (namespace, peer register, DaemonSet, baseline policy),
|
||||
`clusters/base/vpn-ingress.yaml` (its Flux tier), `clusters/base/cluster-vars.yaml`
|
||||
(`WG_CLIENT_CIDR`), `infra/ansible/roles/node_baseline/defaults/main.yml` (the `wireguard` module),
|
||||
`infra/vpn-ingress/new-peer.sh` (device enrolment).
|
||||
**Design rationale:** `platform/vpn-ingress/README.md` — read "Why `hostNetwork` is required"
|
||||
before changing anything about how this is wired.
|
||||
**Not to be confused with** `vpn-down` / `vpn-egress-stale`, which are about `platform/vpn-egress`
|
||||
(pods *out* through PIA). This page is remote clients *in*.
|
||||
|
||||
## What this gives you
|
||||
|
||||
A device — Android, Linux, Windows, macOS, iOS — dials in over **udp/51820** from anywhere and
|
||||
behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's web UIs, PiHole
|
||||
DNS, and internet egress out the house WAN.
|
||||
|
||||
Authorisation is one thing only: the device's **public key is in
|
||||
`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
|
||||
|
||||
> **An enrolled device is as privileged as a machine sitting in the house** — including the three
|
||||
> k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
|
||||
> the entire local network" means. Revoke a lost device promptly: delete its stanza, PR, merge.
|
||||
|
||||
## Step 1 — load the `wireguard` module on every node (once)
|
||||
|
||||
The DaemonSet creates its `wg0` link in the host netns with `CAP_NET_ADMIN` only; it is
|
||||
deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module. Without it
|
||||
`wg-quick` fails outright and the pod CrashLoops.
|
||||
|
||||
```bash
|
||||
# operator action (Ansible is not an agent action)
|
||||
cd infra/ansible && ./site.sh --tags system --limit k3s_prod
|
||||
```
|
||||
|
||||
Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:
|
||||
|
||||
```bash
|
||||
# expected: a `wireguard` line, and the modules-load.d file naming it for the next boot
|
||||
lsmod | grep '^wireguard'
|
||||
grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
|
||||
```
|
||||
|
||||
## Step 2 — create the server key (once)
|
||||
|
||||
This is the **only** secret in the component, and the one thing that must never be generated in an
|
||||
agent session or pasted into a transcript. Generate it on your own machine:
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
|
||||
umask 077
|
||||
wg genkey > ~/.config/wireguard-home/server.key # keep this; see "Back it up" below
|
||||
```
|
||||
|
||||
Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that
|
||||
lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6):
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \
|
||||
--from-file=privatekey=/work/.cache/server.key'
|
||||
```
|
||||
|
||||
…where `/work/.cache/server.key` is a path you copied the key to **inside the repo's gitignored
|
||||
cache** (or mount it however you prefer — the requirement is only that the bytes never pass through
|
||||
a log). Delete that copy afterwards: `shred -u .cache/server.key`.
|
||||
|
||||
Confirm it exists **by key name only**:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1'
|
||||
# expected: {"privatekey -- a NAME. If you see base64 payload, you printed the key. Rotate it.
|
||||
```
|
||||
|
||||
**Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that
|
||||
passed through a session transcript would have to be treated as compromised. The trade-off is that
|
||||
this Secret is **not reconstructible from git** — so:
|
||||
|
||||
**Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only
|
||||
`monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured
|
||||
off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work
|
||||
again after a cluster rebuild — **losing it means re-enrolling every device.**
|
||||
|
||||
## Step 3 — forward udp/51820 on the router (once)
|
||||
|
||||
On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
|
||||
`192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.
|
||||
|
||||
Pick any node — all three run an identical listener. Note which one you chose here:
|
||||
|
||||
```
|
||||
# WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set <date>
|
||||
```
|
||||
|
||||
**Node loss = failover is this one line.** Re-point the forward at another node address; every
|
||||
device keeps working with the same profile, because all three nodes answer the same keys.
|
||||
|
||||
Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
|
||||
already silent to unauthenticated probes (README, "Why this is safe to expose").
|
||||
|
||||
## Step 4 — confirm the listener is healthy
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress get ds wireguard -o wide'
|
||||
# expected: DESIRED 3 CURRENT 3 READY 3
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
|
||||
# expected: `interface: wg0`, `listening port: 51820`, and (before step 5) NO peers.
|
||||
```
|
||||
|
||||
A zero-peer interface that is Ready is the **correct** state before any device is enrolled: it
|
||||
listens, and refuses every handshake because no key is trusted.
|
||||
|
||||
The readiness/liveness probe is `wg show wg0`, and that is load-bearing rather than cosmetic: a
|
||||
port probe cannot distinguish "wg0 never came up" from "working", because WireGuard is silent to
|
||||
unauthenticated traffic either way. If the pod is Ready, the interface exists.
|
||||
|
||||
## Step 5 — enrol a device
|
||||
|
||||
```bash
|
||||
./infra/vpn-ingress/new-peer.sh pixel-8 2 # full tunnel (LAN + WAN egress)
|
||||
./infra/vpn-ingress/new-peer.sh thinkpad 3 --split # split tunnel (LAN only)
|
||||
```
|
||||
|
||||
It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout,
|
||||
prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then:
|
||||
|
||||
1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
|
||||
`# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
|
||||
comment is the only record of which stanza belongs to which device.
|
||||
2. PR → merge. Flux rolls the DaemonSet (~1 min).
|
||||
3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
|
||||
Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
|
||||
4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
|
||||
device's private key — the server never sees it and it cannot be re-derived.
|
||||
|
||||
Until the PR is merged the device handshakes and gets silence, exactly like an unknown key.
|
||||
|
||||
## Step 6 — verify it end to end (from OFF the LAN)
|
||||
|
||||
**This is the only honest test, and it cannot be run from inside the house.** Put the phone on
|
||||
cellular, with Wi-Fi off, and check all four — each proves a different hop:
|
||||
|
||||
| # | Check | Proves |
|
||||
|---|---|---|
|
||||
| 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza |
|
||||
| 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path |
|
||||
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd |
|
||||
| 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier |
|
||||
|
||||
Then confirm server-side:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
|
||||
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer.
|
||||
```
|
||||
|
||||
**If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not
|
||||
WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up.
|
||||
|
||||
**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60
|
||||
bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
|
||||
|
||||
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside —
|
||||
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
|
||||
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
|
||||
|
||||
## Revoking a device
|
||||
|
||||
Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
|
||||
next reconcile. **Never re-use that address octet**: an old profile would silently start working
|
||||
again as a different device.
|
||||
|
||||
## Why there is no connection alert
|
||||
|
||||
A WireGuard exporter would publish `wireguard_latest_handshake_seconds` per peer and make
|
||||
"something connected at 03:00" alertable. It needs a second `NET_ADMIN` container inside the pod
|
||||
that terminates the internet-facing tunnel, and that was not worth shipping unverified. **It is the
|
||||
top follow-up for this component.** Until then:
|
||||
|
||||
* pod-down is covered generically by kube-prometheus-stack's `KubePodNotReady` /
|
||||
`KubeDeploymentReplicasMismatch`;
|
||||
* "who is connected" is an on-demand question — `wg show`, above;
|
||||
* **never** `wg showconf` — it prints the server's private key.
|
||||
|
||||
## Known limitations
|
||||
|
||||
* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside
|
||||
the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point.
|
||||
* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's
|
||||
address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
|
||||
preserve them — at the cost of another hand-maintained off-repo route.
|
||||
* **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients
|
||||
re-handshake automatically (retry every 5s, rekey every 2 min).
|
||||
Reference in new issue
Block a user