# vpn-remote-access — standing up the WireGuard remote-access VPN, and proving it **Status: the in-cluster half is shipped; the VPN is INERT until three manual steps are done.** All three are off-repo by necessity — a kernel module, a private key, and a router port-forward. **Alert:** pod-down is covered generically (`KubePodNotReady` / `KubeDeploymentReplicasMismatch`). There is **no peer-level alerting** — see "Why there is no connection alert". This is an **operator procedure**, not an incident page. **Files:** `platform/vpn-ingress/` (namespace, peer register, DaemonSet, baseline policy), `clusters/base/vpn-ingress.yaml` (its Flux tier), `clusters/base/cluster-vars.yaml` (`WG_CLIENT_CIDR`), `infra/ansible/roles/node_baseline/defaults/main.yml` (the `wireguard` module), `infra/vpn-ingress/new-peer.sh` (device enrolment). **Design rationale:** `platform/vpn-ingress/README.md` — read "Why `hostNetwork` is required" before changing anything about how this is wired. **Not to be confused with** `vpn-down` / `vpn-egress-stale`, which are about `platform/vpn-egress` (pods *out* through PIA). This page is remote clients *in*. ## What this gives you A device — Android, Linux, Windows, macOS, iOS — dials in over **udp/51820** from anywhere and behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's web UIs, PiHole DNS, and internet egress out the house WAN. Authorisation is one thing only: the device's **public key is in `platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all. > **An enrolled device is as privileged as a machine sitting in the house** — including the three > k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach > the entire local network" means. Revoke a lost device promptly: delete its stanza, PR, merge. ## Step 1 — load the `wireguard` module on every node (once) The DaemonSet creates its `wg0` link in the host netns with `CAP_NET_ADMIN` only; it is deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module. Without it `wg-quick` fails outright and the pod CrashLoops. ```bash # operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox, # and ansible.cfg is only picked up from infra/ansible, so cd there inside the container. ./infra/devcontainer/dev.sh run bash -c \ 'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system' ``` Three details in that command are load-bearing, not style: - **`-e k3s_hosts=k3s_prod`.** `node-baseline.yml` targets `k3s_servers` by default, which is the **dev** group — and dev was decommissioned, so an unqualified run does nothing useful. This is the targeting pattern `infra/ansible/README.md` "Targeting prod" documents for every playbook. - **`--tags system`, and never an untagged run.** The two kernel-module tasks carry the `system` tag. An untagged run would also execute the `disks` tasks, which assert on block-device variables prod has deliberately left as placeholders (`infra/PROD-PROVISIONING.md` §2) — the playbook's own header warns about this. `--tags system` is broader than the modules alone (it also re-asserts sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and already in its desired state on a live node. - **It is a playbook, not a wrapper script.** There is no `site.sh` in this repo; `playbooks/site.yml` is the full bring-up chain and is the wrong tool here. Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded: ```bash # expected: a `wireguard` line, and the modules-load.d file naming it for the next boot lsmod | grep '^wireguard' grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf ``` ## Step 2 — create the server key (once) This is the **only** secret in the component, and the one thing that must never be generated in an agent session or pasted into a transcript. Generate it on your own machine: ```bash mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home umask 077 wg genkey > ~/.config/wireguard-home/server.key # keep this; see "Back it up" below ``` Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6): ```bash ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \ --from-file=privatekey=/work/.cache/server.key' ``` …where `/work/.cache/server.key` is a path you copied the key to (`.cache/` is not tracked; create it if absent). Delete that copy as soon as the Secret exists: `shred -u .cache/server.key`. > **KEEP THE `.key` SUFFIX.** That filename is what keeps the key out of git: `.gitignore` line 11 > is `*.key`, and it is the **only** rule covering this path — `.cache/` itself is NOT ignored > (`git check-ignore -v .cache/serverkey.txt` matches nothing). So a copy named `server.pem`, > `wg-server` or `privatekey` sitting in that directory is **committable**, and the gate's > `no-plaintext-secrets` stage does not know a WireGuard private key when it sees one. Verify before > you trust it: `git check-ignore -v .cache/` must print a rule, and > `git status --short` must not list the file. Confirm it exists **by key name only**: ```bash ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1' # expected: {"privatekey -- a NAME. If you see base64 payload, you printed the key. Rotate it. ``` **Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that passed through a session transcript would have to be treated as compromised. The trade-off is that this Secret is **not reconstructible from git** — so: **Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only `monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work again after a cluster rebuild — **losing it means re-enrolling every device.** ## Step 3 — forward udp/51820 on the router (once) On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g. `192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check. Pick any node — all three run an identical listener. Note which one you chose here: ``` # WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set ``` **Node loss = failover is this one line.** Re-point the forward at another node address; every device keeps working with the same profile, because all three nodes answer the same keys. Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is already silent to unauthenticated probes (README, "Why this is safe to expose"). ## Step 4 — confirm the listener is healthy ```bash ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress get ds wireguard -o wide' # expected: DESIRED 3 CURRENT 3 READY 3 ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show' # expected: `interface: wg0`, `listening port: 51820`, and (before step 5) NO peers. ``` A zero-peer interface that is Ready is the **correct** state before any device is enrolled: it listens, and refuses every handshake because no key is trusted. The readiness/liveness probe is `wg show wg0`, and that is load-bearing rather than cosmetic: a port probe cannot distinguish "wg0 never came up" from "working", because WireGuard is silent to unauthenticated traffic either way. If the pod is Ready, the interface exists. ## Step 5 — enrol a device ```bash ./infra/vpn-ingress/new-peer.sh pixel-8 2 # full tunnel (LAN + WAN egress) ./infra/vpn-ingress/new-peer.sh thinkpad 3 --split # split tunnel (LAN only) ``` It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout, prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then: 1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the `# — , enrolled ` comment. `wg show` can only ever print the key, so that comment is the only record of which stanza belongs to which device. 2. PR → merge. Flux rolls the DaemonSet (~1 min). 3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on Windows/macOS; `/etc/wireguard/.conf` + `wg-quick@` on Linux). 4. **Delete the profile from your machine**: `shred -u `. It is the only copy of that device's private key — the server never sees it and it cannot be re-derived. Until the PR is merged the device handshakes and gets silence, exactly like an unknown key. ## Step 6 — verify it end to end (from OFF the LAN) **This is the only honest test, and it cannot be run from inside the house.** Put the phone on cellular, with Wi-Fi off, and check all four — each proves a different hop: | # | Check | Proves | |---|---|---| | 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza | | 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path | | 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd | | 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier | Then confirm server-side: ```bash ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show' # a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer. ``` **If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up. **If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60 bytes of WireGuard encapsulation); drop the client to `1280` and re-test. **If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside — by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes if the router is not delivering the packets, and a `handshake` with no completed session if it is. ## Revoking a device Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the next reconcile. **Never re-use that address octet**: an old profile would silently start working again as a different device. ## Why there is no connection alert A WireGuard exporter would publish `wireguard_latest_handshake_seconds` per peer and make "something connected at 03:00" alertable. It needs a second `NET_ADMIN` container inside the pod that terminates the internet-facing tunnel, and that was not worth shipping unverified. **It is the top follow-up for this component.** Until then: * pod-down is covered generically by kube-prometheus-stack's `KubePodNotReady` / `KubeDeploymentReplicasMismatch`; * "who is connected" is an on-demand question — `wg show`, above; * **never** `wg showconf` — it prints the server's private key. ## Known limitations * **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point. * **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would preserve them — at the cost of another hand-maintained off-repo route. * **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients re-handshake automatically (retry every 5s, rekey every 2 min).