home-network-k3s-impl/runbooks/vpn-remote-access.md

vpn-remote-access — standing up the WireGuard remote-access VPN, and proving it

Status: the in-cluster half is shipped; the VPN is INERT until three manual steps are done. All three are off-repo by necessity — a kernel module, a private key, and a router port-forward. Alert: pod-down is covered generically (KubePodNotReady / KubeDeploymentReplicasMismatch). There is no peer-level alerting — see "Why there is no connection alert". This is an operator procedure, not an incident page. Files: platform/vpn-ingress/ (namespace, peer register, DaemonSet, baseline policy), clusters/base/vpn-ingress.yaml (its Flux tier), clusters/base/cluster-vars.yaml (WG_CLIENT_CIDR), infra/ansible/roles/node_baseline/defaults/main.yml (the wireguard module), infra/vpn-ingress/new-peer.sh (device enrolment). Design rationale: platform/vpn-ingress/README.md — read "Why hostNetwork is required" before changing anything about how this is wired. Not to be confused with vpn-down / vpn-egress-stale, which are about platform/vpn-egress (pods out through PIA). This page is remote clients in.

What this gives you

A device — Android, Linux, Windows, macOS, iOS — dials in over udp/51820 from anywhere and behaves like a machine on the LAN: full reach to 192.168.2.0/24, the Gateway's web UIs, PiHole DNS, and internet egress out the house WAN.

Authorisation is one thing only: the device's public key is in platform/vpn-ingress/peers.yaml. An unlisted key gets no reply at all.

An enrolled device is as privileged as a machine sitting in the house — including the three k3s nodes (192.168.2.10/.11/.12: apiserver 6443, kubelet 10250, SSH 22). That is what "reach the entire local network" means. Revoke a lost device promptly: delete its stanza, PR, merge.

Step 1 — load the wireguard module on every node (once)

The DaemonSet creates its wg0 link in the host netns with CAP_NET_ADMIN only; it is deliberately not granted CAP_SYS_MODULE, so it cannot autoload the module. Without it wg-quick fails outright and the pod CrashLoops.

# operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox,
# and ansible.cfg is only picked up from infra/ansible, so cd there inside the container.
./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'

Three details in that command are load-bearing, not style:

• -e k3s_hosts=k3s_prod. node-baseline.yml targets k3s_servers by default, which is the dev group — and dev was decommissioned, so an unqualified run does nothing useful. This is the targeting pattern infra/ansible/README.md "Targeting prod" documents for every playbook.
• --tags system, and never an untagged run. The two kernel-module tasks carry the system tag. An untagged run would also execute the disks tasks, which assert on block-device variables prod has deliberately left as placeholders (infra/PROD-PROVISIONING.md §2) — the playbook's own header warns about this. --tags system is broader than the modules alone (it also re-asserts sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and already in its desired state on a live node.
• It is a playbook, not a wrapper script. There is no site.sh in this repo; playbooks/site.yml is the full bring-up chain and is the wrong tool here.

Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:

# expected: a `wireguard` line, and the modules-load.d file naming it for the next boot
lsmod | grep '^wireguard'
grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf

Step 2 — create the server key (once)

This is the only secret in the component, and the one thing that must never be generated in an agent session or pasted into a transcript. Generate it on your own machine:

mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
umask 077
wg genkey > ~/.config/wireguard-home/server.key          # keep this; see "Back it up" below

Load it into the cluster. Do not echo it, do not pass it on an interactive command line that lands in shell history, and never print the resulting Secret's .data (CLAUDE.md §6):

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \
     --from-file=privatekey=/work/.cache/server.key'

…where /work/.cache/server.key is a path you copied the key to (.cache/ is not tracked; create it if absent). Delete that copy as soon as the Secret exists: shred -u .cache/server.key.

KEEP THE .key SUFFIX. That filename is what keeps the key out of git: .gitignore line 11 is *.key, and it is the only rule covering this path — .cache/ itself is NOT ignored (git check-ignore -v .cache/serverkey.txt matches nothing). So a copy named server.pem, wg-server or privatekey sitting in that directory is committable, and the gate's no-plaintext-secrets stage does not know a WireGuard private key when it sees one. Verify before you trust it: git check-ignore -v .cache/<your-filename> must print a rule, and git status --short must not list the file.

Confirm it exists by key name only:

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1'
# expected: {"privatekey   -- a NAME. If you see base64 payload, you printed the key. Rotate it.

Why out of band and not SOPS. An agent must not mint a long-lived private key, and a key that passed through a session transcript would have to be treated as compromised. The trade-off is that this Secret is not reconstructible from git — so:

Back it up. Keep server.key in your password manager. Velero's daily schedule excludes only monitoring and unifi (platform/velero/helmrelease.yaml), so this namespace is captured off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work again after a cluster rebuild — losing it means re-enrolling every device.

Step 3 — forward udp/51820 on the router (once)

On the SOHO router at 192.168.2.1, forward WAN udp/51820 to one node address, e.g. 192.168.2.10:51820. This is hand-made and off-repo; it is the half CI can never check.

Pick any node — all three run an identical listener. Note which one you chose here:

# WAN udp/51820 -> 192.168.2.__  (k3s-prod-_)   set <date>

Node loss = failover is this one line. Re-point the forward at another node address; every device keeps working with the same profile, because all three nodes answer the same keys.

Do not also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is already silent to unauthenticated probes (README, "Why this is safe to expose").

Step 4 — confirm the listener is healthy

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress get ds wireguard -o wide'
# expected: DESIRED 3  CURRENT 3  READY 3

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
# expected: `interface: wg0`, `listening port: 51820`, and (before step 5) NO peers.

A zero-peer interface that is Ready is the correct state before any device is enrolled: it listens, and refuses every handshake because no key is trusted.

The readiness/liveness probe is wg show wg0, and that is load-bearing rather than cosmetic: a port probe cannot distinguish "wg0 never came up" from "working", because WireGuard is silent to unauthenticated traffic either way. If the pod is Ready, the interface exists.

Step 5 — enrol a device

./infra/vpn-ingress/new-peer.sh pixel-8 2              # full tunnel  (LAN + WAN egress)
./infra/vpn-ingress/new-peer.sh thinkpad 3 --split     # split tunnel (LAN only)

It generates the keypair on your machine, writes a 0600 client profile outside the checkout, prints a QR code if qrencode is installed, and prints the [Peer] stanza to add. Then:

1. Paste the stanza into platform/vpn-ingress/peers.yaml under data.peers.conf, keeping the # <name> — <owner>, enrolled <date> comment. wg show can only ever print the key, so that comment is the only record of which stanza belongs to which device.
2. PR → merge. Flux rolls the DaemonSet (~1 min).
3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on Windows/macOS; /etc/wireguard/<name>.conf + wg-quick@<name> on Linux).
4. Delete the profile from your machine: shred -u <path>. It is the only copy of that device's private key — the server never sees it and it cannot be re-derived.

Until the PR is merged the device handshakes and gets silence, exactly like an unknown key.

Step 6 — verify it end to end (from OFF the LAN)

This is the only honest test, and it cannot be run from inside the house. Put the phone on cellular, with Wi-Fi off, and check all four — each proves a different hop:

#	Check	Proves
1	tunnel shows a recent handshake in the WireGuard app	the WAN forward + the server key + the peer stanza
2	ping 192.168.2.1 (the router)	decrypt → FORWARD → LAN, and the SNAT return path
3	open https://plex.home.keithjackson.dev	PiHole DNS over the tunnel and the Gateway accepting a host-identity source — the specific thing a pod-networked VPN would have 403'd
4	visit an IP-echo site; it should show 136.51.115.252	full-tunnel WAN egress out the house, not the carrier

Then confirm server-side:

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer.

If check 3 fails while 2 passes, the tunnel is fine and the problem is DNS or the Gateway, not WireGuard. Check the client's DNS = 192.168.2.23 and that PiHole is up.

If 1 and 2 pass but large transfers hang, it is MTU. The profile ships MTU = 1420 (1500 − 60 bytes of WireGuard encapsulation); drop the client to 1280 and re-test.

If nothing handshakes at all, you cannot tell a wrong forward from a wrong key from outside — by design. Work inwards: wg show on the node holding the forward will show zero received bytes if the router is not delivering the packets, and a handshake with no completed session if it is.

Revoking a device

Delete its [Peer] stanza from peers.yaml, PR, merge. Nothing else — the key stops working on the next reconcile. Never re-use that address octet: an old profile would silently start working again as a different device.

Why there is no connection alert

A WireGuard exporter would publish wireguard_latest_handshake_seconds per peer and make "something connected at 03:00" alertable. It needs a second NET_ADMIN container inside the pod that terminates the internet-facing tunnel, and that was not worth shipping unverified. It is the top follow-up for this component. Until then:

• pod-down is covered generically by kube-prometheus-stack's KubePodNotReady / KubeDeploymentReplicasMismatch;
• "who is connected" is an on-demand question — wg show, above;
• never wg showconf — it prints the server's private key.

Known limitations

• No IPv6 in the tunnel. A full-tunnel client with native IPv6 still reaches IPv6 sites outside the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point.
• Client IPs are not preserved on the LAN side: everything arrives SNATed as the node's address. A static route for 10.90.0.0/24 → the forwarded node, added on the router, would preserve them — at the cost of another hand-maintained off-repo route.
• Rollouts briefly drop the forwarded node. maxUnavailable: 1, so one node at a time; clients re-handshake automatically (retry every 5s, rekey every 2 min).