home-network-k3s-impl/runbooks/vpn-remote-access.md

vpn-remote-access — standing up the WireGuard remote-access VPN, and proving it

Status: the in-cluster half is shipped; the VPN is INERT until three manual steps are done. All three are off-repo by necessity — a kernel module, a private key, and a router port-forward. Alert: pod-down is covered generically (KubePodNotReady / KubeDeploymentReplicasMismatch). There is no peer-level alerting — see "Why there is no connection alert". This is an operator procedure, not an incident page. Files: platform/vpn-ingress/ (namespace, peer register, DaemonSet, baseline policy), clusters/base/vpn-ingress.yaml (its Flux tier), clusters/base/cluster-vars.yaml (WG_CLIENT_CIDR), infra/ansible/roles/node_baseline/defaults/main.yml (the wireguard module), infra/vpn-ingress/new-peer.sh (device enrolment). Design rationale: platform/vpn-ingress/README.md — read "Why hostNetwork is required" before changing anything about how this is wired. Not to be confused with vpn-down / vpn-egress-stale, which are about platform/vpn-egress (pods out through PIA). This page is remote clients in.

What this gives you

A device — Android, Linux, Windows, macOS, iOS — dials in over udp/51820 from anywhere and behaves like a machine on the LAN: full reach to 192.168.2.0/24, the Gateway's web UIs, PiHole DNS, and internet egress out the house WAN.

Authorisation is one thing only: the device's public key is in platform/vpn-ingress/peers.conf. An unlisted key gets no reply at all.

An enrolled device is as privileged as a machine sitting in the house — including the three k3s nodes (192.168.2.10/.11/.12: apiserver 6443, kubelet 10250, SSH 22). That is what "reach the entire local network" means. Revoke a lost device promptly: delete its stanza, PR, merge.

Step 1 — load the wireguard module on every node (once)

The DaemonSet creates its wg0 link in the host netns with CAP_NET_ADMIN only; it is deliberately not granted CAP_SYS_MODULE, so it cannot autoload the module. Without it wg-quick fails outright and the pod CrashLoops.

# operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox,
# and ansible.cfg is only picked up from infra/ansible, so cd there inside the container.
./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'

Three details in that command are load-bearing, not style:

• -e k3s_hosts=k3s_prod. node-baseline.yml targets k3s_servers by default, which is the dev group — and dev was decommissioned, so an unqualified run does nothing useful. This is the targeting pattern infra/ansible/README.md "Targeting prod" documents for every playbook.
• --tags system, and never an untagged run. The two kernel-module tasks carry the system tag. An untagged run would also execute the disks tasks, which assert on block-device variables prod has deliberately left as placeholders (infra/PROD-PROVISIONING.md §2) — the playbook's own header warns about this. --tags system is broader than the modules alone (it also re-asserts sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and already in its desired state on a live node.
• It is a playbook, not a wrapper script. There is no site.sh in this repo; playbooks/site.yml is the full bring-up chain and is the wrong tool here.

Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:

# expected: a `wireguard` line, and the modules-load.d file naming it for the next boot
lsmod | grep '^wireguard'
grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf

Step 2 — fill in the server key (once)

The key is SOPS-encrypted in this tree (R-SEC-01, spec docs/07), not created by hand: platform/vpn-ingress/wg-server-key.sops.yaml. It is already committed — encrypted, with a REPLACE-ME placeholder — and it is already listed in the component's resources: with a decryption: patch in both overlays. Nothing needs wiring. You are only replacing one value.

Why in git rather than kubectl create secret: an out-of-band Secret puts the one credential every enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and losing it means re-enrolling every device, because each client profile pins this key's public half.

Everything below runs inside the dev container — wg, sops and qrencode are all baked into the image (CLAUDE.md §5: the toolchain lives only in the pinned container; don't install it on the host). One shell, one toolchain.

1. Generate the key straight into the encrypted file. One pipeline: wg genkey makes it, sops set writes it into the already-encrypted document in place. The plaintext never lands on disk and never reaches a terminal.

./infra/devcontainer/dev.sh run bash -c '
  cd /work
  umask 077
  sops set platform/vpn-ingress/wg-server-key.sops.yaml \
    "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
'

2. Record it in your PASSWORD MANAGER. Derive both halves from the file you just wrote:

# the PUBLIC half — safe to display, and what every client profile carries
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'

# the PRIVATE half — paste into the password manager, then clear your scrollback
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
  platform/vpn-ingress/wg-server-key.sops.yaml'

The second command is the one place this key is deliberately shown. The encrypted file is in git and Velero captures this namespace to R2, so the password-manager copy is belt-and-braces — but it is the only copy that survives losing the cluster and the backups, and the key cannot be regenerated without invalidating every client profile.

3. Confirm the value is a real key, without printing it — if wg pubkey prints a key, the value is valid; if it errors, it is still wrong:

./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null && echo "valid WireGuard key"'

That is the same check the pod's init container runs, so a pass here predicts a pass there.

4. PR it, and merge. Flux rolls the DaemonSet and the pods leave CrashLoop.

If you would rather use an editor than sops set: ./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml (replace REPLACE-ME, save, exit — it re-encrypts on write).

What the pod tells you if this went wrong

The init container validates the value with wg pubkey before writing wg0.conf, and names the specific failure in its log — so you never get a server that silently cannot complete a handshake:

pod log says	meaning
still the committed REPLACE-ME PLACEHOLDER	step 3 not done (the expected fresh-install state)
It is EMPTY	the value was blanked
still SOPS CIPHERTEXT (starts with ENC[)	the Secret reached the cluster undecrypted — this environment's overlay is missing its vpn-ingress decryption patch. This is the dangerous one: the Secret applies cleanly and every Flux condition stays green
wg pubkey rejected it with none of the above	a mangled value, or a public key pasted by mistake
net.ipv4.ip_forward is 0 on this node	the node_baseline role has not run here (step 1). The pod cannot set this itself — under hostNetwork the runtime mounts /proc/sys read-only, because it is the host's global setting
0 peer(s) after you merged a peer	the DaemonSet did not roll. Check its generation (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted peers.conf to a plain resource instead of a configMapGenerator

What a healthy start looks like

Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key:

wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820.

then the wireguard container brings the tunnel up:

**** Found WG conf /config/wg_confs/wg0.conf, adding to list ****
**** Activating tunnel /config/wg_confs/wg0.conf ****
[#] ip link add dev wg0 type wireguard
[#] wg addconf wg0 /dev/fd/63
[#] ip -4 address add 10.90.0.1/24 dev wg0
[#] ip link set mtu 1420 up dev wg0

Ignore the chown: ... Operation not permitted run before that, and the image's own "Permissions could not be set ... we will not provide support for it". Capabilities are dropped to NET_ADMIN only, so its s6 init has no CAP_CHOWN; /config is a writable tmpfs, everything runs as root, and wg-quick proceeds. It is wrong about the cause and it does not matter.

A pod stuck 0/1 Running with Startup probe failed: Unable to access interface: No such device means wg-quick tore the tunnel down — read the wireguard container log for the [#] line it died on. That is the shape of a real failure; the probe is doing its job of refusing to call a server without a wg0 "ready".

Step 3 — forward udp/51820 on the router (once)

On the SOHO router at 192.168.2.1, forward WAN udp/51820 to one node address, e.g. 192.168.2.10:51820. This is hand-made and off-repo; it is the half CI can never check.

Pick any node — all three run an identical listener. Note which one you chose here:

# WAN udp/51820 -> 192.168.2.__  (k3s-prod-_)   set <date>

Node loss = failover is this one line. Re-point the forward at another node address; every device keeps working with the same profile, because all three nodes answer the same keys.

Do not also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is already silent to unauthenticated probes (README, "Why this is safe to expose").

Step 4 — confirm the listener is healthy

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress get ds wireguard -o wide'
# expected: DESIRED 3  CURRENT 3  READY 3

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
# expected: `interface: wg0`, `listening port: 51820`, and (before step 5) NO peers.

A zero-peer interface that is Ready is the correct state before any device is enrolled: it listens, and refuses every handshake because no key is trusted.

The readiness/liveness probe is wg show wg0, and that is load-bearing rather than cosmetic: a port probe cannot distinguish "wg0 never came up" from "working", because WireGuard is silent to unauthenticated traffic either way. If the pod is Ready, the interface exists.

Step 5 — enrol a device

# in the dev container, like everything else
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2'            # full tunnel
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh thinkpad 3 --split'   # LAN only

It derives the server's public key from wg-server-key.sops.yaml itself, so there is no local key copy to keep in step with anything. Profiles land in .cache/wg-peers/ — gitignored, and persistent because the repo is the container's only read-write mount.

It generates the device's keypair, writes a 0600 client profile, prints a QR code, and prints the [Peer] stanza to add. It refuses to write a profile anywhere git does not ignore.

Then:

1. Append the stanza to platform/vpn-ingress/peers.conf, keeping the # <name> — <owner>, enrolled <date> comment. wg show can only ever print the key, so that comment is the only record of which stanza belongs to which device.

2. PR → merge. Flux rolls the DaemonSet (~1 min), because peers.conf is a content-hashed configMapGenerator: editing it changes the ConfigMap's generated name, which changes the pod template, which forces the roll. (It did not work this way at first — a fixed-name ConfigMap updates in place, the pod template never changes, and the init container keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:

   ./infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
        -o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
   

   The generation must have increased since before the merge. If it is still 1, the roll did not happen and wg show will list no peer no matter how long you wait.

3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on Windows/macOS; /etc/wireguard/<name>.conf + wg-quick@<name> on Linux).

4. Delete the profile from your machine: shred -u <path>. It is the only copy of that device's private key — the server never sees it and it cannot be re-derived.

Until the PR is merged the device handshakes and gets silence, exactly like an unknown key.

Step 6 — verify it end to end (from OFF the LAN)

This is the only honest test, and it cannot be run from inside the house. Put the phone on cellular, with Wi-Fi off, and check all four — each proves a different hop:

#	Check	Proves
1	tunnel shows a recent handshake in the WireGuard app	the WAN forward + the server key + the peer stanza
2	ping 192.168.2.1 (the router)	decrypt → FORWARD → LAN, and the SNAT return path
3	open https://plex.home.keithjackson.dev	PiHole DNS over the tunnel and the Gateway admitting a tunnel client, which a pod-networked VPN would have 403'd. Expected source identity is world, like a LAN browser: measured 2026-11-09 on VIPs .23 and .24, not yet observed on the Gateway's .20
4	visit an IP-echo site; it should show 136.51.115.252	full-tunnel WAN egress out the house, not the carrier

Then confirm server-side:

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer. `ds/wireguard`
# picks ONE pod; if it reads 0, ask the pod on the node holding the forward (see Troubleshooting).

If check 3 fails while 2 passes, the tunnel is fine and the problem is DNS or the Gateway, not WireGuard. Check the client's DNS = 192.168.2.23 and that PiHole is up.

If 1 and 2 pass but pages or large transfers hang, suspect the client MTU; see Troubleshooting.

If nothing handshakes at all, you cannot tell a wrong forward from a wrong key from the device, by design. wg show cannot tell them apart either: it reads the same in both cases. The node's conntrack table can; see Troubleshooting.

Troubleshooting

Ask the pod on the node carrying the traffic: the node the WAN forward names for away, k3s-prod-1 for home. Sessions are per node, and the other two pods read 0.

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress get pods -o wide'

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- wg show wg0 latest-handshakes'
# likewise: `wg show wg0 endpoints`, `wg show wg0 transfer`

Use only those three subcommands to diagnose. Never wg showconf, wg show wg0 private-key or wg show wg0 preshared-keys: they print key material into your terminal and transcript.

output	reading
latest-handshakes: <public key> 0	no handshake with this peer since the pod started
endpoints: <public key> (none)	the server has not accepted a packet from this peer since the pod started
transfer: <public key> <rx> <tx>	bytes, as the server counts them

The app shows TX, no RX and no "Latest handshake"

Symptom. The app's transmit counter climbs and receive stays at zero. On the server, endpoints is (none) and latest-handshakes is 0 for that peer.

Cause. A key mismatch: the server's public key in the profile, or the device's public key in peers.conf. WireGuard drops an initiation it cannot authenticate and sends nothing back, so neither end logs an error. Measured 2026-11-09: a mistyped server public key in the phone's profile.

Check. First prove the packets arrive. The pod has no conntrack binary, but in hostNetwork it can read the host's table:

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- grep dport=51820 /proc/net/nf_conntrack'

• A line whose first tuple is src=<the phone's public address> ... dport=51820, marked [UNREPLIED]: the datagrams reach the node and the server stays silent. It is the keys.
• No such line: the packets are not arriving. Look at the router forward, the DNS name, or the device's network.

Then compare the keys by copy and paste, never by eye and never retyped: the server's public half (the first command in Step 2's "Record it in your PASSWORD MANAGER") against the profile's PublicKey, and the device's public key from the app against its stanza in peers.conf.

Works on Wi-Fi, stalls on mobile data, handshake healthy

Symptom. latest-handshakes is recent and some traffic flows, but names or pages fail on mobile data. The same device is fine on Wi-Fi.

Cause. Most likely a client MTU too large for the carrier's path: a tunnel MTU of 1420 means outer packets of up to 1480 bytes. This is inferred from one before-and-after, on one phone and one carrier (2026-11-09): 1420 failed and clearing the field fixed it. The path MTU itself was not measured.

Check. In the Android WireGuard app, edit the tunnel, clear the MTU field or set it to 1280, save, then turn the tunnel off and on. A blank field means 1280 in the app's default userspace backend (GoBackend: getMtu().orElse(1280)). On other clients set MTU = 1280 in the profile.

new-peer.sh now generates MTU = 1280 unless WG_MTU overrides it. A profile already on a device does not change, so edit it by hand. If 1280 does not help, it is not the MTU: go back to Step 6's check 3.

The away profile cannot connect from home

Symptom. On the home network, away never handshakes.

Cause. Its Endpoint is wan.home.keithjackson.dev, the house's WAN address, and the router has no hairpin NAT. WireGuard cannot use the Cloudflare Tunnel instead.

Check. Turn Wi-Fi off. If away handshakes on mobile data, the profile is fine: use the home tunnel indoors, as described in Phone profiles: one tunnel for home, one for away.

Revoking a device

Delete its [Peer] stanza from peers.conf, PR, merge. Nothing else — the key stops working on the next reconcile. Never re-use that address octet: an old profile would silently start working again as a different device.

Why there is no connection alert

A WireGuard exporter would publish wireguard_latest_handshake_seconds per peer and make "something connected at 03:00" alertable. It needs a second NET_ADMIN container inside the pod that terminates the internet-facing tunnel, and that was not worth shipping unverified. It is the top follow-up for this component. Until then:

• pod-down is covered generically by kube-prometheus-stack's KubePodNotReady / KubeDeploymentReplicasMismatch;
• "who is connected" is an on-demand question — wg show, above;
• never wg showconf — it prints the server's private key.

Known limitations

• No IPv6 in the tunnel. What a client's IPv6 does then depends on the client. In the Android app's default GoBackend, a config with one peer and a default route skips allowFamily(), so Android should block IPv6 while the tunnel is up and apps fall back to IPv4. That is read from upstream source, not observed on a device. A --split profile has no default route, so its IPv6 stays outside the tunnel. Expect wg-quick on a dual-stack Linux host to leave IPv6 outside too (not tested).
• Client IPs are only partly preserved. LAN hosts and the internet see the node's address (MASQUERADE). A static route for 10.90.0.0/24 → the forwarded node, added on the router, would preserve them there, at the cost of another hand-maintained off-repo route. LoadBalancer VIPs are different: Cilium handles the packet before MASQUERADE, so the source stays 10.90.0.x, identity world (BPF conntrack on prod-1, 2026-11-09, VIPs .23 and .24). Any loadBalancerSourceRanges must therefore list the tunnel subnet, as PiHole's do.
• Rollouts briefly drop the forwarded node. maxUnavailable: 1, so one node at a time; clients re-handshake automatically (retry every 5s, rekey every 2 min).