# vpn-remote-access — standing up the WireGuard remote-access VPN, and proving it **Status: the in-cluster half is shipped; the VPN is INERT until three manual steps are done.** All three are off-repo by necessity — a kernel module, a private key, and a router port-forward. **Alert:** pod-down is covered generically (`KubePodNotReady` / `KubeDeploymentReplicasMismatch`). There is **no peer-level alerting** — see "Why there is no connection alert". This is an **operator procedure**, not an incident page. **Files:** `platform/vpn-ingress/` (namespace, peer register, DaemonSet, baseline policy), `clusters/base/vpn-ingress.yaml` (its Flux tier), `clusters/base/cluster-vars.yaml` (`WG_CLIENT_CIDR`), `infra/ansible/roles/node_baseline/defaults/main.yml` (the `wireguard` module), `infra/vpn-ingress/new-peer.sh` (device enrolment). **Design rationale:** `platform/vpn-ingress/README.md` — read "Why `hostNetwork` is required" before changing anything about how this is wired. **Not to be confused with** `vpn-down` / `vpn-egress-stale`, which are about `platform/vpn-egress` (pods *out* through PIA). This page is remote clients *in*. ## What this gives you A device — Android, Linux, Windows, macOS, iOS — dials in over **udp/51820** from anywhere and behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's web UIs, PiHole DNS, and internet egress out the house WAN. Authorisation is one thing only: the device's **public key is in `platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all. > **An enrolled device is as privileged as a machine sitting in the house** — including the three > k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach > the entire local network" means. Revoke a lost device promptly: delete its stanza, PR, merge. ## Step 1 — load the `wireguard` module on every node (once) The DaemonSet creates its `wg0` link in the host netns with `CAP_NET_ADMIN` only; it is deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module. Without it `wg-quick` fails outright and the pod CrashLoops. ```bash # operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox, # and ansible.cfg is only picked up from infra/ansible, so cd there inside the container. ./infra/devcontainer/dev.sh run bash -c \ 'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system' ``` Three details in that command are load-bearing, not style: - **`-e k3s_hosts=k3s_prod`.** `node-baseline.yml` targets `k3s_servers` by default, which is the **dev** group — and dev was decommissioned, so an unqualified run does nothing useful. This is the targeting pattern `infra/ansible/README.md` "Targeting prod" documents for every playbook. - **`--tags system`, and never an untagged run.** The two kernel-module tasks carry the `system` tag. An untagged run would also execute the `disks` tasks, which assert on block-device variables prod has deliberately left as placeholders (`infra/PROD-PROVISIONING.md` §2) — the playbook's own header warns about this. `--tags system` is broader than the modules alone (it also re-asserts sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and already in its desired state on a live node. - **It is a playbook, not a wrapper script.** There is no `site.sh` in this repo; `playbooks/site.yml` is the full bring-up chain and is the wrong tool here. Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded: ```bash # expected: a `wireguard` line, and the modules-load.d file naming it for the next boot lsmod | grep '^wireguard' grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf ``` ## Step 2 — fill in the server key (once) The key is **SOPS-encrypted in this tree** (R-SEC-01, spec docs/07), not created by hand: `platform/vpn-ingress/wg-server-key.sops.yaml`. It is already committed — encrypted, with a `REPLACE-ME` placeholder — and it is already listed in the component's `resources:` with a `decryption:` patch in both overlays. **Nothing needs wiring. You are only replacing one value.** Why in git rather than `kubectl create secret`: an out-of-band Secret puts the one credential every enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and losing it means re-enrolling every device, because each client profile pins this key's public half. Everything below runs **inside the dev container** — `wg`, `sops` and `qrencode` are all baked into the image (CLAUDE.md §5: the toolchain lives only in the pinned container; don't install it on the host). One shell, one toolchain. **1. Generate the key straight into the encrypted file.** One pipeline: `wg genkey` makes it, `sops set` writes it into the already-encrypted document in place. The plaintext never lands on disk and never reaches a terminal. ```bash ./infra/devcontainer/dev.sh run bash -c ' cd /work umask 077 sops set platform/vpn-ingress/wg-server-key.sops.yaml \ "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\"" ' ``` **2. Record it in your PASSWORD MANAGER.** Derive both halves from the file you just wrote: ```bash # the PUBLIC half — safe to display, and what every client profile carries ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \ platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey' # the PRIVATE half — paste into the password manager, then clear your scrollback ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \ platform/vpn-ingress/wg-server-key.sops.yaml' ``` The second command is the one place this key is deliberately shown. The encrypted file is in git and Velero captures this namespace to R2, so the password-manager copy is belt-and-braces — but it is the only copy that survives losing the cluster *and* the backups, and the key cannot be regenerated without invalidating every client profile. **3. Confirm the value is a real key**, without printing it — if `wg pubkey` prints a key, the value is valid; if it errors, it is still wrong: ```bash ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \ platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null && echo "valid WireGuard key"' ``` That is the same check the pod's init container runs, so a pass here predicts a pass there. **4. PR it, and merge.** Flux rolls the DaemonSet and the pods leave CrashLoop. If you would rather use an editor than `sops set`: `./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml` (replace `REPLACE-ME`, save, exit — it re-encrypts on write). ### What the pod tells you if this went wrong The init container validates the value with `wg pubkey` before writing `wg0.conf`, and names the specific failure in its log — so you never get a server that silently cannot complete a handshake: | pod log says | meaning | |---|---| | `still the committed REPLACE-ME PLACEHOLDER` | step 3 not done (the expected fresh-install state) | | `It is EMPTY` | the value was blanked | | `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green | | `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake | | `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting | | `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` | ### What a healthy start looks like Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key: ``` wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820. ``` then the `wireguard` container brings the tunnel up: ``` **** Found WG conf /config/wg_confs/wg0.conf, adding to list **** **** Activating tunnel /config/wg_confs/wg0.conf **** [#] ip link add dev wg0 type wireguard [#] wg addconf wg0 /dev/fd/63 [#] ip -4 address add 10.90.0.1/24 dev wg0 [#] ip link set mtu 1420 up dev wg0 ``` **Ignore the `chown: ... Operation not permitted` run before that**, and the image's own "Permissions could not be set ... we will not provide support for it". Capabilities are dropped to `NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter. **A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device` means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it died on. That is the shape of a real failure; the probe is doing its job of refusing to call a server without a `wg0` "ready". ## Step 3 — forward udp/51820 on the router (once) On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g. `192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check. Pick any node — all three run an identical listener. Note which one you chose here: ``` # WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set ``` **Node loss = failover is this one line.** Re-point the forward at another node address; every device keeps working with the same profile, because all three nodes answer the same keys. Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is already silent to unauthenticated probes (README, "Why this is safe to expose"). ## Step 4 — confirm the listener is healthy ```bash ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress get ds wireguard -o wide' # expected: DESIRED 3 CURRENT 3 READY 3 ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show' # expected: `interface: wg0`, `listening port: 51820`, and (before step 5) NO peers. ``` A zero-peer interface that is Ready is the **correct** state before any device is enrolled: it listens, and refuses every handshake because no key is trusted. The readiness/liveness probe is `wg show wg0`, and that is load-bearing rather than cosmetic: a port probe cannot distinguish "wg0 never came up" from "working", because WireGuard is silent to unauthenticated traffic either way. If the pod is Ready, the interface exists. ## Step 5 — enrol a device ```bash # in the dev container, like everything else ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2' # full tunnel ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh thinkpad 3 --split' # LAN only ``` It derives the server's public key from `wg-server-key.sops.yaml` itself, so there is no local key copy to keep in step with anything. Profiles land in `.cache/wg-peers/` — gitignored, and persistent because the repo is the container's only read-write mount. It generates the device's keypair, writes a `0600` client profile, prints a QR code, and prints the `[Peer]` stanza to add. It refuses to write a profile anywhere git does not ignore. Then: 1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the `# — , enrolled ` comment. `wg show` can only ever print the key, so that comment is the only record of which stanza belongs to which device. 2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a **content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name, which changes the pod template, which forces the roll. (It did **not** work this way at first — a fixed-name ConfigMap updates in place, the pod template never changes, and the init container keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled: ```bash ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \ -o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"' ``` The generation must have **increased** since before the merge. If it is still `1`, the roll did not happen and `wg show` will list no peer no matter how long you wait. 3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on Windows/macOS; `/etc/wireguard/.conf` + `wg-quick@` on Linux). 4. **Delete the profile from your machine**: `shred -u `. It is the only copy of that device's private key — the server never sees it and it cannot be re-derived. Until the PR is merged the device handshakes and gets silence, exactly like an unknown key. ## Step 6 — verify it end to end (from OFF the LAN) **This is the only honest test, and it cannot be run from inside the house.** Put the phone on cellular, with Wi-Fi off, and check all four — each proves a different hop: | # | Check | Proves | |---|---|---| | 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza | | 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path | | 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway admitting a tunnel client, which a pod-networked VPN would have 403'd. Expected source identity is `world`, like a LAN browser: measured 2026-11-09 on VIPs `.23` and `.24`, not yet observed on the Gateway's `.20` | | 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier | Then confirm server-side: ```bash ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show' # a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer. `ds/wireguard` # picks ONE pod; if it reads 0, ask the pod on the node holding the forward (see Troubleshooting). ``` **If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up. **If 1 and 2 pass but pages or large transfers hang**, suspect the client MTU; see Troubleshooting. **If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from the device, by design. `wg show` cannot tell them apart either: it reads the same in both cases. The node's conntrack table can; see Troubleshooting. ## Troubleshooting Ask the pod on **the node carrying the traffic**: the node the WAN forward names for `away`, `k3s-prod-1` for `home`. Sessions are per node, and the other two pods read `0`. ```bash ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress get pods -o wide' ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress exec -c wireguard -- wg show wg0 latest-handshakes' # likewise: `wg show wg0 endpoints`, `wg show wg0 transfer` ``` **Use only those three subcommands to diagnose.** Never `wg showconf`, `wg show wg0 private-key` or `wg show wg0 preshared-keys`: they print key material into your terminal and transcript. | output | reading | |---|---| | `latest-handshakes`: ` 0` | no handshake with this peer since the pod started | | `endpoints`: ` (none)` | the server has not accepted a packet from this peer since the pod started | | `transfer`: ` ` | bytes, as the server counts them | ### The app shows TX, no RX and no "Latest handshake" **Symptom.** The app's transmit counter climbs and receive stays at zero. On the server, `endpoints` is `(none)` and `latest-handshakes` is `0` for that peer. **Cause.** A key mismatch: the server's public key in the profile, or the device's public key in `peers.conf`. WireGuard drops an initiation it cannot authenticate and sends nothing back, so neither end logs an error. Measured 2026-11-09: a mistyped server public key in the phone's profile. **Check.** First prove the packets arrive. The pod has no `conntrack` binary, but in `hostNetwork` it can read the host's table: ```bash ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress exec -c wireguard -- grep dport=51820 /proc/net/nf_conntrack' ``` - A line whose first tuple is `src= ... dport=51820`, marked `[UNREPLIED]`: the datagrams reach the node and the server stays silent. It is the keys. - No such line: the packets are not arriving. Look at the router forward, the DNS name, or the device's network. Then compare the keys **by copy and paste, never by eye and never retyped**: the server's public half (the first command in Step 2's "Record it in your PASSWORD MANAGER") against the profile's `PublicKey`, and the device's public key from the app against its stanza in `peers.conf`. ### Works on Wi-Fi, stalls on mobile data, handshake healthy **Symptom.** `latest-handshakes` is recent and some traffic flows, but names or pages fail on mobile data. The same device is fine on Wi-Fi. **Cause.** Most likely a client MTU too large for the carrier's path: a tunnel MTU of `1420` means outer packets of up to 1480 bytes. This is inferred from one before-and-after, on one phone and one carrier (2026-11-09): `1420` failed and clearing the field fixed it. The path MTU itself was not measured. **Check.** In the Android WireGuard app, edit the tunnel, clear the `MTU` field or set it to `1280`, save, then turn the tunnel off and on. A blank field means `1280` in the app's default userspace backend (`GoBackend`: `getMtu().orElse(1280)`). On other clients set `MTU = 1280` in the profile. `new-peer.sh` now generates `MTU = 1280` unless `WG_MTU` overrides it. A profile already on a device does not change, so edit it by hand. If `1280` does not help, it is not the MTU: go back to Step 6's check 3. ### The `away` profile cannot connect from home **Symptom.** On the home network, `away` never handshakes. **Cause.** Its Endpoint is `wan.home.keithjackson.dev`, the house's WAN address, and the router has no hairpin NAT. WireGuard cannot use the Cloudflare Tunnel instead. **Check.** Turn Wi-Fi off. If `away` handshakes on mobile data, the profile is fine: use the `home` tunnel indoors, as described in [Phone profiles: one tunnel for home, one for away](../platform/vpn-ingress/README.md#phone-profiles-one-tunnel-for-home-one-for-away). ## Revoking a device Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the next reconcile. **Never re-use that address octet**: an old profile would silently start working again as a different device. ## Why there is no connection alert A WireGuard exporter would publish `wireguard_latest_handshake_seconds` per peer and make "something connected at 03:00" alertable. It needs a second `NET_ADMIN` container inside the pod that terminates the internet-facing tunnel, and that was not worth shipping unverified. **It is the top follow-up for this component.** Until then: * pod-down is covered generically by kube-prometheus-stack's `KubePodNotReady` / `KubeDeploymentReplicasMismatch`; * "who is connected" is an on-demand question — `wg show`, above; * **never** `wg showconf` — it prints the server's private key. ## Known limitations * **No IPv6 in the tunnel.** What a client's IPv6 does then depends on the client. In the Android app's default `GoBackend`, a config with one peer and a default route skips `allowFamily()`, so Android should block IPv6 while the tunnel is up and apps fall back to IPv4. That is read from upstream source, not observed on a device. A `--split` profile has no default route, so its IPv6 stays outside the tunnel. Expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside too (not tested). * **Client IPs are only partly preserved.** LAN hosts and the internet see the node's address (MASQUERADE). A static route for `10.90.0.0/24` → the forwarded node, added on the router, would preserve them there, at the cost of another hand-maintained off-repo route. LoadBalancer VIPs are different: Cilium handles the packet before MASQUERADE, so the source stays `10.90.0.x`, identity `world` (BPF conntrack on prod-1, 2026-11-09, VIPs `.23` and `.24`). Any `loadBalancerSourceRanges` must therefore list the tunnel subnet, as PiHole's do. * **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients re-handshake automatically (retry every 5s, rekey every 2 min).