home-network-k3s-impl/runbooks/vpn-remote-access.md

# vpn-remote-access — standing up the WireGuard remote-access VPN, and proving it

**Status: the in-cluster half is shipped; the VPN is INERT until three manual steps are done.**
All three are off-repo by necessity — a kernel module, a private key, and a router port-forward.
**Alert:** pod-down is covered generically (`KubePodNotReady` / `KubeDeploymentReplicasMismatch`).
There is **no peer-level alerting** — see "Why there is no connection alert". This is an **operator
procedure**, not an incident page.
**Files:** `platform/vpn-ingress/` (namespace, peer register, DaemonSet, baseline policy),
`clusters/base/vpn-ingress.yaml` (its Flux tier), `clusters/base/cluster-vars.yaml`
(`WG_CLIENT_CIDR`), `infra/ansible/roles/node_baseline/defaults/main.yml` (the `wireguard` module),
`infra/vpn-ingress/new-peer.sh` (device enrolment).
**Design rationale:** `platform/vpn-ingress/README.md` — read "Why `hostNetwork` is required"
before changing anything about how this is wired.
**Not to be confused with** `vpn-down` / `vpn-egress-stale`, which are about `platform/vpn-egress`
(pods *out* through PIA). This page is remote clients *in*.

## What this gives you

A device — Android, Linux, Windows, macOS, iOS — dials in over **udp/51820** from anywhere and
behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's web UIs, PiHole
DNS, and internet egress out the house WAN.

Authorisation is one thing only: the device's **public key is in
`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.

> **An enrolled device is as privileged as a machine sitting in the house** — including the three
> k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
> the entire local network" means. Revoke a lost device promptly: delete its stanza, PR, merge.

## Step 1 — load the `wireguard` module on every node (once)

The DaemonSet creates its `wg0` link in the host netns with `CAP_NET_ADMIN` only; it is
deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module. Without it
`wg-quick` fails outright and the pod CrashLoops.

```bash
# operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox,
# and ansible.cfg is only picked up from infra/ansible, so cd there inside the container.
./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
```

Three details in that command are load-bearing, not style:

- **`-e k3s_hosts=k3s_prod`.** `node-baseline.yml` targets `k3s_servers` by default, which is the
  **dev** group — and dev was decommissioned, so an unqualified run does nothing useful. This is
  the targeting pattern `infra/ansible/README.md` "Targeting prod" documents for every playbook.
- **`--tags system`, and never an untagged run.** The two kernel-module tasks carry the `system`
  tag. An untagged run would also execute the `disks` tasks, which assert on block-device variables
  prod has deliberately left as placeholders (`infra/PROD-PROVISIONING.md` §2) — the playbook's own
  header warns about this. `--tags system` is broader than the modules alone (it also re-asserts
  sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and
  already in its desired state on a live node.
- **It is a playbook, not a wrapper script.** There is no `site.sh` in this repo; `playbooks/site.yml`
  is the full bring-up chain and is the wrong tool here.

Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:

```bash
# expected: a `wireguard` line, and the modules-load.d file naming it for the next boot
lsmod | grep '^wireguard'
grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
```

## Step 2 — fill in the server key (once)

The key is **SOPS-encrypted in this tree** (R-SEC-01, spec docs/07), not created by hand:
`platform/vpn-ingress/wg-server-key.sops.yaml`. It is already committed — encrypted, with a
`REPLACE-ME` placeholder — and it is already listed in the component's `resources:` with a
`decryption:` patch in both overlays. **Nothing needs wiring. You are only replacing one value.**

Why in git rather than `kubectl create secret`: an out-of-band Secret puts the one credential every
enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and
losing it means re-enrolling every device, because each client profile pins this key's public half.

Everything below runs **inside the dev container** — `wg`, `sops` and `qrencode` are all baked into
the image (CLAUDE.md §5: the toolchain lives only in the pinned container; don't install it on the
host). One shell, one toolchain.

**1. Generate the key straight into the encrypted file.** One pipeline: `wg genkey` makes it,
`sops set` writes it into the already-encrypted document in place. The plaintext never lands on
disk and never reaches a terminal.

```bash
./infra/devcontainer/dev.sh run bash -c '
  cd /work
  umask 077
  sops set platform/vpn-ingress/wg-server-key.sops.yaml \
    "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
'
```

**2. Record it in your PASSWORD MANAGER.** Derive both halves from the file you just wrote:

```bash
# the PUBLIC half — safe to display, and what every client profile carries
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'

# the PRIVATE half — paste into the password manager, then clear your scrollback
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
  platform/vpn-ingress/wg-server-key.sops.yaml'
```

The second command is the one place this key is deliberately shown. The encrypted file is in git and
Velero captures this namespace to R2, so the password-manager copy is belt-and-braces — but it is
the only copy that survives losing the cluster *and* the backups, and the key cannot be regenerated
without invalidating every client profile.

**3. Confirm the value is a real key**, without printing it — if `wg pubkey` prints a key, the
value is valid; if it errors, it is still wrong:

```bash
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null && echo "valid WireGuard key"'
```

That is the same check the pod's init container runs, so a pass here predicts a pass there.

**4. PR it, and merge.** Flux rolls the DaemonSet and the pods leave CrashLoop.

If you would rather use an editor than `sops set`:
`./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml` (replace
`REPLACE-ME`, save, exit — it re-encrypts on write).

### What the pod tells you if this went wrong

The init container validates the value with `wg pubkey` before writing `wg0.conf`, and names the
specific failure in its log — so you never get a server that silently cannot complete a handshake:

| pod log says | meaning |
|---|---|
| `still the committed REPLACE-ME PLACEHOLDER` | step 3 not done (the expected fresh-install state) |
| `It is EMPTY` | the value was blanked |
| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |

### What a healthy start looks like

Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key:

```
wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820.
```

then the `wireguard` container brings the tunnel up:

```
**** Found WG conf /config/wg_confs/wg0.conf, adding to list ****
**** Activating tunnel /config/wg_confs/wg0.conf ****
[#] ip link add dev wg0 type wireguard
[#] wg addconf wg0 /dev/fd/63
[#] ip -4 address add 10.90.0.1/24 dev wg0
[#] ip link set mtu 1420 up dev wg0
```

**Ignore the `chown: ... Operation not permitted` run before that**, and the image's own
"Permissions could not be set ... we will not provide support for it". Capabilities are dropped to
`NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs
as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter.

**A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device`
means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it
died on. That is the shape of a real failure; the probe is doing its job of refusing to call a
server without a `wg0` "ready".

## Step 3 — forward udp/51820 on the router (once)

On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
`192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.

Pick any node — all three run an identical listener. Note which one you chose here:

```
# WAN udp/51820 -> 192.168.2.__  (k3s-prod-_)   set <date>
```

**Node loss = failover is this one line.** Re-point the forward at another node address; every
device keeps working with the same profile, because all three nodes answer the same keys.

Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
already silent to unauthenticated probes (README, "Why this is safe to expose").

## Step 4 — confirm the listener is healthy

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress get ds wireguard -o wide'
# expected: DESIRED 3  CURRENT 3  READY 3

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
# expected: `interface: wg0`, `listening port: 51820`, and (before step 5) NO peers.
```

A zero-peer interface that is Ready is the **correct** state before any device is enrolled: it
listens, and refuses every handshake because no key is trusted.

The readiness/liveness probe is `wg show wg0`, and that is load-bearing rather than cosmetic: a
port probe cannot distinguish "wg0 never came up" from "working", because WireGuard is silent to
unauthenticated traffic either way. If the pod is Ready, the interface exists.

## Step 5 — enrol a device

```bash
# in the dev container, like everything else
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2'            # full tunnel
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh thinkpad 3 --split'   # LAN only
```

It derives the server's public key from `wg-server-key.sops.yaml` itself, so there is no local key
copy to keep in step with anything. Profiles land in `.cache/wg-peers/` — gitignored, and persistent
because the repo is the container's only read-write mount.

It generates the device's keypair, writes a `0600` client profile, prints a QR code, and prints the
`[Peer]` stanza to add. It refuses to write a profile anywhere git does not ignore.

Then:

1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
   `# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
   comment is the only record of which stanza belongs to which device.
2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
   **content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
   which changes the pod template, which forces the roll. (It did **not** work this way at first —
   a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
   keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:

   ```bash
   ./infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
        -o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
   ```

   The generation must have **increased** since before the merge. If it is still `1`, the roll did
   not happen and `wg show` will list no peer no matter how long you wait.
3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
   Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
   device's private key — the server never sees it and it cannot be re-derived.

Until the PR is merged the device handshakes and gets silence, exactly like an unknown key.

## Step 6 — verify it end to end (from OFF the LAN)

**This is the only honest test, and it cannot be run from inside the house.** Put the phone on
cellular, with Wi-Fi off, and check all four — each proves a different hop:

| # | Check | Proves |
|---|---|---|
| 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza |
| 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path |
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd |
| 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier |

Then confirm server-side:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer.
```

**If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not
WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up.

**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60
bytes of WireGuard encapsulation); drop the client to `1280` and re-test.

**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside —
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
if the router is not delivering the packets, and a `handshake` with no completed session if it is.

## Triage — "the client reaches the LAN and cluster apps, but not the internet"

Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom
almost always lives, and because the server side has been measured and is not the cause.

**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.**

| the profile says | meaning |
|---|---|
| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. |
| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. |

`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit —
`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client
profile only**, so an already-issued profile keeps whatever it was rendered with until it is
re-rendered and re-imported on the device. The two forms differ by one flag:

```bash
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet>'           # full tunnel
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet> --split'   # LAN only
```

Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into
`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is
editing `AllowedIPs` in the profile **on the device** — the server does not care.

**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the
client connected, then watch the two counters that only forwarded traffic moves:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c "
     wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5;
     iptables -t nat -L POSTROUTING -n -v | head -5"'
```

* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which
  node the router's `udp/51820` forward points at (the router is unreadable from the cluster).
* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is
  upstream of the cluster (router forward, client, or carrier), not here.
* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side
  failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there,
  which breaks internet while node-local cluster VIPs still work. It has not been observed.

**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly:

* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack
  client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel —
  so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4:
  `curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it,
  and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is
  blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it).
* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified
  2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a
  blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on
  a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all.

**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the
`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain
**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route
`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the
PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled
`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the
**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6).

## Triage — which L2 announcement lease a node holds does NOT matter

Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node
could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`:
that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh`
for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer
VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend
is a node-local socket. Full measurements and the two misleading probes
(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`.

`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a
separate, pre-existing fault in that service and has nothing to do with the VPN.

## Revoking a device

Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
next reconcile. **Never re-use that address octet**: an old profile would silently start working
again as a different device.

## Why there is no connection alert

A WireGuard exporter would publish `wireguard_latest_handshake_seconds` per peer and make
"something connected at 03:00" alertable. It needs a second `NET_ADMIN` container inside the pod
that terminates the internet-facing tunnel, and that was not worth shipping unverified. **It is the
top follow-up for this component.** Until then:

* pod-down is covered generically by kube-prometheus-stack's `KubePodNotReady` /
  `KubeDeploymentReplicasMismatch`;
* "who is connected" is an on-demand question — `wg show`, above;
* **never** `wg showconf` — it prints the server's private key.

## Known limitations

* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside
  the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point.
* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's
  address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
  preserve them — at the cost of another hand-maintained off-repo route.
* **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients
  re-handshake automatically (retry every 5s, rekey every 2 min).