jakman85andClaude Opus 5
8059297036
Operator correction: "sop, wg, and any other tool requirement is supposed to be
baked into the devcontainer."
My previous commit discovered that `wg` existed only on the host, `sops` only in
the container, and `qrencode` on neither -- and fixed it the wrong way, by
splitting every step of the key procedure across the boundary and telling the
operator to `dnf install qrencode` on their workstation. That breaks CLAUDE.md §5
and the devcontainer README's "Don't install the toolchain on the host", produces
a procedure that cannot be reproduced in CI, and is miserable to follow.
wireguard-tools + qrencode are now distro packages in the Containerfile. Both are
in Fedora 44's repos (wireguard-tools 1.0.20260223 -- the same upstream version as
the image this component runs -- and qrencode 4.1.1), unpinned like every other
dnf package there: the pinned thing is the Fedora base, and neither is a gate tool
whose version changes what green means.
Three things got SIMPLER rather than merely relocated, which is the tell that the
rule was right:
* Step 2 is one pipeline in one shell -- `sops set ... "$(wg genkey)"` writes the
key straight into the already-encrypted document. No staged copy in .cache, no
`*.key`-suffix warning, no sha256 comparison to prove the right bytes landed.
* new-peer.sh derives the server public key from wg-server-key.sops.yaml itself
(`sops -d --extract | wg pubkey`), so there is no local key copy to keep in
step with the encrypted one, and no magic host path it must live at.
* `.cache/` is gitignored at the repo root (mirroring infra/ansible/.gitignore),
giving the container a persistent non-committable scratch dir. new-peer.sh
writes profiles to .cache/wg-peers/ -- necessary because the repo bind-mount is
the container's only persistent read-write mount, so a profile in /tmp there is
gone before it can reach a desktop. Its blanket "refuse to write inside the
checkout" guard is now a `git check-ignore` test of the real property.
Verified by execution, not reasoning: the image rebuilds; wg/qrencode/sops all
resolve in it; `sops -d | wg pubkey` runs in one shell; new-peer.sh refuses the
REPLACE-ME placeholder with an actionable message; and a full run with a throwaway
server key renders a correct profile at 0600 under a 0700 gitignored dir, prints
the QR, and refuses a non-ignored destination inside the repo.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
259 lines
14 KiB
Markdown
259 lines
14 KiB
Markdown
# vpn-remote-access — standing up the WireGuard remote-access VPN, and proving it
|
||
|
||
**Status: the in-cluster half is shipped; the VPN is INERT until three manual steps are done.**
|
||
All three are off-repo by necessity — a kernel module, a private key, and a router port-forward.
|
||
**Alert:** pod-down is covered generically (`KubePodNotReady` / `KubeDeploymentReplicasMismatch`).
|
||
There is **no peer-level alerting** — see "Why there is no connection alert". This is an **operator
|
||
procedure**, not an incident page.
|
||
**Files:** `platform/vpn-ingress/` (namespace, peer register, DaemonSet, baseline policy),
|
||
`clusters/base/vpn-ingress.yaml` (its Flux tier), `clusters/base/cluster-vars.yaml`
|
||
(`WG_CLIENT_CIDR`), `infra/ansible/roles/node_baseline/defaults/main.yml` (the `wireguard` module),
|
||
`infra/vpn-ingress/new-peer.sh` (device enrolment).
|
||
**Design rationale:** `platform/vpn-ingress/README.md` — read "Why `hostNetwork` is required"
|
||
before changing anything about how this is wired.
|
||
**Not to be confused with** `vpn-down` / `vpn-egress-stale`, which are about `platform/vpn-egress`
|
||
(pods *out* through PIA). This page is remote clients *in*.
|
||
|
||
## What this gives you
|
||
|
||
A device — Android, Linux, Windows, macOS, iOS — dials in over **udp/51820** from anywhere and
|
||
behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's web UIs, PiHole
|
||
DNS, and internet egress out the house WAN.
|
||
|
||
Authorisation is one thing only: the device's **public key is in
|
||
`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
|
||
|
||
> **An enrolled device is as privileged as a machine sitting in the house** — including the three
|
||
> k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
|
||
> the entire local network" means. Revoke a lost device promptly: delete its stanza, PR, merge.
|
||
|
||
## Step 1 — load the `wireguard` module on every node (once)
|
||
|
||
The DaemonSet creates its `wg0` link in the host netns with `CAP_NET_ADMIN` only; it is
|
||
deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module. Without it
|
||
`wg-quick` fails outright and the pod CrashLoops.
|
||
|
||
```bash
|
||
# operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox,
|
||
# and ansible.cfg is only picked up from infra/ansible, so cd there inside the container.
|
||
./infra/devcontainer/dev.sh run bash -c \
|
||
'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
|
||
```
|
||
|
||
Three details in that command are load-bearing, not style:
|
||
|
||
- **`-e k3s_hosts=k3s_prod`.** `node-baseline.yml` targets `k3s_servers` by default, which is the
|
||
**dev** group — and dev was decommissioned, so an unqualified run does nothing useful. This is
|
||
the targeting pattern `infra/ansible/README.md` "Targeting prod" documents for every playbook.
|
||
- **`--tags system`, and never an untagged run.** The two kernel-module tasks carry the `system`
|
||
tag. An untagged run would also execute the `disks` tasks, which assert on block-device variables
|
||
prod has deliberately left as placeholders (`infra/PROD-PROVISIONING.md` §2) — the playbook's own
|
||
header warns about this. `--tags system` is broader than the modules alone (it also re-asserts
|
||
sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and
|
||
already in its desired state on a live node.
|
||
- **It is a playbook, not a wrapper script.** There is no `site.sh` in this repo; `playbooks/site.yml`
|
||
is the full bring-up chain and is the wrong tool here.
|
||
|
||
Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:
|
||
|
||
```bash
|
||
# expected: a `wireguard` line, and the modules-load.d file naming it for the next boot
|
||
lsmod | grep '^wireguard'
|
||
grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
|
||
```
|
||
|
||
## Step 2 — fill in the server key (once)
|
||
|
||
The key is **SOPS-encrypted in this tree** (R-SEC-01, spec docs/07), not created by hand:
|
||
`platform/vpn-ingress/wg-server-key.sops.yaml`. It is already committed — encrypted, with a
|
||
`REPLACE-ME` placeholder — and it is already listed in the component's `resources:` with a
|
||
`decryption:` patch in both overlays. **Nothing needs wiring. You are only replacing one value.**
|
||
|
||
Why in git rather than `kubectl create secret`: an out-of-band Secret puts the one credential every
|
||
enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and
|
||
losing it means re-enrolling every device, because each client profile pins this key's public half.
|
||
|
||
Everything below runs **inside the dev container** — `wg`, `sops` and `qrencode` are all baked into
|
||
the image (CLAUDE.md §5: the toolchain lives only in the pinned container; don't install it on the
|
||
host). One shell, one toolchain.
|
||
|
||
**1. Generate the key straight into the encrypted file.** One pipeline: `wg genkey` makes it,
|
||
`sops set` writes it into the already-encrypted document in place. The plaintext never lands on
|
||
disk and never reaches a terminal.
|
||
|
||
```bash
|
||
./infra/devcontainer/dev.sh run bash -c '
|
||
cd /work
|
||
umask 077
|
||
sops set platform/vpn-ingress/wg-server-key.sops.yaml \
|
||
"[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
|
||
'
|
||
```
|
||
|
||
**2. Record it in your PASSWORD MANAGER.** Derive both halves from the file you just wrote:
|
||
|
||
```bash
|
||
# the PUBLIC half — safe to display, and what every client profile carries
|
||
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
|
||
platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
|
||
|
||
# the PRIVATE half — paste into the password manager, then clear your scrollback
|
||
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
|
||
platform/vpn-ingress/wg-server-key.sops.yaml'
|
||
```
|
||
|
||
The second command is the one place this key is deliberately shown. The encrypted file is in git and
|
||
Velero captures this namespace to R2, so the password-manager copy is belt-and-braces — but it is
|
||
the only copy that survives losing the cluster *and* the backups, and the key cannot be regenerated
|
||
without invalidating every client profile.
|
||
|
||
**3. Confirm the value is a real key**, without printing it — if `wg pubkey` prints a key, the
|
||
value is valid; if it errors, it is still wrong:
|
||
|
||
```bash
|
||
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
|
||
platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null && echo "valid WireGuard key"'
|
||
```
|
||
|
||
That is the same check the pod's init container runs, so a pass here predicts a pass there.
|
||
|
||
**4. PR it, and merge.** Flux rolls the DaemonSet and the pods leave CrashLoop.
|
||
|
||
If you would rather use an editor than `sops set`:
|
||
`./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml` (replace
|
||
`REPLACE-ME`, save, exit — it re-encrypts on write).
|
||
|
||
### What the pod tells you if this went wrong
|
||
|
||
The init container validates the value with `wg pubkey` before writing `wg0.conf`, and names the
|
||
specific failure in its log — so you never get a server that silently cannot complete a handshake:
|
||
|
||
| pod log says | meaning |
|
||
|---|---|
|
||
| `still the committed REPLACE-ME PLACEHOLDER` | step 3 not done (the expected fresh-install state) |
|
||
| `It is EMPTY` | the value was blanked |
|
||
| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
|
||
| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
|
||
|
||
## Step 3 — forward udp/51820 on the router (once)
|
||
|
||
On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
|
||
`192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.
|
||
|
||
Pick any node — all three run an identical listener. Note which one you chose here:
|
||
|
||
```
|
||
# WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set <date>
|
||
```
|
||
|
||
**Node loss = failover is this one line.** Re-point the forward at another node address; every
|
||
device keeps working with the same profile, because all three nodes answer the same keys.
|
||
|
||
Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
|
||
already silent to unauthenticated probes (README, "Why this is safe to expose").
|
||
|
||
## Step 4 — confirm the listener is healthy
|
||
|
||
```bash
|
||
./infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n vpn-ingress get ds wireguard -o wide'
|
||
# expected: DESIRED 3 CURRENT 3 READY 3
|
||
|
||
./infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
|
||
# expected: `interface: wg0`, `listening port: 51820`, and (before step 5) NO peers.
|
||
```
|
||
|
||
A zero-peer interface that is Ready is the **correct** state before any device is enrolled: it
|
||
listens, and refuses every handshake because no key is trusted.
|
||
|
||
The readiness/liveness probe is `wg show wg0`, and that is load-bearing rather than cosmetic: a
|
||
port probe cannot distinguish "wg0 never came up" from "working", because WireGuard is silent to
|
||
unauthenticated traffic either way. If the pod is Ready, the interface exists.
|
||
|
||
## Step 5 — enrol a device
|
||
|
||
```bash
|
||
# in the dev container, like everything else
|
||
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2' # full tunnel
|
||
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh thinkpad 3 --split' # LAN only
|
||
```
|
||
|
||
It derives the server's public key from `wg-server-key.sops.yaml` itself, so there is no local key
|
||
copy to keep in step with anything. Profiles land in `.cache/wg-peers/` — gitignored, and persistent
|
||
because the repo is the container's only read-write mount.
|
||
|
||
It generates the device's keypair, writes a `0600` client profile, prints a QR code, and prints the
|
||
`[Peer]` stanza to add. It refuses to write a profile anywhere git does not ignore.
|
||
|
||
Then:
|
||
|
||
1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
|
||
`# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
|
||
comment is the only record of which stanza belongs to which device.
|
||
2. PR → merge. Flux rolls the DaemonSet (~1 min).
|
||
3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
|
||
Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
|
||
4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
|
||
device's private key — the server never sees it and it cannot be re-derived.
|
||
|
||
Until the PR is merged the device handshakes and gets silence, exactly like an unknown key.
|
||
|
||
## Step 6 — verify it end to end (from OFF the LAN)
|
||
|
||
**This is the only honest test, and it cannot be run from inside the house.** Put the phone on
|
||
cellular, with Wi-Fi off, and check all four — each proves a different hop:
|
||
|
||
| # | Check | Proves |
|
||
|---|---|---|
|
||
| 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza |
|
||
| 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path |
|
||
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd |
|
||
| 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier |
|
||
|
||
Then confirm server-side:
|
||
|
||
```bash
|
||
./infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
|
||
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer.
|
||
```
|
||
|
||
**If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not
|
||
WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up.
|
||
|
||
**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60
|
||
bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
|
||
|
||
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside —
|
||
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
|
||
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
|
||
|
||
## Revoking a device
|
||
|
||
Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
|
||
next reconcile. **Never re-use that address octet**: an old profile would silently start working
|
||
again as a different device.
|
||
|
||
## Why there is no connection alert
|
||
|
||
A WireGuard exporter would publish `wireguard_latest_handshake_seconds` per peer and make
|
||
"something connected at 03:00" alertable. It needs a second `NET_ADMIN` container inside the pod
|
||
that terminates the internet-facing tunnel, and that was not worth shipping unverified. **It is the
|
||
top follow-up for this component.** Until then:
|
||
|
||
* pod-down is covered generically by kube-prometheus-stack's `KubePodNotReady` /
|
||
`KubeDeploymentReplicasMismatch`;
|
||
* "who is connected" is an on-demand question — `wg show`, above;
|
||
* **never** `wg showconf` — it prints the server's private key.
|
||
|
||
## Known limitations
|
||
|
||
* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside
|
||
the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point.
|
||
* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's
|
||
address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
|
||
preserve them — at the cost of another hand-maintained off-repo route.
|
||
* **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients
|
||
re-handshake automatically (retry every 5s, rekey every 2 min).
|