jakman85andClaude Opus 5
f5353cbd30
A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
holds itself: the announcer does not answer its own ARP request, so on the
lease holder the neighbour entry settles at FAILED and packets to that VIP die
before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes
x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced.
platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and
MASQUERADEs remote clients to the node's own address, so a VPN client takes
the node's LAN path and inherits its ARP failures. The router's udp/51820
forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16
Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering
(its VIP has two Services and k3s-prod-2 held the other lease). The tunnel
handshook, names resolved, every app hung, and nothing reported it.
on k3s-prod-1: ip neigh show 192.168.2.20 -> FAILED (its own VIP)
on prod-2/3: ip neigh show 192.168.2.20 -> lladdr ... (prod-1), 443 OK
Mark the node and exclude it from announcing:
* node_baseline labels the forwarded node (node_baseline_vpn_ingress_node,
k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes
the label from any node that no longer matches so moving it MOVES the label.
* l2-announcement-policy.yaml excludes that label, so the node ARP-resolves
every VIP. Fixes all VIPs for VPN clients, not just the one that collided.
Cost: announcement candidacy drops from 3 nodes to 2.
* VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels
(label added to the kube-state-metrics allowlist), with
VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches,
so a missing label would silently re-arm the whole failure AND leave the
critical alert inert. Four promtool cases, including that inertness.
A node LABEL rather than a cluster-vars knob because the alert must read it
too, and extract_rules.py rightly refuses a ${...} placeholder in an alert
expr -- the rule unit-tested would not be the rule deployed. One source of
truth, and it touches no sensitive path.
Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and
"failover is this one line". Both false; the first caused this.
UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the
handoff entry's third section.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
346 lines
19 KiB
Markdown
346 lines
19 KiB
Markdown
# vpn-remote-access — standing up the WireGuard remote-access VPN, and proving it
|
||
|
||
**Status: the in-cluster half is shipped; the VPN is INERT until three manual steps are done.**
|
||
All three are off-repo by necessity — a kernel module, a private key, and a router port-forward.
|
||
**Alert:** pod-down is covered generically (`KubePodNotReady` / `KubeDeploymentReplicasMismatch`).
|
||
There is **no peer-level alerting** — see "Why there is no connection alert". This is an **operator
|
||
procedure**, not an incident page.
|
||
**Files:** `platform/vpn-ingress/` (namespace, peer register, DaemonSet, baseline policy),
|
||
`clusters/base/vpn-ingress.yaml` (its Flux tier), `clusters/base/cluster-vars.yaml`
|
||
(`WG_CLIENT_CIDR`), `infra/ansible/roles/node_baseline/defaults/main.yml` (the `wireguard` module),
|
||
`infra/vpn-ingress/new-peer.sh` (device enrolment).
|
||
**Design rationale:** `platform/vpn-ingress/README.md` — read "Why `hostNetwork` is required"
|
||
before changing anything about how this is wired.
|
||
**Not to be confused with** `vpn-down` / `vpn-egress-stale`, which are about `platform/vpn-egress`
|
||
(pods *out* through PIA). This page is remote clients *in*.
|
||
|
||
## What this gives you
|
||
|
||
A device — Android, Linux, Windows, macOS, iOS — dials in over **udp/51820** from anywhere and
|
||
behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's web UIs, PiHole
|
||
DNS, and internet egress out the house WAN.
|
||
|
||
Authorisation is one thing only: the device's **public key is in
|
||
`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.
|
||
|
||
> **An enrolled device is as privileged as a machine sitting in the house** — including the three
|
||
> k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
|
||
> the entire local network" means. Revoke a lost device promptly: delete its stanza, PR, merge.
|
||
|
||
## Step 1 — load the `wireguard` module on every node (once)
|
||
|
||
The DaemonSet creates its `wg0` link in the host netns with `CAP_NET_ADMIN` only; it is
|
||
deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module. Without it
|
||
`wg-quick` fails outright and the pod CrashLoops.
|
||
|
||
```bash
|
||
# operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox,
|
||
# and ansible.cfg is only picked up from infra/ansible, so cd there inside the container.
|
||
./infra/devcontainer/dev.sh run bash -c \
|
||
'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
|
||
```
|
||
|
||
Three details in that command are load-bearing, not style:
|
||
|
||
- **`-e k3s_hosts=k3s_prod`.** `node-baseline.yml` targets `k3s_servers` by default, which is the
|
||
**dev** group — and dev was decommissioned, so an unqualified run does nothing useful. This is
|
||
the targeting pattern `infra/ansible/README.md` "Targeting prod" documents for every playbook.
|
||
- **`--tags system`, and never an untagged run.** The two kernel-module tasks carry the `system`
|
||
tag. An untagged run would also execute the `disks` tasks, which assert on block-device variables
|
||
prod has deliberately left as placeholders (`infra/PROD-PROVISIONING.md` §2) — the playbook's own
|
||
header warns about this. `--tags system` is broader than the modules alone (it also re-asserts
|
||
sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and
|
||
already in its desired state on a live node.
|
||
- **It is a playbook, not a wrapper script.** There is no `site.sh` in this repo; `playbooks/site.yml`
|
||
is the full bring-up chain and is the wrong tool here.
|
||
|
||
Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:
|
||
|
||
```bash
|
||
# expected: a `wireguard` line, and the modules-load.d file naming it for the next boot
|
||
lsmod | grep '^wireguard'
|
||
grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
|
||
```
|
||
|
||
## Step 2 — fill in the server key (once)
|
||
|
||
The key is **SOPS-encrypted in this tree** (R-SEC-01, spec docs/07), not created by hand:
|
||
`platform/vpn-ingress/wg-server-key.sops.yaml`. It is already committed — encrypted, with a
|
||
`REPLACE-ME` placeholder — and it is already listed in the component's `resources:` with a
|
||
`decryption:` patch in both overlays. **Nothing needs wiring. You are only replacing one value.**
|
||
|
||
Why in git rather than `kubectl create secret`: an out-of-band Secret puts the one credential every
|
||
enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and
|
||
losing it means re-enrolling every device, because each client profile pins this key's public half.
|
||
|
||
Everything below runs **inside the dev container** — `wg`, `sops` and `qrencode` are all baked into
|
||
the image (CLAUDE.md §5: the toolchain lives only in the pinned container; don't install it on the
|
||
host). One shell, one toolchain.
|
||
|
||
**1. Generate the key straight into the encrypted file.** One pipeline: `wg genkey` makes it,
|
||
`sops set` writes it into the already-encrypted document in place. The plaintext never lands on
|
||
disk and never reaches a terminal.
|
||
|
||
```bash
|
||
./infra/devcontainer/dev.sh run bash -c '
|
||
cd /work
|
||
umask 077
|
||
sops set platform/vpn-ingress/wg-server-key.sops.yaml \
|
||
"[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
|
||
'
|
||
```
|
||
|
||
**2. Record it in your PASSWORD MANAGER.** Derive both halves from the file you just wrote:
|
||
|
||
```bash
|
||
# the PUBLIC half — safe to display, and what every client profile carries
|
||
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
|
||
platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
|
||
|
||
# the PRIVATE half — paste into the password manager, then clear your scrollback
|
||
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
|
||
platform/vpn-ingress/wg-server-key.sops.yaml'
|
||
```
|
||
|
||
The second command is the one place this key is deliberately shown. The encrypted file is in git and
|
||
Velero captures this namespace to R2, so the password-manager copy is belt-and-braces — but it is
|
||
the only copy that survives losing the cluster *and* the backups, and the key cannot be regenerated
|
||
without invalidating every client profile.
|
||
|
||
**3. Confirm the value is a real key**, without printing it — if `wg pubkey` prints a key, the
|
||
value is valid; if it errors, it is still wrong:
|
||
|
||
```bash
|
||
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
|
||
platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null && echo "valid WireGuard key"'
|
||
```
|
||
|
||
That is the same check the pod's init container runs, so a pass here predicts a pass there.
|
||
|
||
**4. PR it, and merge.** Flux rolls the DaemonSet and the pods leave CrashLoop.
|
||
|
||
If you would rather use an editor than `sops set`:
|
||
`./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml` (replace
|
||
`REPLACE-ME`, save, exit — it re-encrypts on write).
|
||
|
||
### What the pod tells you if this went wrong
|
||
|
||
The init container validates the value with `wg pubkey` before writing `wg0.conf`, and names the
|
||
specific failure in its log — so you never get a server that silently cannot complete a handshake:
|
||
|
||
| pod log says | meaning |
|
||
|---|---|
|
||
| `still the committed REPLACE-ME PLACEHOLDER` | step 3 not done (the expected fresh-install state) |
|
||
| `It is EMPTY` | the value was blanked |
|
||
| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
|
||
| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
|
||
| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
|
||
| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |
|
||
|
||
### What a healthy start looks like
|
||
|
||
Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key:
|
||
|
||
```
|
||
wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820.
|
||
```
|
||
|
||
then the `wireguard` container brings the tunnel up:
|
||
|
||
```
|
||
**** Found WG conf /config/wg_confs/wg0.conf, adding to list ****
|
||
**** Activating tunnel /config/wg_confs/wg0.conf ****
|
||
[#] ip link add dev wg0 type wireguard
|
||
[#] wg addconf wg0 /dev/fd/63
|
||
[#] ip -4 address add 10.90.0.1/24 dev wg0
|
||
[#] ip link set mtu 1420 up dev wg0
|
||
```
|
||
|
||
**Ignore the `chown: ... Operation not permitted` run before that**, and the image's own
|
||
"Permissions could not be set ... we will not provide support for it". Capabilities are dropped to
|
||
`NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs
|
||
as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter.
|
||
|
||
**A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device`
|
||
means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it
|
||
died on. That is the shape of a real failure; the probe is doing its job of refusing to call a
|
||
server without a `wg0` "ready".
|
||
|
||
## Step 3 — forward udp/51820 on the router (once)
|
||
|
||
On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
|
||
`192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.
|
||
|
||
> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node.
|
||
>
|
||
> This step used to say "pick any node — all three run an identical listener". **That sentence
|
||
> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a
|
||
> cluster-vars knob instead of a free choice here.
|
||
>
|
||
> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
|
||
> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease
|
||
> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they
|
||
> leave the node. Measured on prod 2026-10-08:
|
||
>
|
||
> ```text
|
||
> on k3s-prod-1 (held the shared Gateway lease): ip neigh show 192.168.2.20 -> FAILED
|
||
> on k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr … (prod-1)
|
||
> ```
|
||
>
|
||
> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN
|
||
> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway
|
||
> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so
|
||
> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported
|
||
> it.
|
||
>
|
||
`infra/ansible/roles/node_baseline` now labels the forwarded node
|
||
> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in
|
||
> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes
|
||
> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI
|
||
> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a
|
||
> labelled node taking a lease, but it CANNOT catch the forward being moved without the label.
|
||
|
||
Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`).
|
||
Note it here:
|
||
|
||
```
|
||
# WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set <date>
|
||
# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml
|
||
```
|
||
|
||
**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that
|
||
was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and
|
||
reintroduces the failure above against a different set of addresses (prod-2 announces crafty and
|
||
unifi; prod-3 announces plex). Do both, in this order:
|
||
|
||
1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role
|
||
(`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium
|
||
then moves the leases off the newly-labelled node and onto its peers.
|
||
2. **Re-point the router** forward at the new node's address.
|
||
|
||
Every device keeps working with the same profile either way; all three nodes answer the same keys,
|
||
and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that
|
||
handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because
|
||
that alert watches the node the knob names, not the node your router actually points at.
|
||
|
||
Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
|
||
already silent to unauthenticated probes (README, "Why this is safe to expose").
|
||
|
||
## Step 4 — confirm the listener is healthy
|
||
|
||
```bash
|
||
./infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n vpn-ingress get ds wireguard -o wide'
|
||
# expected: DESIRED 3 CURRENT 3 READY 3
|
||
|
||
./infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
|
||
# expected: `interface: wg0`, `listening port: 51820`, and (before step 5) NO peers.
|
||
```
|
||
|
||
A zero-peer interface that is Ready is the **correct** state before any device is enrolled: it
|
||
listens, and refuses every handshake because no key is trusted.
|
||
|
||
The readiness/liveness probe is `wg show wg0`, and that is load-bearing rather than cosmetic: a
|
||
port probe cannot distinguish "wg0 never came up" from "working", because WireGuard is silent to
|
||
unauthenticated traffic either way. If the pod is Ready, the interface exists.
|
||
|
||
## Step 5 — enrol a device
|
||
|
||
```bash
|
||
# in the dev container, like everything else
|
||
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2' # full tunnel
|
||
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh thinkpad 3 --split' # LAN only
|
||
```
|
||
|
||
It derives the server's public key from `wg-server-key.sops.yaml` itself, so there is no local key
|
||
copy to keep in step with anything. Profiles land in `.cache/wg-peers/` — gitignored, and persistent
|
||
because the repo is the container's only read-write mount.
|
||
|
||
It generates the device's keypair, writes a `0600` client profile, prints a QR code, and prints the
|
||
`[Peer]` stanza to add. It refuses to write a profile anywhere git does not ignore.
|
||
|
||
Then:
|
||
|
||
1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
|
||
`# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
|
||
comment is the only record of which stanza belongs to which device.
|
||
2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
|
||
**content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
|
||
which changes the pod template, which forces the roll. (It did **not** work this way at first —
|
||
a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
|
||
keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
|
||
|
||
```bash
|
||
./infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
|
||
-o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
|
||
```
|
||
|
||
The generation must have **increased** since before the merge. If it is still `1`, the roll did
|
||
not happen and `wg show` will list no peer no matter how long you wait.
|
||
3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
|
||
Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
|
||
4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
|
||
device's private key — the server never sees it and it cannot be re-derived.
|
||
|
||
Until the PR is merged the device handshakes and gets silence, exactly like an unknown key.
|
||
|
||
## Step 6 — verify it end to end (from OFF the LAN)
|
||
|
||
**This is the only honest test, and it cannot be run from inside the house.** Put the phone on
|
||
cellular, with Wi-Fi off, and check all four — each proves a different hop:
|
||
|
||
| # | Check | Proves |
|
||
|---|---|---|
|
||
| 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza |
|
||
| 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path |
|
||
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd |
|
||
| 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier |
|
||
|
||
Then confirm server-side:
|
||
|
||
```bash
|
||
./infra/devcontainer/dev.sh run bash -c \
|
||
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
|
||
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer.
|
||
```
|
||
|
||
**If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not
|
||
WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up.
|
||
|
||
**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60
|
||
bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
|
||
|
||
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside —
|
||
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
|
||
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
|
||
|
||
## Revoking a device
|
||
|
||
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
|
||
next reconcile. **Never re-use that address octet**: an old profile would silently start working
|
||
again as a different device.
|
||
|
||
## Why there is no connection alert
|
||
|
||
A WireGuard exporter would publish `wireguard_latest_handshake_seconds` per peer and make
|
||
"something connected at 03:00" alertable. It needs a second `NET_ADMIN` container inside the pod
|
||
that terminates the internet-facing tunnel, and that was not worth shipping unverified. **It is the
|
||
top follow-up for this component.** Until then:
|
||
|
||
* pod-down is covered generically by kube-prometheus-stack's `KubePodNotReady` /
|
||
`KubeDeploymentReplicasMismatch`;
|
||
* "who is connected" is an on-demand question — `wg show`, above;
|
||
* **never** `wg showconf` — it prints the server's private key.
|
||
|
||
## Known limitations
|
||
|
||
* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside
|
||
the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point.
|
||
* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's
|
||
address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
|
||
preserve them — at the cost of another hand-maintained off-repo route.
|
||
* **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients
|
||
re-handshake automatically (retry every 5s, rekey every 2 min).
|