home-network-k3s-impl/decisions/2026-10-05-vpn-remote-access-wireguard.md

date, branch, title

date	branch	title
2026-10-05	feat/vpn-remote-access-wireguard	Remote-access VPN — WireGuard on udp/51820, hostNetwork because a pod identity would 403 every Gateway hostname

What changed

New platform component platform/vpn-ingress/: a WireGuard remote-access VPN so the operator's Android / Linux / Windows / macOS devices can dial in from anywhere on the internet and behave like machines on the LAN — full reach to 192.168.2.0/24 plus internet egress out the house WAN.

This is the opposite direction to the existing platform/vpn-egress (pods out through PIA, ADR-0015). They share nothing but the word "VPN"; both READMEs now say so, and README.md's layout block labels the two.

Transport: WireGuard, and the security argument is the handshake's failure mode, not "it is encrypted". WireGuard's first handshake message carries a MAC keyed with the responder's own public key, so a sender that does not already know it gets no reply at all — no error, no reset, no banner. From the internet, udp/51820 is indistinguishable from a closed port: nothing to brute-force, no version string, no amplification. That is what makes an unfiltered WAN port defensible here, and it is why there is deliberately no loadBalancerSourceRanges-style allowlist (which would also break roaming clients). It also has first-party clients on all four named platforms, which OpenVPN (needs a CA + per-device certs + a CRL) and IKEv2 do not.

hostNetwork: true, and this is THE decision — forced by a measured property of this cluster. A pod-networked WireGuard server must SNAT client traffic to its own pod IP (Cilium drops a packet whose source is not the endpoint's address), so every remote client would have arrived wearing that pod's security identity. decisions/2026-10-04-plex-advertise-plexdirect.md measured what that costs: a request to the shared Gateway VIP from inside a pod returns 403 Access denied from Envoy for all 16 Gateway hostnames, because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy, and every app admits reserved:ingress plus named peers — never an arbitrary pod. So the obvious build would have produced a tunnel that came up healthy, showed connected peers, routed packets, and 403'd every cluster web UI — and *.home.keithjackson.dev would not even have resolved, because apps/wave-0/pihole-dns.yaml's ingress policy deliberately omits the cluster entity. In the host netns there is no pod identity to carry: forwarded traffic is SNATed to the node address and arrives as host/remote-node, byte-for-byte what a LAN client looks like. Three further problems vanish with it: ip_forward is already 1 on these nodes (so no --allowed-unsafe-sysctls kubelet change — it is not a safe sysctl); no LoadBalancer VIP, so no L2-announcement lease lottery and no externalTrafficPolicy trade-off; and no double encapsulation, which is what makes MTU = 1420 simply correct.

hostNetwork is permitted by restrict-host-namespaces' own stated carve-out (it matches app-namespace=true only, and names Cilium/node-exporter as the precedent); namespace.yaml is deliberately unlabelled. The cost is stated, not buried: a hostNetwork pod has no Cilium endpoint, so no NetworkPolicy constrains this workload. networkpolicy.yaml says that at length rather than shipping an allow world → 51820 rule that would read as the control and enforce nothing; the real boundary is WireGuard's peer authentication.

A DaemonSet, not a Deployment. The WAN forward names one node address by hand, off-repo (same shape as the plex tcp/32400 forward). A floating single replica would silently invalidate it on every reschedule; an identical listener on all three nodes makes failover a one-line router change with no cluster work and no re-issued client profiles. A pinned LoadBalancer VIP was the considered alternative and was rejected: it reintroduces the L2 lease, SNAT and an extra node hop into the one path whose failure mode is "I am away from home and cannot get in".

Authorisation is split so that it stays reviewable. Peer public keys live in platform/vpn-ingress/peers.yaml in plaintext — a public key grants nothing by itself, and this makes "who can get into my house" a PR diff with git log as its audit trail. The only real secret, the server's private key, is the wg-server-key Secret, created out of band by the operator: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key that passed through a session transcript would have to be treated as compromised. The two halves are joined only at pod start, by an init container, into a medium: Memory emptyDir — so the assembled config holding the key never touches a node's disk and dies with the pod.

There are no peers yet, and that is the correct initial state, for the same reason. The VPN ships INERT-BUT-HEALTHY: a zero-peer interface listens and refuses every handshake.

Files:

• platform/vpn-ingress/ (new): namespace.yaml, peers.yaml (the plaintext peer register), daemonset.yaml (hostNetwork listener + config assembly), networkpolicy.yaml (namespace baseline, with the honest caveat), kustomization.yaml, README.md (the design record).
• clusters/base/vpn-ingress.yaml (new): its own non-gating post-platform Flux Kustomization (dependsOn platform-config, wait:false). Non-gating is load-bearing rather than conventional here: the DaemonSet cannot start until the operator creates the server key, so in the gating platform tier an unfinished manual step would have stalled every apps wave on merge.
• clusters/base/kustomization.yaml: the new tier added.
• clusters/base/cluster-vars.yaml: new WG_CLIENT_CIDR (10.90.0.0/24), declared next to LAN_CIDR because it is the one range that must be collision-checked against every other; the comment enumerates the five it avoids. No prod override, deliberately — unlike every VIP, this range is private to the tunnel, so the same value is correct in both environments and a prod patch would imply otherwise. Substitution inventory updated.
• infra/ansible/roles/node_baseline/defaults/main.yml: wireguard added to node_baseline_kernel_modules. The pod has NET_ADMIN but deliberately not SYS_MODULE, so it cannot autoload the module — it CrashLoops with a clear error rather than quietly degrading to userspace wireguard-go (which would need /dev/net/tun, a device a hostPath mount cannot actually grant, as the /dev/dri attempt proved).
• infra/vpn-ingress/new-peer.sh (new): enrols a device. Generates the keypair on the operator's machine, writes a 0600 profile outside the checkout (it refuses to write into the repo), prints a QR code for phones, and prints only the public [Peer] stanza. It also refuses a duplicate address, because two devices on one address does not error anywhere — the server routes to whichever handshook last, so both appear to work intermittently.
• runbooks/vpn-remote-access.md (new) + its kustomization.yaml generator entry + README.md index line: the three manual steps, the four off-LAN verification checks, revocation, and why no connection alert exists.
• README.md: vpn-ingress added to the layout, with the egress/ingress distinction spelled out.

THREE MANUAL STEPS REMAIN AND THE VPN DOES NOT WORK UNTIL ALL THREE ARE DONE (all off-repo by necessity; runbooks/vpn-remote-access.md): run node_baseline to load the kernel module, create the wg-server-key Secret, and forward WAN udp/51820 to one node address on the SOHO router.

SENSITIVE PATHS — this PR touches clusters/** (three files), so ci/approval-policy.sh will BLOCK it. It needs the operator's /approve-override <reason> posted as a PR review of type "Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own PR, and the agent account holds owner, so the prohibition is a rule, not a limit.

What I verified

Live prod, read-only, 2026-10-05, plus the offline gate.

• Toolbox gate (./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'): rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass. The rc 3 is two environmental skips — gitea-settings (no local SETTINGS_API_TOKEN) and conformance L2/L3 (CLUSTER_AVAILABLE unset). Not green. Reported as rc 3, not as pass. Run from an isolated git worktree with both CLAUDE.md §3 workarounds applied (DEV_KUBECONFIG=/dev/null plus an ENGINE shim bind-mounting the main checkout's .git and the sibling spec repo) — which is why spec-immutability is green here and skipped in CI, where the runner has no spec checkout. approval-policy passing locally is an offline N/A, not a verdict; the real verdict is the CI run below.
• The CI gate ran on PR #55 (run 244, head 1a67fc1b) and approval-policy is the ONLY failing stage — read from the job log, not from the red tick. Verbatim: "This PR changes the pipeline or the security posture it rests on, and has NO approving review from anyone other than its author ('gitea-admin')", naming exactly the three files I predicted: clusters/base/cluster-vars.yaml, clusters/base/kustomization.yaml, clusters/base/vpn-ingress.yaml. Every other stage that ran reported green — gitea-settings, decisions-log, lint, l0, render-golden, l1. So the branch is blocked on the operator's /approve-override <reason> (posted as a PR review of type "Comment") and on nothing else. I have not posted it — CLAUDE.md §1.
• Branch protection is live, measured by CI's own gitea-settings stage (it has the token this workstation lacks): required_approvals = 0, enable_status_check = true, status_check_contexts = ["gate / gate (pull_request)"], block_admin_merge_override = true, enable_push = false, dismiss_stale_approvals = true, and 1 runner online. That is the §2b re-measurement of CLAUDE.md §2's table for this branch, not a restatement of it.
• flux-postbuild-substitution (L0) actually exercised the new wiring, and caught a real defect in it. On the first run it FAILED, naming platform/vpn-ingress/daemonset.yaml:118 and attributing it to "Flux Kustomization 'vpn-ingress' (clusters/base/vpn-ingress.yaml, path ./platform/vpn-ingress)". Two things are proven by that message: the new tier's postBuild.substituteFrom declaration is correctly detected (so ${WG_CLIENT_CIDR} will be substituted, not shipped literally), and the check reads comments as well as values — the hazard was a comment that spelled out the forbidden braced form literally while claiming not to use it. Fixed; the check now passes 444/444 with 0 hazards.
• The wireguard kernel module is available on these nodes by construction: all three run Debian 13 (trixie) on kernel 7.1.8+deb13-amd64 (measured), and WireGuard has been in-tree since 5.6, so it is present in the distro kernel and only needs loading.
• net.ipv4.ip_forward = 1 is already written by node_baseline (roles/node_baseline/tasks/main.yml, /etc/sysctl.d/99-k3s-cilium.conf), and no kubelet carries --allowed-unsafe-sysctls (grepped across roles/k3s_server, roles/node_baseline and group_vars: zero hits). That pair is what makes the host netns the only option that does not require a node-restart change.
• Cilium's datapath, measured from cilium-config rather than from the values file: enable-ipv4-masquerade=true, enable-bpf-masquerade=true, routing-mode=tunnel, tunnel-protocol=vxlan, ipv4-native-routing-cidr EMPTY, Cilium v1.20.1. The vxlan underlay is where MTU = 1420 comes from, and the empty native-routing CIDR is why pod→LAN is masqueraded today.
• WG_CLIENT_CIDR collides with nothing. Pod CIDR 10.0.0.0/16 (CLUSTER_POD_CIDR), service CIDR 10.43.0.0/16 (measured: kubernetes Service clusterIP is 10.43.0.1), the PIA tunnel's 10.72.192.0/18 (platform/vpn-egress/README.md), and both LANs 192.168.1.0/24 / 192.168.2.0/24. 10.90.0.0/24 is outside all five.
• The egress-class label matters and is set to internet. Read live: CiliumEgressGatewayPolicy vpn-egress-steering selects podSelector: homelab.keithjackson.dev/egress-class: vpn with no namespace scope, so vpn here would have steered every remote client's internet traffic out PIA.
• All three nodes are untainted (spec.taints: None on each), so the DaemonSet covers the cluster with no tolerations.
• new-peer.sh was run end to end against a throwaway server key in the scratchpad (deleted immediately after; it never existed in the repo). It rendered a correct profile — Address 10.90.0.7/32, DNS 192.168.2.23, Endpoint wan.home.keithjackson.dev:51820, MTU 1420, PersistentKeepalive 25, AllowedIPs 0.0.0.0/0 — at mode 0600, printed the matching public-key stanza, and both argument-validation paths (missing name, out-of-range octet) refuse correctly. The private key was redacted from every line I displayed.
• The endpoint name is resolver-independent, which is the specific trap this avoids. wan.${BASE_DOMAIN} is an unproxied public Cloudflare A record kept current by platform/externaldns/ddns.yaml (checked every 5 min), so it resolves from a cellular connection — unlike any name PiHole answers, which a remote client cannot resolve until it is already connected. (This is the plex.direct lesson applied.)
• Velero does cover this namespace. platform/velero/helmrelease.yaml's daily schedule excludes only monitoring and unifi, so vpn-ingress and the wg-server-key Secret are captured off-cluster to R2. Read from the manifest, not from a backup listing.

What I did NOT verify

• THE ACTUAL FIX. Nothing in this branch has been proven to carry a single packet. The DaemonSet has never been reconciled, no wg0 has ever existed, and no handshake has ever been attempted. Three blockers are off-repo and manual (kernel module, server key, WAN forward), and this branch removes none of them — it only makes them the only remaining work.
• Outside-in reachability, at all. This repo has no off-LAN vantage point, which is exactly the gap the plex remote-access work hit. Step 6 of the runbook (a phone on cellular) is the only honest test and it has not been run.
• The in-pod iptables → host-netfilter interaction, which is my biggest single doubt. Debian 13 uses nf_tables; the container writes MASQUERADE/FORWARD rules into the host's tables via the image's own iptables. If that binary is the legacy backend while the host is nft, the rules can apply to a table the kernel does not consult — and the symptom would be the nastiest available: handshakes succeed, wg show looks perfect, and every forwarded connection hangs. Check this first if the tunnel comes up but nothing is reachable: iptables-save -t nat | grep MASQUERADE and nft list table ip nat on the node, from the host, not from the pod.
• Whether allowPrivilegeEscalation: false is compatible with the image's s6 init. It is the correct posture and I set it, but this repo has already been bitten by no-new-privileges misbehaving (platform/externaldns/ddns.yaml's README note). If the pod fails before wg-quick runs, that flag is the first thing to try flipping.
• Whether NET_ADMIN alone is sufficient. Reasoned from what wg-quick does (create the link, set a sysctl, write two netfilter rules), not measured. If iptables complains, NET_RAW is the likely addition — deliberately not granted pre-emptively.
• The image's custom-config path. lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124 (tag confirmed to exist and to be the current one, published 2026-10-01) is expected to bring up whatever is in /config/wg_confs/ when PEERS is unset. That is its documented behaviour; it has not been run here, and the init container's assembly of that file has never executed.
• Whether the probe is right. wg show wg0 is asserted to exit non-zero when the interface is absent. Not executed.
• Cilium's reaction to a new wg0 in the host netns. platform/cilium/values.yaml does not pin devices, so Cilium auto-detects; whether it attaches to wg0 — and whether that matters — is unknown. wg interfaces are already filtered out by node_health_collect.py, so something in the estate anticipates them, but that is not evidence about Cilium.
• Zero peers means the end-to-end authorisation path is untested, including revocation. The first enrolment is also the first test of peers.yaml's format being what wg-quick accepts.
• No peer-level observability exists. An exporter publishing wireguard_latest_handshake_seconds is the top follow-up; until then "who is connected" is an on-demand wg show and nothing alerts on an unexpected session. Pod-down is covered only by kube-prometheus-stack's generic KubePodNotReady / KubeDeploymentReplicasMismatch, which I did not confirm fire for a DaemonSet in this namespace.
• The security consequence I am most keen for a human to review rather than take from me: an enrolled device reaches the three k3s nodes themselves — apiserver 6443, kubelet 10250, SSH 22 — because "reach the entire local network" includes them. That is stated in the README and the runbook with the narrowing recipe, but it is a posture decision, not a verified fact, and it is the right thing to argue about on this PR.