---
date: 2026-10-05
branch: feat/vpn-remote-access-wireguard
title: Remote-access VPN — WireGuard on udp/51820, hostNetwork because a pod identity would 403 every Gateway hostname
---

## What changed

New platform component `platform/vpn-ingress/`: a **WireGuard** remote-access VPN so the operator's
Android / Linux / Windows / macOS devices can dial in from anywhere on the internet and behave like
machines on the LAN — full reach to `192.168.2.0/24` plus internet egress out the house WAN.

This is the **opposite direction** to the existing `platform/vpn-egress` (pods *out* through PIA,
ADR-0015). They share nothing but the word "VPN"; both READMEs now say so, and `README.md`'s layout
block labels the two.

**Transport: WireGuard, and the security argument is the handshake's failure mode, not "it is
encrypted".** WireGuard's first handshake message carries a MAC keyed with the responder's own
public key, so a sender that does not already know it gets **no reply at all** — no error, no reset,
no banner. From the internet, udp/51820 is indistinguishable from a closed port: nothing to
brute-force, no version string, no amplification. That is what makes an unfiltered WAN port
defensible here, and it is why there is deliberately no `loadBalancerSourceRanges`-style allowlist
(which would also break roaming clients). It also has first-party clients on all four named
platforms, which OpenVPN (needs a CA + per-device certs + a CRL) and IKEv2 do not.

**`hostNetwork: true`, and this is THE decision — forced by a measured property of this cluster.**
A pod-networked WireGuard server must SNAT client traffic to its own pod IP (Cilium drops a packet
whose source is not the endpoint's address), so every remote client would have arrived wearing *that
pod's* security identity. `decisions/2026-10-04-plex-advertise-plexdirect.md` measured what that
costs: a request to the shared Gateway VIP **from inside a pod returns `403 Access denied` from
Envoy for all 16 Gateway hostnames**, because Cilium's Gateway preserves the original source
identity and enforces the *destination's* ingress policy, and every app admits `reserved:ingress`
plus named peers — never an arbitrary pod. So the obvious build would have produced a tunnel that
came up healthy, showed connected peers, routed packets, and 403'd every cluster web UI — and
`*.home.keithjackson.dev` would not even have resolved, because `apps/wave-0/pihole-dns.yaml`'s
ingress policy deliberately omits the `cluster` entity. In the host netns there is no pod identity
to carry: forwarded traffic is SNATed to the node address and arrives as `host`/`remote-node`,
byte-for-byte what a LAN client looks like. Three further problems vanish with it: `ip_forward` is
already 1 on these nodes (so no `--allowed-unsafe-sysctls` kubelet change — it is *not* a safe
sysctl); no LoadBalancer VIP, so no L2-announcement lease lottery and no `externalTrafficPolicy`
trade-off; and no double encapsulation, which is what makes `MTU = 1420` simply correct.

`hostNetwork` is permitted by `restrict-host-namespaces`' own stated carve-out (it matches
`app-namespace=true` only, and names Cilium/node-exporter as the precedent); `namespace.yaml` is
deliberately unlabelled. **The cost is stated, not buried:** a hostNetwork pod has no Cilium
endpoint, so **no NetworkPolicy constrains this workload**. `networkpolicy.yaml` says that at
length rather than shipping an `allow world → 51820` rule that would read as the control and
enforce nothing; the real boundary is WireGuard's peer authentication.

**A DaemonSet, not a Deployment.** The WAN forward names one node address by hand, off-repo (same
shape as the plex tcp/32400 forward). A floating single replica would silently invalidate it on
every reschedule; an identical listener on all three nodes makes **failover a one-line router
change** with no cluster work and no re-issued client profiles. A pinned LoadBalancer VIP was the
considered alternative and was rejected: it reintroduces the L2 lease, SNAT and an extra node hop
into the one path whose failure mode is "I am away from home and cannot get in".

**Authorisation is split so that it stays reviewable.** Peer **public** keys live in
`platform/vpn-ingress/peers.yaml` in **plaintext** — a public key grants nothing by itself, and this
makes "who can get into my house" a PR diff with `git log` as its audit trail. The only real secret,
the server's private key, is the `wg-server-key` Secret, **created out of band by the operator**: an
agent must not mint a long-lived private key (CLAUDE.md §6), and a key that passed through a session
transcript would have to be treated as compromised. The two halves are joined only at pod start, by
an init container, into a `medium: Memory` emptyDir — so the assembled config holding the key never
touches a node's disk and dies with the pod.

**There are no peers yet, and that is the correct initial state**, for the same reason. The VPN ships
INERT-BUT-HEALTHY: a zero-peer interface listens and refuses every handshake.

Files:

- `platform/vpn-ingress/` (new): `namespace.yaml`, `peers.yaml` (the plaintext peer register),
  `daemonset.yaml` (hostNetwork listener + config assembly), `networkpolicy.yaml` (namespace
  baseline, with the honest caveat), `kustomization.yaml`, `README.md` (the design record).
- `clusters/base/vpn-ingress.yaml` (new): its own **non-gating post-platform** Flux Kustomization
  (`dependsOn platform-config`, `wait:false`). Non-gating is load-bearing rather than conventional
  here: the DaemonSet *cannot* start until the operator creates the server key, so in the gating
  `platform` tier an unfinished manual step would have stalled every apps wave on merge.
- `clusters/base/kustomization.yaml`: the new tier added.
- `clusters/base/cluster-vars.yaml`: new `WG_CLIENT_CIDR` (`10.90.0.0/24`), declared next to
  `LAN_CIDR` because it is the one range that must be collision-checked against every other; the
  comment enumerates the five it avoids. **No prod override, deliberately** — unlike every VIP, this
  range is private to the tunnel, so the same value is correct in both environments and a prod patch
  would imply otherwise. Substitution inventory updated.
- `infra/ansible/roles/node_baseline/defaults/main.yml`: `wireguard` added to
  `node_baseline_kernel_modules`. The pod has `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so
  it cannot autoload the module — it CrashLoops with a clear error rather than quietly degrading to
  userspace wireguard-go (which would need `/dev/net/tun`, a device a hostPath mount cannot actually
  grant, as the `/dev/dri` attempt proved).
- `infra/vpn-ingress/new-peer.sh` (new): enrols a device. Generates the keypair on the operator's
  machine, writes a 0600 profile **outside** the checkout (it refuses to write into the repo),
  prints a QR code for phones, and prints only the public `[Peer]` stanza. It also refuses a
  duplicate address, because two devices on one address does not error anywhere — the server routes
  to whichever handshook last, so both appear to work intermittently.
- `runbooks/vpn-remote-access.md` (new) + its `kustomization.yaml` generator entry + `README.md`
  index line: the three manual steps, the four off-LAN verification checks, revocation, and why no
  connection alert exists.
- `README.md`: `vpn-ingress` added to the layout, with the egress/ingress distinction spelled out.

**THREE MANUAL STEPS REMAIN AND THE VPN DOES NOT WORK UNTIL ALL THREE ARE DONE** (all off-repo by
necessity; `runbooks/vpn-remote-access.md`): run `node_baseline` to load the kernel module, create
the `wg-server-key` Secret, and forward WAN udp/51820 to one node address on the SOHO router.

**SENSITIVE PATHS — this PR touches `clusters/**` (three files), so `ci/approval-policy.sh` will
BLOCK it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.

## What I verified

Live prod, read-only, 2026-10-05, plus the offline gate.

- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
  Run from an isolated git worktree with both CLAUDE.md §3 workarounds applied
  (`DEV_KUBECONFIG=/dev/null` plus an `ENGINE` shim bind-mounting the main checkout's `.git` and
  the sibling spec repo) — which is why `spec-immutability` is **green here and skipped in CI**,
  where the runner has no spec checkout. `approval-policy` passing locally is an offline N/A, not a
  verdict; the real verdict is the CI run below.
- **The CI gate ran on PR #55 (run 244, head `1a67fc1b`) and `approval-policy` is the ONLY failing
  stage** — read from the job log, not from the red tick. Verbatim: "This PR changes the pipeline or
  the security posture it rests on, and has NO approving review from anyone other than its author
  ('gitea-admin')", naming exactly the three files I predicted:
  `clusters/base/cluster-vars.yaml`, `clusters/base/kustomization.yaml`,
  `clusters/base/vpn-ingress.yaml`. Every other stage that ran reported green —
  `gitea-settings`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. So the branch is blocked
  on the operator's `/approve-override <reason>` (posted as a PR **review** of type "Comment") and
  on nothing else. **I have not posted it** — CLAUDE.md §1.
- **Branch protection is live, measured by CI's own `gitea-settings` stage** (it has the token this
  workstation lacks): `required_approvals = 0`, `enable_status_check = true`,
  `status_check_contexts = ["gate / gate (pull_request)"]`, `block_admin_merge_override = true`,
  `enable_push = false`, `dismiss_stale_approvals = true`, and **1 runner online**. That is the §2b
  re-measurement of CLAUDE.md §2's table for this branch, not a restatement of it.
- **`flux-postbuild-substitution` (L0) actually exercised the new wiring, and caught a real defect
  in it.** On the first run it FAILED, naming `platform/vpn-ingress/daemonset.yaml:118` and
  attributing it to "Flux Kustomization 'vpn-ingress' (clusters/base/vpn-ingress.yaml, path
  ./platform/vpn-ingress)". Two things are proven by that message: the new tier's
  `postBuild.substituteFrom` declaration is correctly detected (so `${WG_CLIENT_CIDR}` will be
  substituted, not shipped literally), and the check reads comments as well as values — the hazard
  was a comment that spelled out the forbidden braced form literally while claiming not to use it.
  Fixed; the check now passes 444/444 with 0 hazards.
- **The `wireguard` kernel module is available on these nodes by construction**: all three run
  Debian 13 (trixie) on kernel `7.1.8+deb13-amd64` (measured), and WireGuard has been in-tree since
  5.6, so it is present in the distro kernel and only needs loading.
- **`net.ipv4.ip_forward = 1` is already written by `node_baseline`**
  (`roles/node_baseline/tasks/main.yml`, `/etc/sysctl.d/99-k3s-cilium.conf`), and **no kubelet
  carries `--allowed-unsafe-sysctls`** (grepped across `roles/k3s_server`, `roles/node_baseline` and
  `group_vars`: zero hits). That pair is what makes the host netns the only option that does not
  require a node-restart change.
- **Cilium's datapath, measured from `cilium-config` rather than from the values file**:
  `enable-ipv4-masquerade=true`, `enable-bpf-masquerade=true`, `routing-mode=tunnel`,
  `tunnel-protocol=vxlan`, `ipv4-native-routing-cidr` EMPTY, Cilium `v1.20.1`. The vxlan underlay is
  where `MTU = 1420` comes from, and the empty native-routing CIDR is why pod→LAN is masqueraded
  today.
- **`WG_CLIENT_CIDR` collides with nothing.** Pod CIDR `10.0.0.0/16` (`CLUSTER_POD_CIDR`), service
  CIDR `10.43.0.0/16` (measured: `kubernetes` Service clusterIP is `10.43.0.1`), the PIA tunnel's
  `10.72.192.0/18` (`platform/vpn-egress/README.md`), and both LANs `192.168.1.0/24` /
  `192.168.2.0/24`. `10.90.0.0/24` is outside all five.
- **The egress-class label matters and is set to `internet`.** Read live:
  `CiliumEgressGatewayPolicy vpn-egress-steering` selects `podSelector:
  homelab.keithjackson.dev/egress-class: vpn` with **no namespace scope**, so `vpn` here would have
  steered every remote client's internet traffic out PIA.
- **All three nodes are untainted** (`spec.taints: None` on each), so the DaemonSet covers the
  cluster with no tolerations.
- **`new-peer.sh` was run end to end** against a throwaway server key in the scratchpad (deleted
  immediately after; it never existed in the repo). It rendered a correct profile — `Address
  10.90.0.7/32`, `DNS 192.168.2.23`, `Endpoint wan.home.keithjackson.dev:51820`, `MTU 1420`,
  `PersistentKeepalive 25`, `AllowedIPs 0.0.0.0/0` — at mode 0600, printed the matching public-key
  stanza, and both argument-validation paths (missing name, out-of-range octet) refuse correctly.
  The private key was redacted from every line I displayed.
- **The endpoint name is resolver-independent, which is the specific trap this avoids.**
  `wan.${BASE_DOMAIN}` is an unproxied public Cloudflare A record kept current by
  `platform/externaldns/ddns.yaml` (checked every 5 min), so it resolves from a cellular connection
  — unlike any name PiHole answers, which a remote client cannot resolve until it is already
  connected. (This is the `plex.direct` lesson applied.)
- **Velero does cover this namespace.** `platform/velero/helmrelease.yaml`'s `daily` schedule
  excludes only `monitoring` and `unifi`, so `vpn-ingress` and the `wg-server-key` Secret are
  captured off-cluster to R2. Read from the manifest, not from a backup listing.

## What I did NOT verify

- **THE ACTUAL FIX. Nothing in this branch has been proven to carry a single packet.** The
  DaemonSet has never been reconciled, no `wg0` has ever existed, and no handshake has ever been
  attempted. Three blockers are off-repo and manual (kernel module, server key, WAN forward), and
  this branch removes none of them — it only makes them the *only* remaining work.
- **Outside-in reachability, at all.** This repo has **no off-LAN vantage point**, which is exactly
  the gap the plex remote-access work hit. Step 6 of the runbook (a phone on cellular) is the only
  honest test and it has not been run.
- **The in-pod `iptables` → host-netfilter interaction, which is my biggest single doubt.** Debian 13
  uses nf_tables; the container writes MASQUERADE/FORWARD rules into the **host's** tables via the
  image's own `iptables`. If that binary is the legacy backend while the host is nft, the rules can
  apply to a table the kernel does not consult — and the symptom would be the nastiest available:
  handshakes succeed, `wg show` looks perfect, and every forwarded connection hangs. **Check this
  first if the tunnel comes up but nothing is reachable:** `iptables-save -t nat | grep MASQUERADE`
  and `nft list table ip nat` on the node, from the host, not from the pod.
- **Whether `allowPrivilegeEscalation: false` is compatible with the image's s6 init.** It is the
  correct posture and I set it, but this repo has already been bitten by `no-new-privileges`
  misbehaving (`platform/externaldns/ddns.yaml`'s README note). If the pod fails before
  `wg-quick` runs, that flag is the first thing to try flipping.
- **Whether `NET_ADMIN` alone is sufficient.** Reasoned from what `wg-quick` does (create the link,
  set a sysctl, write two netfilter rules), not measured. If `iptables` complains, `NET_RAW` is the
  likely addition — deliberately not granted pre-emptively.
- **The image's custom-config path.** `lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124` (tag
  confirmed to exist and to be the current one, published 2026-10-01) is expected to bring up
  whatever is in `/config/wg_confs/` when `PEERS` is unset. That is its documented behaviour; it has
  not been run here, and the init container's assembly of that file has never executed.
- **Whether the probe is right.** `wg show wg0` is asserted to exit non-zero when the interface is
  absent. Not executed.
- **Cilium's reaction to a new `wg0` in the host netns.** `platform/cilium/values.yaml` does not pin
  `devices`, so Cilium auto-detects; whether it attaches to `wg0` — and whether that matters — is
  unknown. `wg` interfaces are already filtered out by `node_health_collect.py`, so something in
  the estate anticipates them, but that is not evidence about Cilium.
- **Zero peers means the end-to-end authorisation path is untested**, including revocation. The
  first enrolment is also the first test of `peers.yaml`'s format being what `wg-quick` accepts.
- **No peer-level observability exists.** An exporter publishing
  `wireguard_latest_handshake_seconds` is the top follow-up; until then "who is connected" is an
  on-demand `wg show` and nothing alerts on an unexpected session. Pod-down is covered only by
  kube-prometheus-stack's generic `KubePodNotReady` / `KubeDeploymentReplicasMismatch`, which I did
  not confirm fire for a DaemonSet in this namespace.
- **The security consequence I am most keen for a human to review rather than take from me:** an
  enrolled device reaches the three k3s nodes themselves — apiserver 6443, kubelet 10250, SSH 22 —
  because "reach the entire local network" includes them. That is stated in the README and the
  runbook with the narrowing recipe, but it is a posture decision, not a verified fact, and it is
  the right thing to argue about on this PR.