# platform/vpn-ingress — WireGuard remote access (north-south INGRESS)

Lets the operator's own devices — Android, Linux, Windows, macOS, iOS — dial in from anywhere on
the internet and behave as though plugged into the home LAN: full reach to `192.168.2.0/24` and
egress out the house WAN.

> **This is not `platform/vpn-egress`.** They are opposite directions and share nothing but the
> word "VPN". `vpn-egress` pushes *pods out* to the internet through a PIA tunnel (ADR-0015,
> R-NET-05/06/07). This component lets *remote clients in*. Changing one never affects the other.

| | |
|---|---|
| Transport | WireGuard, **udp/51820**, kernel implementation |
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
| Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |

## Why WireGuard

The requirement names four client platforms. WireGuard is the only option that has a *first-party*
client on all of them (official apps for Android, iOS, Windows and macOS; in-kernel on Linux since
5.6, so a Linux client needs no userspace daemon at all). It is also the smallest thing to expose:
one UDP port, no cipher negotiation, no PKI, no user database, no TLS stack.

OpenVPN would have needed a CA, per-device certificates and a revocation list — three pieces of
long-lived state this repo would then have to protect and rotate. IKEv2/IPsec has native clients but
is painful to run in a container and brittle behind double NAT. Neither buys anything here.

### Why this is safe to expose to the whole internet

The honest answer is not "because it is encrypted" — it is the handshake's failure mode:

**An unauthenticated packet gets no reply at all.** WireGuard's first handshake message carries a
MAC keyed with the *responder's own public key*. If the sender does not already know it, the packet
is dropped silently — no error, no reset, no banner, nothing a scanner can fingerprint. From the
outside, udp/51820 is indistinguishable from a closed port. There is no login prompt to brute-force,
no version string to match against a CVE, and no amplification to abuse.

That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
would also have made roaming clients unusable — a phone's address changes with every cell tower).
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN and
this tunnel's client subnet: a resolver answers anybody, so it must be scoped by address. WireGuard
answers nobody.

What it does **not** protect against is a stolen device — anyone holding a device's private key
*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).

## Architecture, and the one decision that drives everything

```
  phone on 5G                    SOHO router                 k3s-prod-N (host netns)
 ┌──────────────┐  udp/51820   ┌───────────────┐  forward   ┌────────────────────────────┐
 │ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24           │
 │ 10.90.0.2/32 │   over the   │ node IP:51820 │            │   decrypt, then FORWARD    │
 └──────────────┘   internet   └───────────────┘            │  SNAT (LAN), eBPF LB (VIPs)│
                                                            └─────────────┬──────────────┘
                                                                          │ translated per destination (below)
                                        ┌─────────────────────────────────┼──────────────────────┐
                                        ▼                                 ▼                      ▼
                               LAN 192.168.2.0/24            Gateway VIP .20 (16 web UIs)   WAN / internet
                               (NAS, router, printers)        PiHole DNS VIP .23
```

### `hostNetwork: true` is required, not convenient

This is the only genuinely interesting decision in the component, and it was forced by a *measured*
property of this cluster rather than chosen for convenience.

In Cilium, what a packet may reach is decided by the **security identity of its source**. A
WireGuard server running as an ordinary pod must SNAT client traffic to its own pod IP (Cilium drops
a packet whose source is not the endpoint's own address), so every remote client would have arrived
everywhere wearing *that pod's* identity. On 2026-10-04 this cluster measured what that costs
(`decisions/2026-10-04-plex-advertise-plexdirect.md`): a request to the shared Gateway VIP **from
inside a pod returns `403 Access denied` from Envoy for all 16 Gateway hostnames**, because Cilium's
Gateway preserves the original source identity and enforces the *destination's* ingress policy —
and every app admits `reserved:ingress` plus named peers, never an arbitrary pod. LAN browsers work
only because they arrive as `reserved:world` / `reserved:host`.

So the pod-networked version of this component would have produced a tunnel that came up healthy,
reported connected peers, routed packets — and 403'd **every single cluster web UI**: plex, gitea,
keycloak, nextcloud, grafana, headlamp, the apex. `*.home.keithjackson.dev` would not even have
resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately omits the `cluster`
entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
manifests would have hinted at it.

In the **host** netns there is no pod identity to carry, which is what avoids the 403s. Clients do
**not**, however, all arrive SNATed to the node. Measured 2026-11-08: Cilium counts `wg0` among its
kube-proxy-replacement devices (`cilium-dbg status`: `eno1`, `wg0`, `tun0`), so what a packet looks
like depends on where it is going:

| destination | handled by | source the destination sees |
|---|---|---|
| LAN hosts, the internet | kernel FORWARD, then the iptables MASQUERADE in [`daemonset.yaml`](./daemonset.yaml) | the node's address |
| LoadBalancer VIPs (Gateway `.20`, PiHole `.23`, ...) | Cilium's eBPF load balancer, as the packet enters from `wg0`, **before** MASQUERADE | `10.90.0.x` at the VIP's source-range check, identity `world` |

The second row is why PiHole's `loadBalancerSourceRanges` has to list `${WG_CLIENT_CIDR}` as well as
the LAN: left LAN-only, tunnel DNS is predicted to die with `DROP_NOT_IN_SRC_RANGE`. That prediction
and the row's source column come from reading Cilium's datapath source; **no peer has ever completed
a handshake, so neither has been observed live.** The Gateway VIP carries no source ranges and needs
no change. Three further problems vanish with the host netns:

* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
  `/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
  pod-netns build would have needed `--allowed-unsafe-sysctls` on all three kubelets — a
  node-restart change to the k3s service, to run a VPN.
* **No LoadBalancer VIP**, so no `CiliumL2AnnouncementPolicy` lease lottery (the backend-blind
  announcer that cost this LAN an 8-day Minecraft outage — see
  `conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
  `externalTrafficPolicy` trade-off to make at all.
* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
  1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` correct on the
  server's `wg0`. It says nothing about a client's own path: `new-peer.sh` defaults client profiles
  to `1280` (see "Troubleshooting" in `runbooks/vpn-remote-access.md`).

`hostNetwork` is *permitted* here, and not by oversight:
`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
app-namespace=true` only, and names this exact carve-out in its own comment — "platform components
that do … e.g. Cilium, node-exporter … live in platform namespaces that do NOT carry the
app-namespace label". [`namespace.yaml`](./namespace.yaml) is deliberately unlabelled.

**The cost, stated rather than buried:** a `hostNetwork` pod has no Cilium endpoint, so **no
NetworkPolicy constrains this workload** — not [`networkpolicy.yaml`](./networkpolicy.yaml), not
anything. That file says so at length instead of implying a protection it cannot provide. The
boundary around this port is WireGuard's peer authentication, full stop.

### Why a DaemonSet

The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
re-issued client profiles. The two idle listeners cost ~15 MiB each.

A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
hop into the one path whose failure mode is *"I am away from home and cannot get in"*. Fewer
translations win here.

### The split that keeps authorisation reviewable

| half | where | why |
|---|---|---|
| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |

The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
so the assembled config holding the private key never touches any node's disk and dies with the pod.

Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with
everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in,
corrected after review — would have put the one credential every enrolled device depends on *outside*
the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this
key's public half.

Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
new line granting a new device access to the whole LAN.

## What a client can reach — and how to narrow it

By design, **everything a LAN machine can reach**: all of `192.168.2.0/24`, the shared Gateway's 16
hostnames, PiHole, the NAS, the router's admin UI, and the internet via the house WAN.

**That includes the three k3s nodes themselves** — `192.168.2.10/.11/.12`, and therefore the
kube-apiserver on 6443, kubelet on 10250 and SSH on 22. An enrolled device is as privileged as a
machine sitting in the house, which is what "reach the entire local network" asks for; it is
recorded here because it is the largest consequence of saying yes to that, and because nobody should
discover it later. **It is also why a lost device must be revoked promptly** (delete its stanza,
merge; ~1 min to take effect).

To grant less, narrow it at the **client** profile first — `new-peer.sh --split` emits a
split-tunnel profile routing only the LAN and the tunnel subnet, so the device keeps its own
internet path. To enforce less *server-side*, add filtering to the `PostUp` chain in
[`daemonset.yaml`](./daemonset.yaml) (e.g. `iptables -A FORWARD -i wg0 -d 192.168.2.10 -j DROP`
before the accept rules) — the FORWARD chain there is the only enforcement point, since no
NetworkPolicy applies.

## Phone profiles: one tunnel for home, one for away

A phone that needs the LAN-only apps runs full tunnel (`AllowedIPs = 0.0.0.0/0`, `DNS = 192.168.2.23`)
everywhere, so it needs **two tunnels**, each its own peer with its own key and address:

| tunnel | peer | address | Endpoint | use when |
|---|---|---|---|---|
| `away` | `keith-phone` | `10.90.0.2` | `wan.home.keithjackson.dev:51820` | off the home network |
| `home` | `keith-phone-home` | `10.90.0.3` | `192.168.2.10:51820` | on the home Wi-Fi |

**`wan.` cannot work from inside the house.** It is an unproxied A record holding the house's WAN
address (`platform/externaldns/README.md`), and the router has no hairpin NAT, so a LAN client that
sends to it never comes back in through the port-forward. The Cloudflare Tunnel is no way round: it
carries HTTP(S) to the Gateway and WireGuard is UDP to a node, so `wan.` has to stay DNS-only and
`away` has to keep using it.

**`home` depends on `k3s-prod-1`.** Its Endpoint is that node's address, with no failover: if
prod-1 is down, `home` cannot handshake even though the other two listeners are healthy. The fix
is client-side: edit the tunnel's Endpoint to `192.168.2.11` or `.12`, since every node answers the
same keys. No server change and no new key is needed. The same dependency applies to `away` if the
WAN forward points at prod-1.

Android allows one active VPN at a time, so switch tunnels by hand when leaving or arriving.

## Operating it

* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
  stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
  stops working as soon as the pods roll (the generator's content hash guarantees they do).
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
  (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
* **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.

## Measured on prod, 2026-10-06 — what is now known rather than assumed

The first real reconcile settled four things this component's design rested on. Three went the
right way; one was a defect in my own manifest.

| question | answer |
|---|---|
| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |

## Prerequisites outside this directory

1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
   (`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
   `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
   CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a
   `REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring
   is complete and the only outstanding act is replacing the value (runbook step 2). Until then the
   DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the
   value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by
   mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is
   the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves
   every Flux condition green.
3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
   and the half CI can never check.

## Not built (deliberate, with reasons)

* **No metrics / no peer-level alerting.** A WireGuard exporter would expose
  `wireguard_latest_handshake_seconds` per peer — genuinely useful for spotting unexpected
  activity — but it needs a second `NET_ADMIN` container in the pod that terminates the
  internet-facing tunnel, and that trade was not worth making unverified. Pod-down is already
  covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
  kube-prometheus-stack. **This is the top follow-up.**
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
  handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
  reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
  `wg-server-key` Secret.
* **No IPv6.** The server offers no IPv6 tunnel address and `new-peer.sh` emits only `0.0.0.0/0`.
  What happens to a client's IPv6 then depends on the client. The **Android app does not leak it**:
  its `GoBackend` skips `allowFamily()` when the config is one peer with a default route, so Android
  blocks IPv6 for the VPN's duration and apps fall back to IPv4. Other clients are not covered by
  that code; expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside the tunnel. Fix is
  an `fd..` ULA on `wg0` plus `::/0` in the client's `AllowedIPs`; not done because nothing here
  needs it.
* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
  browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
  into the tunnel.