home-network-k3s-impl/platform/vpn-ingress/README.md

# platform/vpn-ingress — WireGuard remote access (north-south INGRESS)

Lets the operator's own devices — Android, Linux, Windows, macOS, iOS — dial in from anywhere on
the internet and behave as though plugged into the home LAN: full reach to `192.168.2.0/24` and
egress out the house WAN.

> **This is not `platform/vpn-egress`.** They are opposite directions and share nothing but the
> word "VPN". `vpn-egress` pushes *pods out* to the internet through a PIA tunnel (ADR-0015,
> R-NET-05/06/07). This component lets *remote clients in*. Changing one never affects the other.

| | |
|---|---|
| Transport | WireGuard, **udp/51820**, kernel implementation |
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** |
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
| Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |

## Why WireGuard

The requirement names four client platforms. WireGuard is the only option that has a *first-party*
client on all of them (official apps for Android, iOS, Windows and macOS; in-kernel on Linux since
5.6, so a Linux client needs no userspace daemon at all). It is also the smallest thing to expose:
one UDP port, no cipher negotiation, no PKI, no user database, no TLS stack.

OpenVPN would have needed a CA, per-device certificates and a revocation list — three pieces of
long-lived state this repo would then have to protect and rotate. IKEv2/IPsec has native clients but
is painful to run in a container and brittle behind double NAT. Neither buys anything here.

### Why this is safe to expose to the whole internet

The honest answer is not "because it is encrypted" — it is the handshake's failure mode:

**An unauthenticated packet gets no reply at all.** WireGuard's first handshake message carries a
MAC keyed with the *responder's own public key*. If the sender does not already know it, the packet
is dropped silently — no error, no reset, no banner, nothing a scanner can fingerprint. From the
outside, udp/51820 is indistinguishable from a closed port. There is no login prompt to brute-force,
no version string to match against a CVE, and no amplification to abuse.

That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
would also have made roaming clients unusable — a phone's address changes with every cell tower).
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a
resolver answers anybody, so it must be scoped by address. WireGuard answers nobody.

What it does **not** protect against is a stolen device — anyone holding a device's private key
*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).

## Architecture, and the one decision that drives everything

```
  phone on 5G                    SOHO router                 k3s-prod-N (host netns)
 ┌──────────────┐  udp/51820   ┌───────────────┐  forward   ┌────────────────────────────┐
 │ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24           │
 │ 10.90.0.2/32 │   over the   │ node IP:51820 │            │   decrypt, then FORWARD    │
 └──────────────┘   internet   └───────────────┘            │   SNAT to the node's addr  │
                                                            └─────────────┬──────────────┘
                                                                          │ as `host`/`remote-node`
                                        ┌─────────────────────────────────┼──────────────────────┐
                                        ▼                                 ▼                      ▼
                               LAN 192.168.2.0/24            Gateway VIP .20 (16 web UIs)   WAN / internet
                               (NAS, router, printers)        PiHole DNS VIP .23
```

### `hostNetwork: true` is required, not convenient

This is the only genuinely interesting decision in the component, and it was forced by a *measured*
property of this cluster rather than chosen for convenience.

In Cilium, what a packet may reach is decided by the **security identity of its source**. A
WireGuard server running as an ordinary pod must SNAT client traffic to its own pod IP (Cilium drops
a packet whose source is not the endpoint's own address), so every remote client would have arrived
everywhere wearing *that pod's* identity. On 2026-10-04 this cluster measured what that costs
(`decisions/2026-10-04-plex-advertise-plexdirect.md`): a request to the shared Gateway VIP **from
inside a pod returns `403 Access denied` from Envoy for all 16 Gateway hostnames**, because Cilium's
Gateway preserves the original source identity and enforces the *destination's* ingress policy —
and every app admits `reserved:ingress` plus named peers, never an arbitrary pod. LAN browsers work
only because they arrive as `reserved:world` / `reserved:host`.

So the pod-networked version of this component would have produced a tunnel that came up healthy,
reported connected peers, routed packets — and 403'd **every single cluster web UI**: plex, gitea,
keycloak, nextcloud, grafana, headlamp, the apex. `*.home.keithjackson.dev` would not even have
resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately omits the `cluster`
entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
manifests would have hinted at it.

In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the
node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
byte-for-byte what a LAN client looks like. Three further problems vanish with it:

* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
  `/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
  pod-netns build would have needed `--allowed-unsafe-sysctls` on all three kubelets — a
  node-restart change to the k3s service, to run a VPN.
* **No LoadBalancer VIP**, so no `CiliumL2AnnouncementPolicy` lease lottery (the backend-blind
  announcer that cost this LAN an 8-day Minecraft outage — see
  `conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
  `externalTrafficPolicy` trade-off to make at all.
* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
  1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct.

`hostNetwork` is *permitted* here, and not by oversight:
`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
app-namespace=true` only, and names this exact carve-out in its own comment — "platform components
that do … e.g. Cilium, node-exporter … live in platform namespaces that do NOT carry the
app-namespace label". [`namespace.yaml`](./namespace.yaml) is deliberately unlabelled.

**The cost, stated rather than buried:** a `hostNetwork` pod has no Cilium endpoint, so **no
NetworkPolicy constrains this workload** — not [`networkpolicy.yaml`](./networkpolicy.yaml), not
anything. That file says so at length instead of implying a protection it cannot provide. The
boundary around this port is WireGuard's peer authentication, full stop.

### Why a DaemonSet

The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
re-issued client profiles. The two idle listeners cost ~15 MiB each.

A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
hop into the one path whose failure mode is *"I am away from home and cannot get in"*. Fewer
translations win here.

### The split that keeps authorisation reviewable

| half | where | why |
|---|---|---|
| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) |

The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
so the assembled config holding the private key never touches any node's disk and dies with the pod.

Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
new line granting a new device access to the whole LAN.

## What a client can reach — and how to narrow it

By design, **everything a LAN machine can reach**: all of `192.168.2.0/24`, the shared Gateway's 16
hostnames, PiHole, the NAS, the router's admin UI, and the internet via the house WAN.

**That includes the three k3s nodes themselves** — `192.168.2.10/.11/.12`, and therefore the
kube-apiserver on 6443, kubelet on 10250 and SSH on 22. An enrolled device is as privileged as a
machine sitting in the house, which is what "reach the entire local network" asks for; it is
recorded here because it is the largest consequence of saying yes to that, and because nobody should
discover it later. **It is also why a lost device must be revoked promptly** (delete its stanza,
merge; ~1 min to take effect).

To grant less, narrow it at the **client** profile first — `new-peer.sh --split` emits a
split-tunnel profile routing only the LAN and the tunnel subnet, so the device keeps its own
internet path. To enforce less *server-side*, add filtering to the `PostUp` chain in
[`daemonset.yaml`](./daemonset.yaml) (e.g. `iptables -A FORWARD -i wg0 -d 192.168.2.10 -j DROP`
before the accept rules) — the FORWARD chain there is the only enforcement point, since no
NetworkPolicy applies.

## Operating it

* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
  stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
  (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
* **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.

## Prerequisites outside this directory

1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
   (`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
   `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
   CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start —
   the correct fail-closed state for an unenrolled VPN.
3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
   and the half CI can never check.

## Not built (deliberate, with reasons)

* **No metrics / no peer-level alerting.** A WireGuard exporter would expose
  `wireguard_latest_handshake_seconds` per peer — genuinely useful for spotting unexpected
  activity — but it needs a second `NET_ADMIN` container in the pod that terminates the
  internet-facing tunnel, and that trade was not worth making unverified. Pod-down is already
  covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
  kube-prometheus-stack. **This is the top follow-up.**
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
  handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
  reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
  `wg-server-key` Secret.
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
  still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point
  was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's
  `AllowedIPs`; not done because nothing here needs it.
* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
  browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
  into the tunnel.