home-network-k3s-impl/platform/vpn-ingress

platform/vpn-ingress — WireGuard remote access (north-south INGRESS)

Lets the operator's own devices — Android, Linux, Windows, macOS, iOS — dial in from anywhere on the internet and behave as though plugged into the home LAN: full reach to 192.168.2.0/24 and egress out the house WAN.

This is not platform/vpn-egress. They are opposite directions and share nothing but the word "VPN". vpn-egress pushes pods out to the internet through a PIA tunnel (ADR-0015, R-NET-05/06/07). This component lets remote clients in. Changing one never affects the other.

Transport	WireGuard, udp/51820, kernel implementation
Where it runs	hostNetwork DaemonSet — one warm listener on every node
Client subnet	${WG_CLIENT_CIDR} = 10.90.0.0/24; .1 is the server, .2–.254 devices
Authorisation	a device's public key is in peers.conf, or it cannot connect
The one secret	the server's private key — SOPS-encrypted in this tree, wg-server-key.sops.yaml
Exposure	one hand-made WAN port-forward on the SOHO router, off-repo
Enrolling a device	infra/vpn-ingress/new-peer.sh
Procedure / bring-up	runbooks/vpn-remote-access.md

Why WireGuard

The requirement names four client platforms. WireGuard is the only option that has a first-party client on all of them (official apps for Android, iOS, Windows and macOS; in-kernel on Linux since 5.6, so a Linux client needs no userspace daemon at all). It is also the smallest thing to expose: one UDP port, no cipher negotiation, no PKI, no user database, no TLS stack.

OpenVPN would have needed a CA, per-device certificates and a revocation list — three pieces of long-lived state this repo would then have to protect and rotate. IKEv2/IPsec has native clients but is painful to run in a container and brittle behind double NAT. Neither buys anything here.

Why this is safe to expose to the whole internet

The honest answer is not "because it is encrypted" — it is the handshake's failure mode:

An unauthenticated packet gets no reply at all. WireGuard's first handshake message carries a MAC keyed with the responder's own public key. If the sender does not already know it, the packet is dropped silently — no error, no reset, no banner, nothing a scanner can fingerprint. From the outside, udp/51820 is indistinguishable from a closed port. There is no login prompt to brute-force, no version string to match against a CVE, and no amplification to abuse.

That is the whole security argument, and it is why the exposed port carries no IP allowlist (which would also have made roaming clients unusable — a phone's address changes with every cell tower). Compare apps/wave-0/pihole-dns.yaml, which does pin loadBalancerSourceRanges to the LAN: a resolver answers anybody, so it must be scoped by address. WireGuard answers nobody.

What it does not protect against is a stolen device — anyone holding a device's private key is that device. That is what revocation is for: delete the stanza, merge, done (~1 min).

Architecture, and the one decision that drives everything

  phone on 5G                    SOHO router                 k3s-prod-N (host netns)
 ┌──────────────┐  udp/51820   ┌───────────────┐  forward   ┌────────────────────────────┐
 │ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24           │
 │ 10.90.0.2/32 │   over the   │ node IP:51820 │            │   decrypt, then FORWARD    │
 └──────────────┘   internet   └───────────────┘            │   SNAT to the node's addr  │
                                                            └─────────────┬──────────────┘
                                                                          │ as `host`/`remote-node`
                                        ┌─────────────────────────────────┼──────────────────────┐
                                        ▼                                 ▼                      ▼
                               LAN 192.168.2.0/24            Gateway VIP .20 (16 web UIs)   WAN / internet
                               (NAS, router, printers)        PiHole DNS VIP .23

hostNetwork: true is required, not convenient

This is the only genuinely interesting decision in the component, and it was forced by a measured property of this cluster rather than chosen for convenience.

In Cilium, what a packet may reach is decided by the security identity of its source. A WireGuard server running as an ordinary pod must SNAT client traffic to its own pod IP (Cilium drops a packet whose source is not the endpoint's own address), so every remote client would have arrived everywhere wearing that pod's identity. On 2026-10-04 this cluster measured what that costs (decisions/2026-10-04-plex-advertise-plexdirect.md): a request to the shared Gateway VIP from inside a pod returns 403 Access denied from Envoy for all 16 Gateway hostnames, because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy — and every app admits reserved:ingress plus named peers, never an arbitrary pod. LAN browsers work only because they arrive as reserved:world / reserved:host.

So the pod-networked version of this component would have produced a tunnel that came up healthy, reported connected peers, routed packets — and 403'd every single cluster web UI: plex, gitea, keycloak, nextcloud, grafana, headlamp, the apex. *.home.keithjackson.dev would not even have resolved, because apps/wave-0/pihole-dns.yaml's ingress policy deliberately omits the cluster entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the manifests would have hinted at it.

In the host netns there is no pod identity to carry. Forwarded client traffic is SNATed to the node's own address, so it reaches the Gateway, PiHole and every LAN host as host/remote-node — byte-for-byte what a LAN client looks like. Three further problems vanish with it:

• net.ipv4.ip_forward is already 1 on these nodes (node_baseline writes /etc/sysctl.d/99-k3s-cilium.conf). It is not in the kubelet's safe-sysctl allowlist, so a pod-netns build would have needed --allowed-unsafe-sysctls on all three kubelets — a node-restart change to the k3s service, to run a VPN.
• No LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the backend-blind announcer that cost this LAN an 8-day Minecraft outage — see conformance/tests/l0-static/l2-announced-vips-serve-from-any-node) and no externalTrafficPolicy trade-off to make at all.
• No double encapsulation. Client UDP lands on the node's own NIC, so the underlay is a flat 1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes MTU = 1420 simply correct.

hostNetwork is permitted here, and not by oversight: platform/kyverno/policies/restrict-host-namespaces.yaml matches namespaceSelector: app-namespace=true only, and names this exact carve-out in its own comment — "platform components that do … e.g. Cilium, node-exporter … live in platform namespaces that do NOT carry the app-namespace label". namespace.yaml is deliberately unlabelled.

The cost, stated rather than buried: a hostNetwork pod has no Cilium endpoint, so no NetworkPolicy constrains this workload — not networkpolicy.yaml, not anything. That file says so at length instead of implying a protection it cannot provide. The boundary around this port is WireGuard's peer authentication, full stop.

Why a DaemonSet

The WAN forward names one node address, by hand, off-repo (same shape as the plex tcp/32400 forward). A single-replica Deployment would float on every reschedule and silently invalidate it. A DaemonSet instead puts an identical listener on all three nodes — every node answers the same keys — so losing the forwarded node is a one-line router change, with no cluster work and no re-issued client profiles. The two idle listeners cost ~15 MiB each.

A pinned LoadBalancer VIP was the considered alternative (it would make the forward immune to node replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node hop into the one path whose failure mode is "I am away from home and cannot get in". Fewer translations win here.

The split that keeps authorisation reviewable

half	where	why
peer public keys + addresses	peers.conf, plaintext in git, as a content-hashed configMapGenerator so an edit rolls the pods	a public key grants nothing by itself, and this makes "who can get into my house" a reviewable PR diff with git log as its audit trail
the server's private key	wg-server-key.sops.yaml, SOPS-encrypted in git	the only real secret (R-SEC-01, spec docs/07). Ships with a REPLACE-ME placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator sops sets it in

The two are joined only at pod start, by the init container, into a medium: Memory emptyDir — so the assembled config holding the private key never touches any node's disk and dies with the pod.

Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with everything else. An out-of-band kubectl create secret — the first shape this component shipped in, corrected after review — would have put the one credential every enrolled device depends on outside the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this key's public half.

Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a new line granting a new device access to the whole LAN.

What a client can reach — and how to narrow it

By design, everything a LAN machine can reach: all of 192.168.2.0/24, the shared Gateway's 16 hostnames, PiHole, the NAS, the router's admin UI, and the internet via the house WAN.

That includes the three k3s nodes themselves — 192.168.2.10/.11/.12, and therefore the kube-apiserver on 6443, kubelet on 10250 and SSH on 22. An enrolled device is as privileged as a machine sitting in the house, which is what "reach the entire local network" asks for; it is recorded here because it is the largest consequence of saying yes to that, and because nobody should discover it later. It is also why a lost device must be revoked promptly (delete its stanza, merge; ~1 min to take effect).

To grant less, narrow it at the client profile first — new-peer.sh --split emits a split-tunnel profile routing only the LAN and the tunnel subnet, so the device keeps its own internet path. To enforce less server-side, add filtering to the PostUp chain in daemonset.yaml (e.g. iptables -A FORWARD -i wg0 -d 192.168.2.10 -j DROP before the accept rules) — the FORWARD chain there is the only enforcement point, since no NetworkPolicy applies.

Operating it

• Enrol a device: ./infra/vpn-ingress/new-peer.sh <name> <octet> → paste the printed [Peer] stanza into peers.conf → PR → merge. Phones scan a QR code; desktops import a .conf.
• Revoke a device: delete its stanza from peers.conf → PR → merge. No other step; the key stops working as soon as the pods roll (the generator's content hash guarantees they do).
• Is it up? kubectl --context k3s-prod -n vpn-ingress get ds wireguard — three Ready.
• Who is connected? kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show (latest handshake under ~3 min means live). Never dump wg showconf — it prints the private key.
• Full bring-up and verification: runbooks/vpn-remote-access.md.

Measured on prod, 2026-10-06 — what is now known rather than assumed

The first real reconcile settled four things this component's design rested on. Three went the right way; one was a defect in my own manifest.

question	answer
Does the kernel module reach the pod?	Yes. ip link add dev wg0 type wireguard succeeds, and /proc/modules lists wireguard, after the node_baseline run. CAP_SYS_MODULE is correctly not needed.
Does allowPrivilegeEscalation: false break the image's s6 init?	No. The container reaches wg-quick normally. The flag stays.
Is the image's iptables backend compatible with the host's?	Yes — both nft. The image ships iptables v1.8.13 (nf_tables) via xtables-nft-multi, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed.
Can the pod set net.ipv4.ip_forward?	No, and it must not try. /proc/sys is mounted read-only under hostNetwork (proc /proc/sys proc ro,…), because those are the host's globals; sysctl -w fails with EPERM whatever the capabilities. A failed PostUp is fatal to wg-quick, so this one line took the whole tunnel down. The node already has it set to 1; the init container now verifies rather than writes.

Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach

A change was proposed (PR #69) on the belief that a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds, and that the VPN node therefore had to be excluded from announcing. That belief is wrong, and the change was withdrawn. The half of it that is true is real but inert, and the distinction matters enough to record, because the wrong version is very easy to re-derive from ip neigh alone.

Measured with all eight cilium-l2announce-* leases held by k3s-prod-1 — the extreme case, after the UPS power-cycle — from the host netns of the wireguard pod on each node:

	k3s-prod-1 (holds all 8)	k3s-prod-2 / -3 (hold none)
ip neigh show 192.168.2.20	FAILED (no lladdr)	lladdr bc:e9:2f:7e:c4:a6 REACHABLE
curl -k https://192.168.2.20/	200, 18 ms	200, 13–40 ms
nc -z 192.168.2.20 443	OK	OK
plex 192.168.2.24:32400	200	200

So the announcer really does not answer its own ARP request — that part of the observation is reproducible. It changes nothing, because no traffic path resolves a LoadBalancer VIP by ARP:

• cilium-dbg status --verbose reports KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1] and Masquerading: BPF [eno1, wg0]. Cilium's LB datapath is attached to wg0 as well as eno1, so a remote client's packet arriving on wg0 for a VIP is DNAT'd to a backend in BPF, before any route or neighbour lookup for the VIP happens.
• cilium-dbg service list on k3s-prod-1 programs 192.168.2.20:443 => 127.0.0.1:13606 — the shared Gateway's backend is a node-local Envoy socket. Every node can serve that VIP without a packet crossing the LAN at all.
• Host-originated traffic is socket-LB'd (Socket LB: Enabled, bpf-lb-sock-hostns-only: true), which is why curl succeeds on the lease holder despite FAILED. The discriminator: a real service port connects, a bogus one (nc -z 192.168.2.20 9999) fails, on all three nodes — that is service-aware DNAT, not wire traffic.

Two tests that look like proof and are not. Both were used to support the withdrawn diagnosis:

• ip neigh on its own. FAILED on the holder is permanent and harmless. It is also an artefact of probing: VIPs answer no ICMP, so a ping leaves a failed entry behind whether or not anything is wrong.
• curl --interface wg0. This fails (rc=28, timeout) from all three nodes, including the two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it discriminates nothing — and it is not the client path either (see below).

Why neither probe can see the client path. A MASQUERADEd remote client's packets are forwarded — netfilter FORWARD, then routing — whereas everything runnable from the pod is socket-originated. The two traverse different hooks, so a host-netns probe cannot measure the forwarded path in either direction: it reports success where the forwarded path might fail, and --interface wg0 reports failure where it does not. The forwarded path can only be measured with a client actually connected, by watching the FORWARD -i wg0 and POSTROUTING … MASQUERADE counters move.

Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses

new-peer.sh has emitted a full tunnel (AllowedIPs = 0.0.0.0/0) by default since the component's first commit; --split is the opt-in for LAN-only. There is no full-tunnel feature to add. The server side was checked end to end and is capable and correct — if a full-tunnel client cannot reach the internet, nothing in the list below is the reason:

checked	result on all three nodes
net.ipv4.ip_forward (read from /proc/sys, never written)	1
iptables -t nat -S POSTROUTING	carries -s 10.90.0.0/24 ! -o wg0 -j MASQUERADE — NATs to any outbound interface, so internet NAT needs no change
CILIUM_POST_nat, which precedes it in the chain	empty, 0 packets — it cannot short-circuit the MASQUERADE
iptables -L FORWARD	-i wg0 and -o wg0 ACCEPT both present
default route	default via 192.168.2.1 dev eno1 onlink, intact
node's own internet reachability	http://1.1.1.1/ returns 301

A full tunnel egresses the HOUSE WAN, not PIA. This was the trap worth checking, and it is closed: platform/vpn-egress/egress-steering.yaml's CiliumEgressGatewayPolicy selects by podSelector on egress-class: vpn, and this DaemonSet is deliberately labelled egress-class: internet, so it is not selected. Forwarded wg0 traffic is not pod-identity traffic at all, and tun0 exists on k3s-prod-3 only (10.150.192.3/18) — the egress-gateway node. A remote client's apparent public address is therefore the home WAN IP, which is the intended behaviour and the thing to state when enrolling a device.

Not measurable without a connected client. At the time of writing wg show wg0 listed the one enrolled peer with no endpoint and no handshake on every node, and both the FORWARD -i wg0 and the MASQUERADE counters read 0 packets — so no client traffic had been forwarded since the power-cycle, and which node the router forwards udp/51820 to was undeterminable. Only the forwarded node shows a peer with an endpoint and a recent latest handshake.

Prerequisites outside this directory

1. The wireguard kernel module on every node. Added to node_baseline_kernel_modules (infra/ansible/roles/node_baseline/defaults/main.yml); the operator runs the role. The pod has NET_ADMIN but deliberately not SYS_MODULE, so it cannot load the module itself — it CrashLoops with a clear wg-quick error instead of quietly degrading to userspace WireGuard.
2. The server private key. wg-server-key.sops.yaml is committed, encrypted, with a REPLACE-ME placeholder, and both overlays already carry its decryption patch — so the wiring is complete and the only outstanding act is replacing the value (runbook step 2). Until then the DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the value with wg pubkey, which rejects the placeholder, an empty value, a public key pasted by mistake, and a Secret that arrived as undecrypted ENC[...] ciphertext. That last case is the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves every Flux condition green.
3. The WAN port-forward, udp/51820 → one node address. Hand-made on the SOHO router, off-repo, and the half CI can never check.

Not built (deliberate, with reasons)

• No metrics / no peer-level alerting. A WireGuard exporter would expose wireguard_latest_handshake_seconds per peer — genuinely useful for spotting unexpected activity — but it needs a second NET_ADMIN container in the pod that terminates the internet-facing tunnel, and that trade was not worth making unverified. Pod-down is already covered generically by KubeDeploymentReplicasMismatch/KubePodNotReady from kube-prometheus-stack. This is the top follow-up.
• No preshared keys. They would add a symmetric post-quantum hedge on top of the Noise handshake — and would be a real secret, forcing peers.conf into SOPS and costing the reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the wg-server-key Secret.
• No IPv6. The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6 still reaches IPv6 sites outside the tunnel. Functionally harmless, a privacy leak if the point was to hide the client's network. Fix is an fd.. ULA on wg0 plus ::/0 in the client's AllowedIPs; not done because nothing here needs it.
• No in-cluster-service reach beyond what the LAN has. Clients reach cluster apps the way a LAN browser does — through the Gateway VIP. Pod IPs and 10.43.0.0/16 service IPs are not routed into the tunnel.