feat(vpn-ingress): WireGuard remote access on udp/51820 (hostNetwork, declarative peers)
gate / gate (pull_request) Failing after 1m48s
gate / conformance (pull_request) Has been skipped

Lets the operator's Android / Linux / Windows / macOS devices dial in from
anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus
internet egress out the house WAN. Opposite direction to platform/vpn-egress,
which sends pods OUT through PIA; the two share nothing but the word "VPN".

WireGuard because it is the only option with a first-party client on all four
named platforms, and because its handshake MAC is keyed with the responder's
own public key: an unauthenticated packet gets NO reply, so udp/51820 is
indistinguishable from a closed port. That, not "it is encrypted", is what
makes an unfiltered WAN port defensible, and why there is no source allowlist.

hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT
clients to its own pod IP, so every remote client would carry that pod's Cilium
identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from
Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway
preserves the original source identity and enforces the destination's ingress
policy. The obvious build would have shipped a healthy-looking tunnel on which
every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve.
In the host netns traffic is SNATed to the node and arrives as host/remote-node,
exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the
L2-announcement lease lottery, and double encapsulation (hence MTU 1420).
Permitted by restrict-host-namespaces' own carve-out for platform namespaces.
The cost is stated in networkpolicy.yaml rather than papered over: no
NetworkPolicy constrains a hostNetwork pod.

A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule;
all three nodes answer the same keys, so failover is a one-line router change.

Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my
house" a reviewable PR diff. The only secret -- the server's private key -- is
an out-of-band Secret: an agent must not mint a long-lived private key, and the
two halves are joined only at pod start into a tmpfs that dies with the pod.

Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain
(kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md.

Touches clusters/** , so ci/approval-policy.sh will block pending the
operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-05 22:06:18 -05:00
1 parent 22c94effff
commit 1a67fc1b3c
16 files changed
+1391 -1

No files matched your search

+199
View File
@@ -0,0 +1,199 @@
# platform/vpn-ingress — WireGuard remote access (north-south INGRESS)
Lets the operator's own devices — Android, Linux, Windows, macOS, iOS — dial in from anywhere on
the internet and behave as though plugged into the home LAN: full reach to `192.168.2.0/24` and
egress out the house WAN.
> **This is not `platform/vpn-egress`.** They are opposite directions and share nothing but the
> word "VPN". `vpn-egress` pushes *pods out* to the internet through a PIA tunnel (ADR-0015,
> R-NET-05/06/07). This component lets *remote clients in*. Changing one never affects the other.
| | |
|---|---|
| Transport | WireGuard, **udp/51820**, kernel implementation |
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** |
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
| Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
## Why WireGuard
The requirement names four client platforms. WireGuard is the only option that has a *first-party*
client on all of them (official apps for Android, iOS, Windows and macOS; in-kernel on Linux since
5.6, so a Linux client needs no userspace daemon at all). It is also the smallest thing to expose:
one UDP port, no cipher negotiation, no PKI, no user database, no TLS stack.
OpenVPN would have needed a CA, per-device certificates and a revocation list — three pieces of
long-lived state this repo would then have to protect and rotate. IKEv2/IPsec has native clients but
is painful to run in a container and brittle behind double NAT. Neither buys anything here.
### Why this is safe to expose to the whole internet
The honest answer is not "because it is encrypted" — it is the handshake's failure mode:
**An unauthenticated packet gets no reply at all.** WireGuard's first handshake message carries a
MAC keyed with the *responder's own public key*. If the sender does not already know it, the packet
is dropped silently — no error, no reset, no banner, nothing a scanner can fingerprint. From the
outside, udp/51820 is indistinguishable from a closed port. There is no login prompt to brute-force,
no version string to match against a CVE, and no amplification to abuse.
That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
would also have made roaming clients unusable — a phone's address changes with every cell tower).
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a
resolver answers anybody, so it must be scoped by address. WireGuard answers nobody.
What it does **not** protect against is a stolen device — anyone holding a device's private key
*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).
## Architecture, and the one decision that drives everything
```
phone on 5G SOHO router k3s-prod-N (host netns)
┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐
│ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │
│ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │
└──────────────┘ internet └───────────────┘ │ SNAT to the node's addr │
└─────────────┬──────────────┘
│ as `host`/`remote-node`
┌─────────────────────────────────┼──────────────────────┐
▼ ▼ ▼
LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet
(NAS, router, printers) PiHole DNS VIP .23
```
### `hostNetwork: true` is required, not convenient
This is the only genuinely interesting decision in the component, and it was forced by a *measured*
property of this cluster rather than chosen for convenience.
In Cilium, what a packet may reach is decided by the **security identity of its source**. A
WireGuard server running as an ordinary pod must SNAT client traffic to its own pod IP (Cilium drops
a packet whose source is not the endpoint's own address), so every remote client would have arrived
everywhere wearing *that pod's* identity. On 2026-10-04 this cluster measured what that costs
(`decisions/2026-10-04-plex-advertise-plexdirect.md`): a request to the shared Gateway VIP **from
inside a pod returns `403 Access denied` from Envoy for all 16 Gateway hostnames**, because Cilium's
Gateway preserves the original source identity and enforces the *destination's* ingress policy —
and every app admits `reserved:ingress` plus named peers, never an arbitrary pod. LAN browsers work
only because they arrive as `reserved:world` / `reserved:host`.
So the pod-networked version of this component would have produced a tunnel that came up healthy,
reported connected peers, routed packets — and 403'd **every single cluster web UI**: plex, gitea,
keycloak, nextcloud, grafana, headlamp, the apex. `*.home.keithjackson.dev` would not even have
resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately omits the `cluster`
entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
manifests would have hinted at it.
In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the
node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
byte-for-byte what a LAN client looks like. Three further problems vanish with it:
* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
`/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
pod-netns build would have needed `--allowed-unsafe-sysctls` on all three kubelets — a
node-restart change to the k3s service, to run a VPN.
* **No LoadBalancer VIP**, so no `CiliumL2AnnouncementPolicy` lease lottery (the backend-blind
announcer that cost this LAN an 8-day Minecraft outage — see
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
`externalTrafficPolicy` trade-off to make at all.
* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct.
`hostNetwork` is *permitted* here, and not by oversight:
`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
app-namespace=true` only, and names this exact carve-out in its own comment — "platform components
that do … e.g. Cilium, node-exporter … live in platform namespaces that do NOT carry the
app-namespace label". [`namespace.yaml`](./namespace.yaml) is deliberately unlabelled.
**The cost, stated rather than buried:** a `hostNetwork` pod has no Cilium endpoint, so **no
NetworkPolicy constrains this workload** — not [`networkpolicy.yaml`](./networkpolicy.yaml), not
anything. That file says so at length instead of implying a protection it cannot provide. The
boundary around this port is WireGuard's peer authentication, full stop.
### Why a DaemonSet
The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
re-issued client profiles. The two idle listeners cost ~15 MiB each.
A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
hop into the one path whose failure mode is *"I am away from home and cannot get in"*. Fewer
translations win here.
### The split that keeps authorisation reviewable
| half | where | why |
|---|---|---|
| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) |
The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
so the assembled config holding the private key never touches any node's disk and dies with the pod.
Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
new line granting a new device access to the whole LAN.
## What a client can reach — and how to narrow it
By design, **everything a LAN machine can reach**: all of `192.168.2.0/24`, the shared Gateway's 16
hostnames, PiHole, the NAS, the router's admin UI, and the internet via the house WAN.
**That includes the three k3s nodes themselves** — `192.168.2.10/.11/.12`, and therefore the
kube-apiserver on 6443, kubelet on 10250 and SSH on 22. An enrolled device is as privileged as a
machine sitting in the house, which is what "reach the entire local network" asks for; it is
recorded here because it is the largest consequence of saying yes to that, and because nobody should
discover it later. **It is also why a lost device must be revoked promptly** (delete its stanza,
merge; ~1 min to take effect).
To grant less, narrow it at the **client** profile first — `new-peer.sh --split` emits a
split-tunnel profile routing only the LAN and the tunnel subnet, so the device keeps its own
internet path. To enforce less *server-side*, add filtering to the `PostUp` chain in
[`daemonset.yaml`](./daemonset.yaml) (e.g. `iptables -A FORWARD -i wg0 -d 192.168.2.10 -j DROP`
before the accept rules) — the FORWARD chain there is the only enforcement point, since no
NetworkPolicy applies.
## Operating it
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
(`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
* **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
## Prerequisites outside this directory
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
(`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
`NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start —
the correct fail-closed state for an unenrolled VPN.
3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
and the half CI can never check.
## Not built (deliberate, with reasons)
* **No metrics / no peer-level alerting.** A WireGuard exporter would expose
`wireguard_latest_handshake_seconds` per peer — genuinely useful for spotting unexpected
activity — but it needs a second `NET_ADMIN` container in the pod that terminates the
internet-facing tunnel, and that trade was not worth making unverified. Pod-down is already
covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
kube-prometheus-stack. **This is the top follow-up.**
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
`wg-server-key` Secret.
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point
was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's
`AllowedIPs`; not done because nothing here needs it.
* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
into the tunnel.
+291
View File
@@ -0,0 +1,291 @@
---
# =================================================================================================
# WireGuard remote-access listener — udp/51820 in the HOST network namespace, on every node.
# =================================================================================================
# WHY WireGuard AND NOT OpenVPN/IPsec. The requirement is "Android, Linux, Windows and other
# computing clients", from anywhere on the internet, securely. WireGuard is the only option here
# with a first-party client on all four platforms (official apps for Android/iOS/Windows/macOS, and
# in-kernel on Linux since 5.6 — so Linux clients need no userspace daemon at all). It is also the
# smallest attack surface of the three: one UDP port, ~4k lines of kernel code, no cipher
# negotiation, no PKI, no user database, and — the property that actually matters for a port
# exposed to the whole internet — it NEVER REPLIES to an unauthenticated packet. See README.
#
# -------------------------------------------------------------------------------------------------
# hostNetwork: true — REQUIRED, not convenient. This is THE decision in this file.
# -------------------------------------------------------------------------------------------------
# A VPN client must behave like a machine plugged into the LAN. In Cilium, what a packet is ALLOWED
# to reach is decided by the SECURITY IDENTITY of its source, and after any in-pod SNAT that
# identity is the VPN pod's — not `world`. That distinction is not academic; it was MEASURED on
# this cluster on 2026-10-04 (decisions/2026-10-04-plex-advertise-plexdirect.md): a curl of the
# shared Gateway VIP from inside a pod returns `403 Access denied` from Envoy for **all 16**
# Gateway hostnames, because Cilium's Gateway preserves the original source identity and enforces
# the DESTINATION's ingress CiliumNetworkPolicy, and every app admits `reserved:ingress` plus named
# peers — never an arbitrary pod. LAN browsers are allowed only because they arrive as
# `reserved:world` / `reserved:host`.
#
# So a pod-networked WireGuard server would have handed remote clients a tunnel on which every
# single cluster web UI returned 403 — plex, gitea, keycloak, nextcloud, grafana, headlamp, the
# apex, all of them — while `wg show` reported a perfectly healthy tunnel. PiHole would have failed
# the same way and worse: apps/wave-0/pihole-dns.yaml's ingress policy deliberately OMITS the
# `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either.
# The VPN would have looked up and been useless, which is the worst failure shape available.
#
# In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the
# NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
# exactly what a LAN client looks like. Three further problems disappear with it:
# * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to
# /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
# is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
# required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
# * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
# this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
# * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
# flat 1500-byte LAN path, not Cilium's 1450-byte VXLAN.
#
# hostNetwork is PERMITTED here, and not by accident: platform/kyverno/policies/
# restrict-host-namespaces.yaml matches `namespaceSelector: app-namespace=true` only, and names
# this exact carve-out in its own comment ("platform components that do -- e.g. Cilium,
# node-exporter -- live in platform namespaces that do NOT carry the app-namespace label").
# namespace.yaml is deliberately unlabelled. THE COST IS REAL AND IS STATED: a hostNetwork pod
# bypasses the Cilium datapath, so NO NetworkPolicy constrains this workload (see
# networkpolicy.yaml, which says so rather than pretending otherwise). The boundary protecting
# this port is WireGuard's own cryptographic peer authentication, not the CNI.
#
# -------------------------------------------------------------------------------------------------
# A DaemonSet, not a single-replica Deployment.
# -------------------------------------------------------------------------------------------------
# The WAN port-forward on the SOHO router points at ONE node address, by hand, off-repo (same shape
# as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to
# whichever node the scheduler picked and silently invalidate that forward on every reschedule. A
# DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of
# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no
# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own
# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each.
#
# All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the
# DaemonSet to cover the cluster; add them if a node is ever tainted.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: wireguard
namespace: vpn-ingress
labels:
app.kubernetes.io/name: vpn-ingress
spec:
selector:
matchLabels:
app.kubernetes.io/name: vpn-ingress
# One node at a time, so a rollout never takes every listener down at once. The node named by the
# router's forward does drop for a few seconds during its own replacement; clients re-handshake
# automatically (WireGuard retries every 5s, and rekeys every 2 minutes regardless).
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
metadata:
labels:
app.kubernetes.io/name: vpn-ingress
# Declared for documentation and for symmetry with the app tier. It MUST be `internet`, and
# the value is load-bearing rather than cosmetic: platform/vpn-egress/egress-steering.yaml's
# CiliumEgressGatewayPolicy selects `egress-class: vpn` pods CLUSTER-WIDE, with no namespace
# scope. Setting `vpn` here would steer every remote client's internet traffic out the PIA
# tunnel instead of the house WAN — the opposite of the requirement, and it would look like
# a working VPN with a mysterious foreign exit IP. (The selector matches on the pod label,
# so hostNetwork does not exempt this pod from being selected.)
homelab.keithjackson.dev/egress-class: internet
# WireGuard authenticates its own peers by public key; there is no UI and no IdP hop.
homelab.keithjackson.dev/auth-mode: native
spec:
# See the long rationale at the head of this file. ClusterFirstWithHostNet keeps in-cluster
# DNS working for the pod itself despite hostNetwork (without it the pod inherits the NODE's
# resolv.conf, which points at public resolvers).
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
automountServiceAccountToken: false
# ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
# The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
# joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
# key is therefore never written to any node's disk, never committed, and never logged.
initContainers:
- name: assemble-config
image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124
imagePullPolicy: IfNotPresent
# POSIX sh. Note on $ handling: this script is rendered through Flux's envsubst (the
# `vpn-ingress` Kustomization declares postBuild.substituteFrom), so `${WG_CLIENT_CIDR}`
# below IS substituted on purpose — it is a declared cluster-vars knob. Every SHELL
# variable is written bare (`$net`, `$(cmd)`), which envsubst leaves untouched. There is
# deliberately NO braced shell-variable form anywhere in here: braces are what envsubst
# consumes, so a shell variable written that way silently renders to the empty string.
# (ci: flux-postbuild-substitution enforces this, and caught this very comment when it
# first spelled the hazard out literally — the check reads comments too, correctly.)
command:
- /bin/sh
- -c
- |
set -eu
umask 077
mkdir -p /config/wg_confs
# Derive the server's own tunnel address from the single CIDR knob, so there is no
# second knob to keep in agreement: 10.90.0.0/24 -> 10.90.0.1/24.
net=${WG_CLIENT_CIDR}
mask=$(echo "$net" | cut -d/ -f2)
base=$(echo "$net" | cut -d/ -f1 | cut -d. -f1-3)
# The key arrives as a Secret file; strip any trailing newline the operator's shell
# added, or wg-quick rejects the config with "Key is not the correct length or format".
key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey)
if [ -z "$key" ]; then
echo "FATAL: /run/wg-server-key/privatekey is empty." >&2
echo " See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2
exit 1
fi
{
echo '[Interface]'
echo "Address = $base.1/$mask"
echo 'ListenPort = 51820'
# 1420 = 1500 (flat LAN/WAN path to the node's own NIC) - 60 bytes of WireGuard
# encapsulation (20 IPv4 + 8 UDP + 32 WireGuard). This is also what wg-quick would
# auto-derive, but it is pinned because the symptom of getting it wrong is the
# nastiest kind: handshakes and SSH work while large packets (HTTPS, file copies)
# hang. Inner packets stay under Cilium's 1450-byte VXLAN MTU too, so reaching a
# cluster pod does not fragment either.
echo 'MTU = 1420'
printf 'PrivateKey = %s\n' "$key"
# ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
# ip_forward is already 1 from node_baseline; re-asserted so this component does not
# silently depend on another role's sysctl file surviving.
echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1'
# SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
# reply to 10.90.0.x, find no route, hand the packet to the router, and the router
# would drop it — so every connection would hang rather than fail. Matching on
# `! -o wg0` rather than a named `-o eth0` keeps this independent of the node's NIC
# name. Each rule is add-if-absent so a pod killed without PostDown does not
# accumulate duplicates in the host's tables across restarts.
echo "PostUp = iptables -t nat -C POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || iptables -t nat -A POSTROUTING -s $net ! -o wg0 -j MASQUERADE"
echo 'PostUp = iptables -C FORWARD -i wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -i wg0 -j ACCEPT'
echo 'PostUp = iptables -C FORWARD -o wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -o wg0 -j ACCEPT'
echo "PostDown = iptables -t nat -D POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || true"
echo 'PostDown = iptables -D FORWARD -i wg0 -j ACCEPT 2>/dev/null || true'
echo 'PostDown = iptables -D FORWARD -o wg0 -j ACCEPT 2>/dev/null || true'
echo ''
} > /config/wg_confs/wg0.conf
cat /run/wg-peers/peers.conf >> /config/wg_confs/wg0.conf
chmod 600 /config/wg_confs/wg0.conf
# Report the SHAPE of the result, never its content — the file now holds the server's
# private key, and a log line is a transcript (CLAUDE.md §6).
echo "wg0.conf assembled: $(grep -c '^\[Peer\]' /config/wg_confs/wg0.conf || true) peer(s), server $base.1/$mask, udp/51820."
securityContext:
runAsUser: 0
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
memory: 64Mi
volumeMounts:
- { name: wg-config, mountPath: /config }
- { name: wg-server-key, mountPath: /run/wg-server-key, readOnly: true }
- { name: wg-peers, mountPath: /run/wg-peers, readOnly: true }
containers:
- name: wireguard
# linuxserver/wireguard — provides wg, wg-quick and iptables on a maintained, multi-arch
# base. Pinned to the full upstream+build tag (1.0.20260223, published 2026-10-01) rather
# than `latest`, per the crafty/oauth2-proxy convention; Renovate maintains it.
#
# NO `PEERS` ENV IS SET, DELIBERATELY. With PEERS the image runs in "server mode" and
# GENERATES keypairs at runtime — which would put the authoritative peer list on a
# container filesystem instead of in git, and (worse) print client private keys and QR
# codes into the POD LOG, where they would be readable by anyone with log access and
# shipped to Loki. Supplying our own /config/wg_confs/wg0.conf selects the image's
# bring-up-what-you-are-given path instead.
image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124
imagePullPolicy: IfNotPresent
env:
- name: TZ
value: "America/Chicago"
ports:
# hostNetwork, so containerPort IS the node port. Declared for documentation and so
# `kubectl get pod -o wide` shows what this listener owns.
- { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
securityContext:
# NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and
# write the two netfilter rules above. Everything else is dropped, and the container is
# NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
# SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
# (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
# never needs to load a kernel module. If the module is missing the pod CrashLoops with
# a clear wg-quick error rather than quietly falling back to slow userspace WireGuard.
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
add:
- NET_ADMIN
# readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
# /var during startup. Same deferral as crafty/headlamp.
resources:
requests:
cpu: 25m
memory: 32Mi
limits:
# Kernel WireGuard does its crypto in the KERNEL, not in this container, so throughput
# does not consume the container's budget. The limit covers s6 plus the tmpfs holding
# wg0.conf (a memory-medium emptyDir counts against this).
memory: 128Mi
# `wg show wg0` exits non-zero when the interface is absent, which is the one failure that
# matters and the one that a port probe cannot see: a crashed or never-created wg0 leaves
# udp/51820 silent, and WireGuard is silent to unauthenticated traffic ANYWAY, so "nothing
# answers" is indistinguishable from "working" from the outside. This probe is the only
# thing that tells the difference.
startupProbe:
exec:
command: ["sh", "-c", "wg show wg0 >/dev/null"]
periodSeconds: 5
failureThreshold: 24
livenessProbe:
exec:
command: ["sh", "-c", "wg show wg0 >/dev/null"]
periodSeconds: 30
failureThreshold: 3
volumeMounts:
- { name: wg-config, mountPath: /config }
# Read-only so the image can SEE whether the kernel module is present and fail with a
# real message. It cannot load one (no SYS_MODULE) — that is the node's job.
- { name: lib-modules, mountPath: /lib/modules, readOnly: true }
volumes:
# tmpfs. The assembled wg0.conf contains the server's private key, so it must never land on
# a node's disk — `medium: Memory` guarantees that and guarantees it is gone when the pod is.
- name: wg-config
emptyDir:
medium: Memory
sizeLimit: 1Mi
# The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by
# name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this
# tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key
# that passed through a session transcript would have to be treated as compromised.
# DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace
# and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy
# in the operator's password manager as well — restoring it is what makes every already
# enrolled client work again without re-enrolling.
- name: wg-server-key
secret:
secretName: wg-server-key
defaultMode: 0400
items:
- { key: privatekey, path: privatekey }
- name: wg-peers
configMap:
name: wg-peers
- name: lib-modules
hostPath:
path: /lib/modules
type: Directory
+30
View File
@@ -0,0 +1,30 @@
---
# WireGuard remote-access VPN (north-south INGRESS). Reconciled by its OWN non-gating post-platform
# Flux Kustomization (clusters/base/vpn-ingress.yaml, dependsOn platform-config, wait:false) — the
# same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be
# legitimately unhealthy must never sit in the gating `platform` tier, where one component's
# unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating
# stall"). That rationale is unusually concrete here: until the operator creates the `wg-server-key`
# Secret (runbooks/vpn-remote-access.md step 1) this DaemonSet CANNOT start, so putting it in the
# gating tier would have taken the whole platform down on merge.
#
# NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
# The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
- peers.yaml
# The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
- daemonset.yaml
# Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
# says so at length rather than implying a protection it cannot provide.
- networkpolicy.yaml
#
# DELIBERATELY ABSENT: a Secret. The server's private key (`wg-server-key`) is created out of band
# by the operator and referenced BY NAME from daemonset.yaml. There is no `*.sops.yaml` here, so
# this component needs NO SOPS decryption stanza in clusters/dev or clusters/prod — unlike
# crafty/headlamp/velero. If that key is ever migrated into SOPS, BOTH halves must land together:
# the file here, and a `decryption:` patch naming the `vpn-ingress` Kustomization in each overlay.
# Without the patch, kustomize hands Flux the raw ciphertext and the pod mounts a key full of
# ENC[...] — which applies cleanly and fails only at handshake time.
+26
View File
@@ -0,0 +1,26 @@
---
# WireGuard REMOTE-ACCESS VPN (north-south INGRESS) — the counterpart to platform/vpn-egress.
#
# Do not confuse the two. They are opposite directions and share nothing but the word "VPN":
# * platform/vpn-egress — pods OUT to the internet through a PIA tunnel (ADR-0015, R-NET-05/06/07).
# * platform/vpn-ingress — remote CLIENTS IN from the internet to this LAN. THIS directory.
#
# NOTE: intentionally NOT homelab.keithjackson.dev/app-namespace=true. This is PLATFORM network
# infrastructure (like cilium / vpn-egress), not an application, so Kyverno does NOT generate the app
# default-deny baseline here and the app-scoped auth-mode / egress-class / host-namespace policies do
# not apply — every Kyverno policy in platform/kyverno/policies matches namespaceSelector
# app-namespace=true. THAT EXEMPTION IS LOAD-BEARING HERE, not incidental: this workload runs with
# `hostNetwork: true`, which `restrict-host-namespaces` forbids in an app namespace. It is permitted
# here by that policy's OWN stated carve-out — "platform components that do [need host namespaces] --
# e.g. Cilium, node-exporter -- live in platform namespaces that do NOT carry the app-namespace
# label, so they are out of scope". daemonset.yaml documents why hostNetwork is REQUIRED rather than
# convenient; it is the single most consequential decision in this component.
#
# NO `gateway-access` label. This component serves no HTTP and attaches no HTTPRoute to the shared
# Gateway — it is a raw UDP listener. There is deliberately nothing here for the L7 Gateway to route.
apiVersion: v1
kind: Namespace
metadata:
name: vpn-ingress
labels:
app.kubernetes.io/name: vpn-ingress
+56
View File
@@ -0,0 +1,56 @@
---
# =================================================================================================
# READ THIS BEFORE TRUSTING THIS FILE: it does NOT constrain the WireGuard listener.
# =================================================================================================
# Every other platform namespace in this repo (crafty, headlamp, homepage, runbooks) ships an
# explicit default-deny baseline as defense in depth, because Kyverno's generator only covers
# `app-namespace=true` namespaces. This file keeps that convention — but it would be dishonest to
# leave it at that, because the convention does not actually bite here:
#
# THE `wireguard` DaemonSet RUNS WITH hostNetwork: true, SO IT HAS NO CILIUM ENDPOINT, AND NO
# NetworkPolicy OR CiliumNetworkPolicy APPLIES TO IT — NEITHER THIS ONE NOR ANY OTHER.
#
# That is the documented reason restrict-host-namespaces forbids hostNetwork in app namespaces at
# all ("a pod on hostNetwork bypasses the CNI (Cilium) datapath and therefore ESCAPES the
# default-deny NetworkPolicy fabric -- it is the most direct way to defeat R-NET-01"). Here that
# escape is the deliberate, required design (daemonset.yaml explains why: without it, remote
# clients carry a pod identity and are 403'd by every Gateway hostname). A CiliumNetworkPolicy
# admitting `world` to udp/51820 would therefore be theatre: it would read as the control that
# protects this port, and it would be enforcing nothing. So there isn't one.
#
# WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
# NO REPLY — see README "Why this is safe to expose". This is the real control.
# 2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
# 3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement
# ("clients should reach the entire local network"). README "What a client can reach" states
# the consequence in full, including reach to the nodes' own control plane, and gives the
# narrowing recipe for an operator who wants less than that.
#
# WHAT THIS FILE IS STILL FOR: it is the namespace's standing posture for any FUTURE pod-networked
# workload added here (a metrics exporter is the likely one — README "Not built"). Without it, such
# a pod would be born unrestricted, because Kyverno will not generate a baseline in a namespace
# that is not labelled `app-namespace=true`, and this one deliberately is not.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: vpn-ingress
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
+73
View File
@@ -0,0 +1,73 @@
---
# ============================================================================================
# THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
# ============================================================================================
# WireGuard has no usernames, no passwords and no PKI. Authorisation is exactly one thing: a
# device's PUBLIC key appears in this file, or it cannot complete a handshake. An unlisted key
# gets no reply at all — not a rejection, silence (see README "Why this is safe to expose").
# Revoking a device is therefore deleting its stanza, and nothing else.
#
# WHY PLAINTEXT, AND WHY THIS IS NOT A SECRET LEAK. A WireGuard peer's PUBLIC key is public by
# construction — it is the half you are supposed to hand to the server, and holding it grants
# nothing. The only secret in the whole scheme is the SERVER's private key, which is NOT here:
# it lives in the `wg-server-key` Secret and is assembled into the running config at pod start
# (daemonset.yaml). Keeping the register in cleartext buys the property that matters most for an
# access-control list: **"who can get into my house" is reviewable in a pull-request diff**, and
# its history is `git log` on this file. A SOPS-encrypted register would hide exactly the change
# a reviewer most needs to see — a new line granting a new device entry to the whole LAN.
#
# This is the same division of labour the repo already uses for the opposite case: a Secret's KEY
# NAMES are cleartext while values are encrypted (.sops.yaml `encrypted_regex`), so diffs stay
# readable. Here the whole file is the key-name half.
#
# ADDING A DEVICE — never hand-roll this, and never generate a key in a chat transcript.
# ./infra/vpn-ingress/new-peer.sh <device-name> <last-octet>
# It generates the keypair ON THE OPERATOR'S MACHINE, writes/renders the client profile (and a
# QR code for phones), prints ONLY the public half for pasting below, and never transmits or
# logs the private half. Then: add the stanza, PR, merge; Flux rolls the DaemonSet.
#
# ADDRESS ALLOCATION. ${WG_CLIENT_CIDR} is the tunnel subnet; .1 is the server (every node).
# Give each device ONE address as a /32 in AllowedIPs. A /32 is not a formality — on the SERVER
# side AllowedIPs is the cryptokey ROUTING table AND the inbound source filter: a peer may only
# send from, and receive traffic for, the addresses listed in its own stanza. A sloppy /24 here
# would let any one device impersonate every other device's tunnel address.
#
# .1 the server (this DaemonSet, on every node)
# .2 - .254 devices, one each, allocated below. Keep this list dense and in order so the
# next free octet is obvious; never re-use an octet after revoking a device
# (an old client profile would silently start working again as someone else).
#
# PRESHARED KEYS are deliberately NOT used. They add a symmetric post-quantum hedge on top of
# the Noise handshake and would be a genuine secret — which would force this register into SOPS
# and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
# belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
apiVersion: v1
kind: ConfigMap
metadata:
name: wg-peers
namespace: vpn-ingress
labels:
app.kubernetes.io/name: vpn-ingress
data:
# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
#
# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
# (`wg show` reports an interface with no peers).
#
# COPY THIS TEMPLATE, uncomment, one block per device:
#
# # pixel-8 — Keith's phone, enrolled 2026-10-05
# [Peer]
# PublicKey = <the public key new-peer.sh printed>
# AllowedIPs = 10.90.0.2/32
#
# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
# it an incident responder cannot tell which stanza to delete.
peers.conf: |
# No peers enrolled yet. See the template in this file's comments and
# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.