--- # ================================================================================================= # WireGuard remote-access listener — udp/51820 in the HOST network namespace, on every node. # ================================================================================================= # WHY WireGuard AND NOT OpenVPN/IPsec. The requirement is "Android, Linux, Windows and other # computing clients", from anywhere on the internet, securely. WireGuard is the only option here # with a first-party client on all four platforms (official apps for Android/iOS/Windows/macOS, and # in-kernel on Linux since 5.6 — so Linux clients need no userspace daemon at all). It is also the # smallest attack surface of the three: one UDP port, ~4k lines of kernel code, no cipher # negotiation, no PKI, no user database, and — the property that actually matters for a port # exposed to the whole internet — it NEVER REPLIES to an unauthenticated packet. See README. # # ------------------------------------------------------------------------------------------------- # hostNetwork: true — REQUIRED, not convenient. This is THE decision in this file. # ------------------------------------------------------------------------------------------------- # A VPN client must behave like a machine plugged into the LAN. In Cilium, what a packet is ALLOWED # to reach is decided by the SECURITY IDENTITY of its source, and after any in-pod SNAT that # identity is the VPN pod's — not `world`. That distinction is not academic; it was MEASURED on # this cluster on 2026-10-04 (decisions/2026-10-04-plex-advertise-plexdirect.md): a curl of the # shared Gateway VIP from inside a pod returns `403 Access denied` from Envoy for **all 16** # Gateway hostnames, because Cilium's Gateway preserves the original source identity and enforces # the DESTINATION's ingress CiliumNetworkPolicy, and every app admits `reserved:ingress` plus named # peers — never an arbitrary pod. LAN browsers are allowed only because they arrive as # `reserved:world` / `reserved:host`. # # So a pod-networked WireGuard server would have handed remote clients a tunnel on which every # single cluster web UI returned 403 — plex, gitea, keycloak, nextcloud, grafana, headlamp, the # apex, all of them — while `wg show` reported a perfectly healthy tunnel. PiHole would have failed # the same way and worse: apps/wave-0/pihole-dns.yaml's ingress policy deliberately OMITS the # `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either. # The VPN would have looked up and been useless, which is the worst failure shape available. # # In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the # NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` — # exactly what a LAN client looks like. Three further problems disappear with it: # * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to # /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward` # is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have # required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change). # * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost # this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all. # * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a # flat 1500-byte LAN path, not Cilium's 1450-byte VXLAN. # # hostNetwork is PERMITTED here, and not by accident: platform/kyverno/policies/ # restrict-host-namespaces.yaml matches `namespaceSelector: app-namespace=true` only, and names # this exact carve-out in its own comment ("platform components that do -- e.g. Cilium, # node-exporter -- live in platform namespaces that do NOT carry the app-namespace label"). # namespace.yaml is deliberately unlabelled. THE COST IS REAL AND IS STATED: a hostNetwork pod # bypasses the Cilium datapath, so NO NetworkPolicy constrains this workload (see # networkpolicy.yaml, which says so rather than pretending otherwise). The boundary protecting # this port is WireGuard's own cryptographic peer authentication, not the CNI. # # ------------------------------------------------------------------------------------------------- # A DaemonSet, not a single-replica Deployment. # ------------------------------------------------------------------------------------------------- # The WAN port-forward on the SOHO router points at ONE node address, by hand, off-repo (same shape # as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to # whichever node the scheduler picked and silently invalidate that forward on every reschedule. A # DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of # them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no # new client profiles — every node answers the same keys. Each node binds udp/51820 in its own # netns, so there is no conflict, and the two idle listeners cost ~15 MiB each. # # All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the # DaemonSet to cover the cluster; add them if a node is ever tainted. apiVersion: apps/v1 kind: DaemonSet metadata: name: wireguard namespace: vpn-ingress labels: app.kubernetes.io/name: vpn-ingress spec: selector: matchLabels: app.kubernetes.io/name: vpn-ingress # One node at a time, so a rollout never takes every listener down at once. The node named by the # router's forward does drop for a few seconds during its own replacement; clients re-handshake # automatically (WireGuard retries every 5s, and rekeys every 2 minutes regardless). updateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 template: metadata: labels: app.kubernetes.io/name: vpn-ingress # Declared for documentation and for symmetry with the app tier. It MUST be `internet`, and # the value is load-bearing rather than cosmetic: platform/vpn-egress/egress-steering.yaml's # CiliumEgressGatewayPolicy selects `egress-class: vpn` pods CLUSTER-WIDE, with no namespace # scope. Setting `vpn` here would steer every remote client's internet traffic out the PIA # tunnel instead of the house WAN — the opposite of the requirement, and it would look like # a working VPN with a mysterious foreign exit IP. (The selector matches on the pod label, # so hostNetwork does not exempt this pod from being selected.) homelab.keithjackson.dev/egress-class: internet # WireGuard authenticates its own peers by public key; there is no UI and no IdP hop. homelab.keithjackson.dev/auth-mode: native spec: # See the long rationale at the head of this file. ClusterFirstWithHostNet keeps in-cluster # DNS working for the pod itself despite hostNetwork (without it the pod inherits the NODE's # resolv.conf, which points at public resolvers). hostNetwork: true dnsPolicy: ClusterFirstWithHostNet automountServiceAccountToken: false # ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ── # The two halves are deliberately stored apart (peers.yaml explains why) and are only ever # joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private # key is therefore never written to any node's disk, never committed, and never logged. initContainers: - name: assemble-config image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124 imagePullPolicy: IfNotPresent # POSIX sh. Note on $ handling: this script is rendered through Flux's envsubst (the # `vpn-ingress` Kustomization declares postBuild.substituteFrom), so `${WG_CLIENT_CIDR}` # below IS substituted on purpose — it is a declared cluster-vars knob. Every SHELL # variable is written bare (`$net`, `$(cmd)`), which envsubst leaves untouched. There is # deliberately NO braced shell-variable form anywhere in here: braces are what envsubst # consumes, so a shell variable written that way silently renders to the empty string. # (ci: flux-postbuild-substitution enforces this, and caught this very comment when it # first spelled the hazard out literally — the check reads comments too, correctly.) command: - /bin/sh - -c - | set -eu umask 077 mkdir -p /config/wg_confs # Derive the server's own tunnel address from the single CIDR knob, so there is no # second knob to keep in agreement: 10.90.0.0/24 -> 10.90.0.1/24. net=${WG_CLIENT_CIDR} mask=$(echo "$net" | cut -d/ -f2) base=$(echo "$net" | cut -d/ -f1 | cut -d. -f1-3) # The key arrives as a Secret file; strip any trailing newline the operator's shell # added, or wg-quick rejects the config with "Key is not the correct length or format". key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey) if [ -z "$key" ]; then echo "FATAL: /run/wg-server-key/privatekey is empty." >&2 echo " See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2 exit 1 fi { echo '[Interface]' echo "Address = $base.1/$mask" echo 'ListenPort = 51820' # 1420 = 1500 (flat LAN/WAN path to the node's own NIC) - 60 bytes of WireGuard # encapsulation (20 IPv4 + 8 UDP + 32 WireGuard). This is also what wg-quick would # auto-derive, but it is pinned because the symptom of getting it wrong is the # nastiest kind: handshakes and SSH work while large packets (HTTPS, file copies) # hang. Inner packets stay under Cilium's 1450-byte VXLAN MTU too, so reaching a # cluster pod does not fragment either. echo 'MTU = 1420' printf 'PrivateKey = %s\n' "$key" # ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ── # ip_forward is already 1 from node_baseline; re-asserted so this component does not # silently depend on another role's sysctl file surviving. echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1' # SNAT client traffic to the node's address. REQUIRED: without it a LAN host would # reply to 10.90.0.x, find no route, hand the packet to the router, and the router # would drop it — so every connection would hang rather than fail. Matching on # `! -o wg0` rather than a named `-o eth0` keeps this independent of the node's NIC # name. Each rule is add-if-absent so a pod killed without PostDown does not # accumulate duplicates in the host's tables across restarts. echo "PostUp = iptables -t nat -C POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || iptables -t nat -A POSTROUTING -s $net ! -o wg0 -j MASQUERADE" echo 'PostUp = iptables -C FORWARD -i wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -i wg0 -j ACCEPT' echo 'PostUp = iptables -C FORWARD -o wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -o wg0 -j ACCEPT' echo "PostDown = iptables -t nat -D POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || true" echo 'PostDown = iptables -D FORWARD -i wg0 -j ACCEPT 2>/dev/null || true' echo 'PostDown = iptables -D FORWARD -o wg0 -j ACCEPT 2>/dev/null || true' echo '' } > /config/wg_confs/wg0.conf cat /run/wg-peers/peers.conf >> /config/wg_confs/wg0.conf chmod 600 /config/wg_confs/wg0.conf # Report the SHAPE of the result, never its content — the file now holds the server's # private key, and a log line is a transcript (CLAUDE.md §6). echo "wg0.conf assembled: $(grep -c '^\[Peer\]' /config/wg_confs/wg0.conf || true) peer(s), server $base.1/$mask, udp/51820." securityContext: runAsUser: 0 allowPrivilegeEscalation: false capabilities: drop: - ALL resources: requests: cpu: 10m memory: 16Mi limits: memory: 64Mi volumeMounts: - { name: wg-config, mountPath: /config } - { name: wg-server-key, mountPath: /run/wg-server-key, readOnly: true } - { name: wg-peers, mountPath: /run/wg-peers, readOnly: true } containers: - name: wireguard # linuxserver/wireguard — provides wg, wg-quick and iptables on a maintained, multi-arch # base. Pinned to the full upstream+build tag (1.0.20260223, published 2026-10-01) rather # than `latest`, per the crafty/oauth2-proxy convention; Renovate maintains it. # # NO `PEERS` ENV IS SET, DELIBERATELY. With PEERS the image runs in "server mode" and # GENERATES keypairs at runtime — which would put the authoritative peer list on a # container filesystem instead of in git, and (worse) print client private keys and QR # codes into the POD LOG, where they would be readable by anyone with log access and # shipped to Loki. Supplying our own /config/wg_confs/wg0.conf selects the image's # bring-up-what-you-are-given path instead. image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124 imagePullPolicy: IfNotPresent env: - name: TZ value: "America/Chicago" ports: # hostNetwork, so containerPort IS the node port. Declared for documentation and so # `kubectl get pod -o wide` shows what this listener owns. - { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP } securityContext: # NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and # write the two netfilter rules above. Everything else is dropped, and the container is # NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule. # SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node # (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container # never needs to load a kernel module. If the module is missing the pod CrashLoops with # a clear wg-quick error rather than quietly falling back to slow userspace WireGuard. allowPrivilegeEscalation: false capabilities: drop: - ALL add: - NET_ADMIN # readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and # /var during startup. Same deferral as crafty/headlamp. resources: requests: cpu: 25m memory: 32Mi limits: # Kernel WireGuard does its crypto in the KERNEL, not in this container, so throughput # does not consume the container's budget. The limit covers s6 plus the tmpfs holding # wg0.conf (a memory-medium emptyDir counts against this). memory: 128Mi # `wg show wg0` exits non-zero when the interface is absent, which is the one failure that # matters and the one that a port probe cannot see: a crashed or never-created wg0 leaves # udp/51820 silent, and WireGuard is silent to unauthenticated traffic ANYWAY, so "nothing # answers" is indistinguishable from "working" from the outside. This probe is the only # thing that tells the difference. startupProbe: exec: command: ["sh", "-c", "wg show wg0 >/dev/null"] periodSeconds: 5 failureThreshold: 24 livenessProbe: exec: command: ["sh", "-c", "wg show wg0 >/dev/null"] periodSeconds: 30 failureThreshold: 3 volumeMounts: - { name: wg-config, mountPath: /config } # Read-only so the image can SEE whether the kernel module is present and fail with a # real message. It cannot load one (no SYS_MODULE) — that is the node's job. - { name: lib-modules, mountPath: /lib/modules, readOnly: true } volumes: # tmpfs. The assembled wg0.conf contains the server's private key, so it must never land on # a node's disk — `medium: Memory` guarantees that and guarantees it is gone when the pod is. - name: wg-config emptyDir: medium: Memory sizeLimit: 1Mi # The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by # name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this # tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key # that passed through a session transcript would have to be treated as compromised. # DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace # and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy # in the operator's password manager as well — restoring it is what makes every already # enrolled client work again without re-enrolling. - name: wg-server-key secret: secretName: wg-server-key defaultMode: 0400 items: - { key: privatekey, path: privatekey } - name: wg-peers configMap: name: wg-peers - name: lib-modules hostPath: path: /lib/modules type: Directory