jakman85andClaude Opus 5
f92707f38c
Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus internet egress out the house WAN. Opposite direction to platform/vpn-egress, which sends pods OUT through PIA; the two share nothing but the word "VPN". WireGuard because it is the only option with a first-party client on all four named platforms, and because its handshake MAC is keyed with the responder's own public key: an unauthenticated packet gets NO reply, so udp/51820 is indistinguishable from a closed port. That, not "it is encrypted", is what makes an unfiltered WAN port defensible, and why there is no source allowlist. hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT clients to its own pod IP, so every remote client would carry that pod's Cilium identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. The obvious build would have shipped a healthy-looking tunnel on which every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve. In the host netns traffic is SNATed to the node and arrives as host/remote-node, exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the L2-announcement lease lottery, and double encapsulation (hence MTU 1420). Permitted by restrict-host-namespaces' own carve-out for platform namespaces. The cost is stated in networkpolicy.yaml rather than papered over: no NetworkPolicy constrains a hostNetwork pod. A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule; all three nodes answer the same keys, so failover is a one-line router change. Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my house" a reviewable PR diff. The only secret -- the server's private key -- is an out-of-band Secret: an agent must not mint a long-lived private key, and the two halves are joined only at pod start into a tmpfs that dies with the pod. Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain (kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md. Touches clusters/** , so ci/approval-policy.sh will block pending the operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
296 lines
18 KiB
YAML
296 lines
18 KiB
YAML
---
|
|
# =================================================================================================
|
|
# WireGuard remote-access listener — udp/51820 in the HOST network namespace, on every node.
|
|
# =================================================================================================
|
|
# WHY WireGuard AND NOT OpenVPN/IPsec. The requirement is "Android, Linux, Windows and other
|
|
# computing clients", from anywhere on the internet, securely. WireGuard is the only option here
|
|
# with a first-party client on all four platforms (official apps for Android/iOS/Windows/macOS, and
|
|
# in-kernel on Linux since 5.6 — so Linux clients need no userspace daemon at all). It is also the
|
|
# smallest attack surface of the three: one UDP port, ~4k lines of kernel code, no cipher
|
|
# negotiation, no PKI, no user database, and — the property that actually matters for a port
|
|
# exposed to the whole internet — it NEVER REPLIES to an unauthenticated packet. See README.
|
|
#
|
|
# -------------------------------------------------------------------------------------------------
|
|
# hostNetwork: true — REQUIRED, not convenient. This is THE decision in this file.
|
|
# -------------------------------------------------------------------------------------------------
|
|
# A VPN client must behave like a machine plugged into the LAN. In Cilium, what a packet is ALLOWED
|
|
# to reach is decided by the SECURITY IDENTITY of its source, and after any in-pod SNAT that
|
|
# identity is the VPN pod's — not `world`. That distinction is not academic; it was MEASURED on
|
|
# this cluster on 2026-10-04 (decisions/2026-10-04-plex-advertise-plexdirect.md): a curl of the
|
|
# shared Gateway VIP from inside a pod returns `403 Access denied` from Envoy for **all 16**
|
|
# Gateway hostnames, because Cilium's Gateway preserves the original source identity and enforces
|
|
# the DESTINATION's ingress CiliumNetworkPolicy, and every app admits `reserved:ingress` plus named
|
|
# peers — never an arbitrary pod. LAN browsers are allowed only because they arrive as
|
|
# `reserved:world` / `reserved:host`.
|
|
#
|
|
# So a pod-networked WireGuard server would have handed remote clients a tunnel on which every
|
|
# single cluster web UI returned 403 — plex, gitea, keycloak, nextcloud, grafana, headlamp, the
|
|
# apex, all of them — while `wg show` reported a perfectly healthy tunnel. PiHole would have failed
|
|
# the same way and worse: apps/wave-0/pihole-dns.yaml's ingress policy deliberately OMITS the
|
|
# `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either.
|
|
# The VPN would have looked up and been useless, which is the worst failure shape available.
|
|
#
|
|
# In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the
|
|
# NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
|
|
# exactly what a LAN client looks like. Three further problems disappear with it:
|
|
# * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to
|
|
# /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
|
|
# is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
|
|
# required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
|
|
# * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
|
|
# this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
|
|
# * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
|
|
# flat 1500-byte LAN path, not Cilium's 1450-byte VXLAN.
|
|
#
|
|
# hostNetwork is PERMITTED here, and not by accident: platform/kyverno/policies/
|
|
# restrict-host-namespaces.yaml matches `namespaceSelector: app-namespace=true` only, and names
|
|
# this exact carve-out in its own comment ("platform components that do -- e.g. Cilium,
|
|
# node-exporter -- live in platform namespaces that do NOT carry the app-namespace label").
|
|
# namespace.yaml is deliberately unlabelled. THE COST IS REAL AND IS STATED: a hostNetwork pod
|
|
# bypasses the Cilium datapath, so NO NetworkPolicy constrains this workload (see
|
|
# networkpolicy.yaml, which says so rather than pretending otherwise). The boundary protecting
|
|
# this port is WireGuard's own cryptographic peer authentication, not the CNI.
|
|
#
|
|
# -------------------------------------------------------------------------------------------------
|
|
# A DaemonSet, not a single-replica Deployment.
|
|
# -------------------------------------------------------------------------------------------------
|
|
# The WAN port-forward on the SOHO router points at ONE node address, by hand, off-repo (same shape
|
|
# as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to
|
|
# whichever node the scheduler picked and silently invalidate that forward on every reschedule. A
|
|
# DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of
|
|
# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no
|
|
# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own
|
|
# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each.
|
|
#
|
|
# All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the
|
|
# DaemonSet to cover the cluster; add them if a node is ever tainted.
|
|
apiVersion: apps/v1
|
|
kind: DaemonSet
|
|
metadata:
|
|
name: wireguard
|
|
namespace: vpn-ingress
|
|
labels:
|
|
app.kubernetes.io/name: vpn-ingress
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: vpn-ingress
|
|
# One node at a time, so a rollout never takes every listener down at once. The node named by the
|
|
# router's forward does drop for a few seconds during its own replacement; clients re-handshake
|
|
# automatically (WireGuard retries every 5s, and rekeys every 2 minutes regardless).
|
|
updateStrategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxUnavailable: 1
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: vpn-ingress
|
|
# Declared for documentation and for symmetry with the app tier. It MUST be `internet`, and
|
|
# the value is load-bearing rather than cosmetic: platform/vpn-egress/egress-steering.yaml's
|
|
# CiliumEgressGatewayPolicy selects `egress-class: vpn` pods CLUSTER-WIDE, with no namespace
|
|
# scope. Setting `vpn` here would steer every remote client's internet traffic out the PIA
|
|
# tunnel instead of the house WAN — the opposite of the requirement, and it would look like
|
|
# a working VPN with a mysterious foreign exit IP. (The selector matches on the pod label,
|
|
# so hostNetwork does not exempt this pod from being selected.)
|
|
homelab.keithjackson.dev/egress-class: internet
|
|
# WireGuard authenticates its own peers by public key; there is no UI and no IdP hop.
|
|
homelab.keithjackson.dev/auth-mode: native
|
|
spec:
|
|
# See the long rationale at the head of this file. ClusterFirstWithHostNet keeps in-cluster
|
|
# DNS working for the pod itself despite hostNetwork (without it the pod inherits the NODE's
|
|
# resolv.conf, which points at public resolvers).
|
|
hostNetwork: true
|
|
dnsPolicy: ClusterFirstWithHostNet
|
|
automountServiceAccountToken: false
|
|
# ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
|
|
# The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
|
|
# joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
|
|
# key is therefore never written to any node's disk, never committed, and never logged.
|
|
initContainers:
|
|
- name: assemble-config
|
|
image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124
|
|
imagePullPolicy: IfNotPresent
|
|
# POSIX sh. Note on $ handling: this script is rendered through Flux's envsubst (the
|
|
# `vpn-ingress` Kustomization declares postBuild.substituteFrom), so `${WG_CLIENT_CIDR}`
|
|
# below IS substituted on purpose — it is a declared cluster-vars knob. Every SHELL
|
|
# variable is written bare (`$net`, `$(cmd)`), which envsubst leaves untouched. There is
|
|
# deliberately NO braced shell-variable form anywhere in here: braces are what envsubst
|
|
# consumes, so a shell variable written that way silently renders to the empty string.
|
|
# (ci: flux-postbuild-substitution enforces this, and caught this very comment when it
|
|
# first spelled the hazard out literally — the check reads comments too, correctly.)
|
|
command:
|
|
- /bin/sh
|
|
- -c
|
|
- |
|
|
set -eu
|
|
umask 077
|
|
mkdir -p /config/wg_confs
|
|
|
|
# Derive the server's own tunnel address from the single CIDR knob, so there is no
|
|
# second knob to keep in agreement: 10.90.0.0/24 -> 10.90.0.1/24.
|
|
net=${WG_CLIENT_CIDR}
|
|
mask=$(echo "$net" | cut -d/ -f2)
|
|
base=$(echo "$net" | cut -d/ -f1 | cut -d. -f1-3)
|
|
|
|
# The key arrives as a Secret file; strip any trailing newline the operator's shell
|
|
# added, or wg-quick rejects the config with "Key is not the correct length or format".
|
|
key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey)
|
|
if [ -z "$key" ]; then
|
|
echo "FATAL: /run/wg-server-key/privatekey is empty." >&2
|
|
echo " See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2
|
|
exit 1
|
|
fi
|
|
|
|
{
|
|
echo '[Interface]'
|
|
echo "Address = $base.1/$mask"
|
|
echo 'ListenPort = 51820'
|
|
# 1420 = 1500 (flat LAN/WAN path to the node's own NIC) - 60 bytes of WireGuard
|
|
# encapsulation (20 IPv4 + 8 UDP + 32 WireGuard). This is also what wg-quick would
|
|
# auto-derive, but it is pinned because the symptom of getting it wrong is the
|
|
# nastiest kind: handshakes and SSH work while large packets (HTTPS, file copies)
|
|
# hang. Inner packets stay under Cilium's 1450-byte VXLAN MTU too, so reaching a
|
|
# cluster pod does not fragment either.
|
|
echo 'MTU = 1420'
|
|
printf 'PrivateKey = %s\n' "$key"
|
|
# ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
|
|
# ip_forward is already 1 from node_baseline; re-asserted so this component does not
|
|
# silently depend on another role's sysctl file surviving.
|
|
echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1'
|
|
# SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
|
|
# reply to 10.90.0.x, find no route, hand the packet to the router, and the router
|
|
# would drop it — so every connection would hang rather than fail. Matching on
|
|
# `! -o wg0` rather than a named `-o eth0` keeps this independent of the node's NIC
|
|
# name. Each rule is add-if-absent so a pod killed without PostDown does not
|
|
# accumulate duplicates in the host's tables across restarts.
|
|
echo "PostUp = iptables -t nat -C POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || iptables -t nat -A POSTROUTING -s $net ! -o wg0 -j MASQUERADE"
|
|
echo 'PostUp = iptables -C FORWARD -i wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -i wg0 -j ACCEPT'
|
|
echo 'PostUp = iptables -C FORWARD -o wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -o wg0 -j ACCEPT'
|
|
echo "PostDown = iptables -t nat -D POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || true"
|
|
echo 'PostDown = iptables -D FORWARD -i wg0 -j ACCEPT 2>/dev/null || true'
|
|
echo 'PostDown = iptables -D FORWARD -o wg0 -j ACCEPT 2>/dev/null || true'
|
|
echo ''
|
|
} > /config/wg_confs/wg0.conf
|
|
|
|
cat /run/wg-peers/peers.conf >> /config/wg_confs/wg0.conf
|
|
chmod 600 /config/wg_confs/wg0.conf
|
|
|
|
# Report the SHAPE of the result, never its content — the file now holds the server's
|
|
# private key, and a log line is a transcript (CLAUDE.md §6).
|
|
echo "wg0.conf assembled: $(grep -c '^\[Peer\]' /config/wg_confs/wg0.conf || true) peer(s), server $base.1/$mask, udp/51820."
|
|
securityContext:
|
|
runAsUser: 0
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
resources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 16Mi
|
|
limits:
|
|
memory: 64Mi
|
|
volumeMounts:
|
|
- { name: wg-config, mountPath: /config }
|
|
- { name: wg-server-key, mountPath: /run/wg-server-key, readOnly: true }
|
|
- { name: wg-peers, mountPath: /run/wg-peers, readOnly: true }
|
|
containers:
|
|
- name: wireguard
|
|
# linuxserver/wireguard — provides wg, wg-quick and iptables on a maintained, multi-arch
|
|
# base. Pinned to the full upstream+build tag (1.0.20260223, published 2026-10-01) rather
|
|
# than `latest`, per the crafty/oauth2-proxy convention; Renovate maintains it.
|
|
#
|
|
# NO `PEERS` ENV IS SET, DELIBERATELY. With PEERS the image runs in "server mode" and
|
|
# GENERATES keypairs at runtime — which would put the authoritative peer list on a
|
|
# container filesystem instead of in git, and (worse) print client private keys and QR
|
|
# codes into the POD LOG, where they would be readable by anyone with log access and
|
|
# shipped to Loki. Supplying our own /config/wg_confs/wg0.conf selects the image's
|
|
# bring-up-what-you-are-given path instead.
|
|
image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124
|
|
imagePullPolicy: IfNotPresent
|
|
env:
|
|
- name: TZ
|
|
value: "America/Chicago"
|
|
ports:
|
|
# hostNetwork, so containerPort IS the node port. Declared for documentation and so
|
|
# `kubectl get pod -o wide` shows what this listener owns.
|
|
- { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
|
|
securityContext:
|
|
# NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and
|
|
# write the two netfilter rules above. Everything else is dropped, and the container is
|
|
# NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
|
|
# SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
|
|
# (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
|
|
# never needs to load a kernel module. If the module is missing the pod CrashLoops with
|
|
# a clear wg-quick error rather than quietly falling back to slow userspace WireGuard.
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
add:
|
|
- NET_ADMIN
|
|
# readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
|
|
# /var during startup. Same deferral as crafty/headlamp.
|
|
resources:
|
|
requests:
|
|
cpu: 25m
|
|
memory: 32Mi
|
|
limits:
|
|
# Kernel WireGuard does its crypto in the KERNEL, not in this container, so throughput
|
|
# does not consume the container's budget. The limit covers s6 plus the tmpfs holding
|
|
# wg0.conf (a memory-medium emptyDir counts against this).
|
|
memory: 128Mi
|
|
# `wg show wg0` exits non-zero when the interface is absent, which is the one failure that
|
|
# matters and the one that a port probe cannot see: a crashed or never-created wg0 leaves
|
|
# udp/51820 silent, and WireGuard is silent to unauthenticated traffic ANYWAY, so "nothing
|
|
# answers" is indistinguishable from "working" from the outside. This probe is the only
|
|
# thing that tells the difference.
|
|
startupProbe:
|
|
exec:
|
|
command: ["sh", "-c", "wg show wg0 >/dev/null"]
|
|
periodSeconds: 5
|
|
failureThreshold: 24
|
|
livenessProbe:
|
|
exec:
|
|
command: ["sh", "-c", "wg show wg0 >/dev/null"]
|
|
periodSeconds: 30
|
|
failureThreshold: 3
|
|
volumeMounts:
|
|
- { name: wg-config, mountPath: /config }
|
|
# Read-only so the image can SEE whether the kernel module is present and fail with a
|
|
# real message. It cannot load one (no SYS_MODULE) — that is the node's job.
|
|
- { name: lib-modules, mountPath: /lib/modules, readOnly: true }
|
|
volumes:
|
|
# tmpfs. The assembled wg0.conf contains the server's private key, so it must never land on
|
|
# a node's disk — `medium: Memory` guarantees that and guarantees it is gone when the pod is.
|
|
- name: wg-config
|
|
emptyDir:
|
|
medium: Memory
|
|
# 8Mi, not 1Mi: /config is this image's whole config ROOT, not just our one file — its
|
|
# s6 init also writes templates and state there, and a tmpfs that small would ENOSPC
|
|
# during startup rather than at a point anyone would connect to the volume size. Still
|
|
# counted against the container's 128Mi limit, and still far more than wg0.conf needs.
|
|
sizeLimit: 8Mi
|
|
# The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by
|
|
# name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this
|
|
# tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key
|
|
# that passed through a session transcript would have to be treated as compromised.
|
|
# DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace
|
|
# and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy
|
|
# in the operator's password manager as well — restoring it is what makes every already
|
|
# enrolled client work again without re-enrolling.
|
|
- name: wg-server-key
|
|
secret:
|
|
secretName: wg-server-key
|
|
defaultMode: 0400
|
|
items:
|
|
- { key: privatekey, path: privatekey }
|
|
- name: wg-peers
|
|
configMap:
|
|
name: wg-peers
|
|
- name: lib-modules
|
|
hostPath:
|
|
path: /lib/modules
|
|
type: Directory
|