jakman85andClaude Opus 5
f5353cbd30
A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
holds itself: the announcer does not answer its own ARP request, so on the
lease holder the neighbour entry settles at FAILED and packets to that VIP die
before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes
x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced.
platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and
MASQUERADEs remote clients to the node's own address, so a VPN client takes
the node's LAN path and inherits its ARP failures. The router's udp/51820
forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16
Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering
(its VIP has two Services and k3s-prod-2 held the other lease). The tunnel
handshook, names resolved, every app hung, and nothing reported it.
on k3s-prod-1: ip neigh show 192.168.2.20 -> FAILED (its own VIP)
on prod-2/3: ip neigh show 192.168.2.20 -> lladdr ... (prod-1), 443 OK
Mark the node and exclude it from announcing:
* node_baseline labels the forwarded node (node_baseline_vpn_ingress_node,
k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes
the label from any node that no longer matches so moving it MOVES the label.
* l2-announcement-policy.yaml excludes that label, so the node ARP-resolves
every VIP. Fixes all VIPs for VPN clients, not just the one that collided.
Cost: announcement candidacy drops from 3 nodes to 2.
* VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels
(label added to the kube-state-metrics allowlist), with
VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches,
so a missing label would silently re-arm the whole failure AND leave the
critical alert inert. Four promtool cases, including that inertness.
A node LABEL rather than a cluster-vars knob because the alert must read it
too, and extract_rules.py rightly refuses a ${...} placeholder in an alert
expr -- the rule unit-tested would not be the rule deployed. One source of
truth, and it touches no sensitive path.
Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and
"failover is this one line". Both false; the first caused this.
UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the
handoff entry's third section.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
403 lines
27 KiB
YAML
403 lines
27 KiB
YAML
---
|
|
# =================================================================================================
|
|
# WireGuard remote-access listener — udp/51820 in the HOST network namespace, on every node.
|
|
# =================================================================================================
|
|
# WHY WireGuard AND NOT OpenVPN/IPsec. The requirement is "Android, Linux, Windows and other
|
|
# computing clients", from anywhere on the internet, securely. WireGuard is the only option here
|
|
# with a first-party client on all four platforms (official apps for Android/iOS/Windows/macOS, and
|
|
# in-kernel on Linux since 5.6 — so Linux clients need no userspace daemon at all). It is also the
|
|
# smallest attack surface of the three: one UDP port, ~4k lines of kernel code, no cipher
|
|
# negotiation, no PKI, no user database, and — the property that actually matters for a port
|
|
# exposed to the whole internet — it NEVER REPLIES to an unauthenticated packet. See README.
|
|
#
|
|
# -------------------------------------------------------------------------------------------------
|
|
# hostNetwork: true — REQUIRED, not convenient. This is THE decision in this file.
|
|
# -------------------------------------------------------------------------------------------------
|
|
# A VPN client must behave like a machine plugged into the LAN. In Cilium, what a packet is ALLOWED
|
|
# to reach is decided by the SECURITY IDENTITY of its source, and after any in-pod SNAT that
|
|
# identity is the VPN pod's — not `world`. That distinction is not academic; it was MEASURED on
|
|
# this cluster on 2026-10-04 (decisions/2026-10-04-plex-advertise-plexdirect.md): a curl of the
|
|
# shared Gateway VIP from inside a pod returns `403 Access denied` from Envoy for **all 16**
|
|
# Gateway hostnames, because Cilium's Gateway preserves the original source identity and enforces
|
|
# the DESTINATION's ingress CiliumNetworkPolicy, and every app admits `reserved:ingress` plus named
|
|
# peers — never an arbitrary pod. LAN browsers are allowed only because they arrive as
|
|
# `reserved:world` / `reserved:host`.
|
|
#
|
|
# So a pod-networked WireGuard server would have handed remote clients a tunnel on which every
|
|
# single cluster web UI returned 403 — plex, gitea, keycloak, nextcloud, grafana, headlamp, the
|
|
# apex, all of them — while `wg show` reported a perfectly healthy tunnel. PiHole would have failed
|
|
# the same way and worse: apps/wave-0/pihole-dns.yaml's ingress policy deliberately OMITS the
|
|
# `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either.
|
|
# The VPN would have looked up and been useless, which is the worst failure shape available.
|
|
#
|
|
# In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the
|
|
# NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
|
|
# exactly what a LAN client looks like. Three further problems disappear with it:
|
|
# * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to
|
|
# /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
|
|
# is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
|
|
# required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
|
|
# COROLLARY LEARNED THE HARD WAY (prod, 2026-10-06): this pod must not merely avoid NEEDING
|
|
# the sysctl, it must not ATTEMPT it. hostNetwork implies /proc/sys is mounted read-only, so
|
|
# `sysctl -w` fails with EPERM regardless of capabilities, and wg-quick treats a failed PostUp
|
|
# as fatal and deletes the interface. Verify the value; never write it.
|
|
# * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
|
|
# this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
|
|
# * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
|
|
# flat 1500-byte LAN path, not Cilium's 1450-byte VXLAN.
|
|
#
|
|
# hostNetwork is PERMITTED here, and not by accident: platform/kyverno/policies/
|
|
# restrict-host-namespaces.yaml matches `namespaceSelector: app-namespace=true` only, and names
|
|
# this exact carve-out in its own comment ("platform components that do -- e.g. Cilium,
|
|
# node-exporter -- live in platform namespaces that do NOT carry the app-namespace label").
|
|
# namespace.yaml is deliberately unlabelled. THE COST IS REAL AND IS STATED: a hostNetwork pod
|
|
# bypasses the Cilium datapath, so NO NetworkPolicy constrains this workload (see
|
|
# networkpolicy.yaml, which says so rather than pretending otherwise). The boundary protecting
|
|
# this port is WireGuard's own cryptographic peer authentication, not the CNI.
|
|
#
|
|
# -------------------------------------------------------------------------------------------------
|
|
# A DaemonSet, not a single-replica Deployment.
|
|
# -------------------------------------------------------------------------------------------------
|
|
# The WAN port-forward on the SOHO router points at ONE node address, by hand, off-repo (same shape
|
|
# as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to
|
|
# whichever node the scheduler picked and silently invalidate that forward on every reschedule. A
|
|
# DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of
|
|
# them, and if that node is lost every other node is already warm and answers the same keys, so no
|
|
# client profile ever changes. Each node binds udp/51820 in its own netns, so there is no conflict,
|
|
# and the two idle listeners cost ~15 MiB each.
|
|
#
|
|
# BUT THE NODES ARE NOT INTERCHANGEABLE, AND THIS COMMENT USED TO CLAIM THEY WERE.
|
|
# It said "FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work". That was wrong, and the
|
|
# wording cost two days of silently broken remote access on 2026-10-08. MEASURED that day: a node
|
|
# cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds ITSELF — the announcer
|
|
# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and packets to
|
|
# that VIP die before leaving the node:
|
|
#
|
|
# k3s-prod-1, holding the shared Gateway lease: ip neigh show 192.168.2.20 -> FAILED
|
|
# k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr …(prod-1)
|
|
#
|
|
# Because this pod MASQUERADEs clients to the node's own address (see PostUp below), a remote client
|
|
# inherits exactly that failure. With the forward on the Gateway's announcing node, all 16 Gateway
|
|
# hostnames were unreachable over the VPN while PiHole DNS still resolved — a tunnel that
|
|
# handshakes, resolves names, and then hangs on every single app.
|
|
#
|
|
# THE FIX IS DECLARATIVE AND LIVES OUTSIDE THIS FILE: the forwarded node is labelled
|
|
# homelab.keithjackson.dev/vpn-ingress=true by infra/ansible/roles/node_baseline (from
|
|
# node_baseline_vpn_ingress_node), and platform-config/cilium/l2-announcement-policy.yaml excludes
|
|
# that label from announcing ANY VIP, so the node can ARP-resolve all of them. A label rather than a
|
|
# cluster-vars knob because the alert that guards this has to read it too, and the alert unit-test
|
|
# harness refuses a postBuild placeholder in an expr. Failover is therefore TWO steps — move
|
|
# node_baseline_vpn_ingress_node and re-run the role, THEN re-point the router
|
|
# (runbooks/vpn-remote-access.md step 3).
|
|
# `VpnIngressNodeAnnouncesL2VIP` (platform/monitoring/alerts.yaml) fires if the two collide again.
|
|
#
|
|
# This pod deliberately does NOT probe that itself. A readinessProbe on VIP reachability would wedge
|
|
# a DaemonSet rollout behind an unrelated Gateway outage (maxUnavailable: 1 below), and a liveness
|
|
# probe would tear down a tunnel that is carrying LAN traffic perfectly well. The condition is a
|
|
# property of the CLUSTER's announcement layout, so it is watched there.
|
|
#
|
|
# All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the
|
|
# DaemonSet to cover the cluster; add them if a node is ever tainted.
|
|
apiVersion: apps/v1
|
|
kind: DaemonSet
|
|
metadata:
|
|
name: wireguard
|
|
namespace: vpn-ingress
|
|
labels:
|
|
app.kubernetes.io/name: vpn-ingress
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: vpn-ingress
|
|
# One node at a time, so a rollout never takes every listener down at once. The node named by the
|
|
# router's forward does drop for a few seconds during its own replacement; clients re-handshake
|
|
# automatically (WireGuard retries every 5s, and rekeys every 2 minutes regardless).
|
|
updateStrategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxUnavailable: 1
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: vpn-ingress
|
|
# Declared for documentation and for symmetry with the app tier. It MUST be `internet`, and
|
|
# the value is load-bearing rather than cosmetic: platform/vpn-egress/egress-steering.yaml's
|
|
# CiliumEgressGatewayPolicy selects `egress-class: vpn` pods CLUSTER-WIDE, with no namespace
|
|
# scope. Setting `vpn` here would steer every remote client's internet traffic out the PIA
|
|
# tunnel instead of the house WAN — the opposite of the requirement, and it would look like
|
|
# a working VPN with a mysterious foreign exit IP. (The selector matches on the pod label,
|
|
# so hostNetwork does not exempt this pod from being selected.)
|
|
homelab.keithjackson.dev/egress-class: internet
|
|
# WireGuard authenticates its own peers by public key; there is no UI and no IdP hop.
|
|
homelab.keithjackson.dev/auth-mode: native
|
|
spec:
|
|
# See the long rationale at the head of this file. ClusterFirstWithHostNet keeps in-cluster
|
|
# DNS working for the pod itself despite hostNetwork (without it the pod inherits the NODE's
|
|
# resolv.conf, which points at public resolvers).
|
|
hostNetwork: true
|
|
dnsPolicy: ClusterFirstWithHostNet
|
|
automountServiceAccountToken: false
|
|
# ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
|
|
# The two halves are deliberately stored apart (peers.conf explains why) and are only ever
|
|
# joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
|
|
# key is therefore never written to any node's disk, never committed, and never logged.
|
|
initContainers:
|
|
- name: assemble-config
|
|
image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124
|
|
imagePullPolicy: IfNotPresent
|
|
# POSIX sh. Note on $ handling: this script is rendered through Flux's envsubst (the
|
|
# `vpn-ingress` Kustomization declares postBuild.substituteFrom), so `${WG_CLIENT_CIDR}`
|
|
# below IS substituted on purpose — it is a declared cluster-vars knob. Every SHELL
|
|
# variable is written bare (`$net`, `$(cmd)`), which envsubst leaves untouched. There is
|
|
# deliberately NO braced shell-variable form anywhere in here: braces are what envsubst
|
|
# consumes, so a shell variable written that way silently renders to the empty string.
|
|
# (ci: flux-postbuild-substitution enforces this, and caught this very comment when it
|
|
# first spelled the hazard out literally — the check reads comments too, correctly.)
|
|
command:
|
|
- /bin/sh
|
|
- -c
|
|
- |
|
|
set -eu
|
|
umask 077
|
|
mkdir -p /config/wg_confs
|
|
|
|
# Derive the server's own tunnel address from the single CIDR knob, so there is no
|
|
# second knob to keep in agreement: 10.90.0.0/24 -> 10.90.0.1/24.
|
|
net=${WG_CLIENT_CIDR}
|
|
mask=$(echo "$net" | cut -d/ -f2)
|
|
base=$(echo "$net" | cut -d/ -f1 | cut -d. -f1-3)
|
|
|
|
# The key arrives as a Secret file; strip any trailing newline the operator's shell
|
|
# added, or wg-quick rejects the config with "Key is not the correct length or format".
|
|
key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey)
|
|
# VALIDATE, don't just check for empty. The Secret ships committed with a REPLACE-ME
|
|
# placeholder so the GitOps wiring is complete before the key exists, so the common
|
|
# failure is a value that is present and WRONG — a placeholder, a public key pasted by
|
|
# mistake, or a SOPS document that reached the cluster undecrypted (ENC[...], which
|
|
# happens if an overlay is missing its `vpn-ingress` decryption patch). `wg pubkey`
|
|
# accepts a value only if it is a real 32-byte curve25519 private key, so it rejects
|
|
# all four. Without this the bad value would be written into wg0.conf and the pod would
|
|
# fail later and less legibly, or come up with a server nobody can handshake with.
|
|
if ! printf '%s' "$key" | wg pubkey >/dev/null 2>&1; then
|
|
echo "FATAL: the server private key in Secret 'wg-server-key' is not a valid" >&2
|
|
echo " WireGuard private key (wg pubkey rejected it)." >&2
|
|
if [ -z "$key" ]; then
|
|
echo " It is EMPTY." >&2
|
|
elif [ "$key" = "REPLACE-ME" ]; then
|
|
echo " It is still the committed REPLACE-ME PLACEHOLDER — the key has not" >&2
|
|
echo " been generated yet. This is the expected state on a fresh install." >&2
|
|
else
|
|
# `case`, NOT a dollar-brace prefix-strip. This manifest is rendered through
|
|
# envsubst, which consumes every dollar-brace expansion it sees, so a shell
|
|
# parameter expansion here renders to the empty string and silently disables this
|
|
# very branch. `case` uses no braces, so there is nothing to escape.
|
|
# (ci: flux-postbuild-substitution caught both the original expansion AND the
|
|
# first version of this comment, which spelled the hazard out literally.)
|
|
case "$key" in
|
|
ENC\[*)
|
|
echo " It is still SOPS CIPHERTEXT (starts with ENC[). The Secret" >&2
|
|
echo " reached the cluster undecrypted — check that this environment's" >&2
|
|
echo " overlay carries the 'vpn-ingress' decryption patch." >&2
|
|
;;
|
|
esac
|
|
fi
|
|
echo " Fix: runbooks/vpn-remote-access.md step 2." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# The pod shares the HOST netns, so this reads the node's real setting. Reading
|
|
# /proc/sys is permitted even though writing is not (see the PostUp note below).
|
|
# Fail closed: a tunnel that comes up but cannot forward would look healthy to every
|
|
# probe here and simply drop every client's traffic.
|
|
if ! grep -qx 1 /proc/sys/net/ipv4/ip_forward; then
|
|
echo "FATAL: net.ipv4.ip_forward is 0 on this node, so forwarded VPN traffic would" >&2
|
|
echo " be dropped. This pod CANNOT set it: with hostNetwork the runtime mounts" >&2
|
|
echo " /proc/sys read-only, because it is the host's global setting." >&2
|
|
echo " Fix on the NODE: run the node_baseline role (it writes" >&2
|
|
echo " /etc/sysctl.d/99-k3s-cilium.conf) — runbooks/vpn-remote-access.md step 1." >&2
|
|
exit 1
|
|
fi
|
|
|
|
{
|
|
echo '[Interface]'
|
|
echo "Address = $base.1/$mask"
|
|
echo 'ListenPort = 51820'
|
|
# 1420 = 1500 (flat LAN/WAN path to the node's own NIC) - 60 bytes of WireGuard
|
|
# encapsulation (20 IPv4 + 8 UDP + 32 WireGuard). This is also what wg-quick would
|
|
# auto-derive, but it is pinned because the symptom of getting it wrong is the
|
|
# nastiest kind: handshakes and SSH work while large packets (HTTPS, file copies)
|
|
# hang. Inner packets stay under Cilium's 1450-byte VXLAN MTU too, so reaching a
|
|
# cluster pod does not fragment either.
|
|
echo 'MTU = 1420'
|
|
printf 'PrivateKey = %s\n' "$key"
|
|
# ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
|
|
# NO `sysctl -w net.ipv4.ip_forward=1` HERE. It is IMPOSSIBLE from this pod and it was
|
|
# the only thing stopping the tunnel coming up — MEASURED on prod 2026-10-06:
|
|
# [#] sysctl -q -w net.ipv4.ip_forward=1
|
|
# sysctl: permission denied on key "net.ipv4.ip_forward"
|
|
# [#] ip link delete dev wg0
|
|
# and wg-quick then tore the whole tunnel down. The cause is not a missing
|
|
# capability: with `hostNetwork: true` the runtime mounts /proc/sys READ-ONLY in the
|
|
# container (confirmed in /proc/mounts: `proc /proc/sys proc ro,...`), precisely
|
|
# because those sysctls are the HOST's globals and a container must not rewrite
|
|
# them. CAP_NET_ADMIN does not and cannot lift a read-only mount.
|
|
# It was never needed anyway: node_baseline writes net.ipv4.ip_forward=1 to
|
|
# /etc/sysctl.d/99-k3s-cilium.conf, and reading it from this pod returns 1. The
|
|
# init container now VERIFIES that (a read, which /proc/sys does allow) and fails
|
|
# closed with a pointed message, which is what the write was pretending to achieve.
|
|
# SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
|
|
# reply to 10.90.0.x, find no route, hand the packet to the router, and the router
|
|
# would drop it — so every connection would hang rather than fail. Matching on
|
|
# `! -o wg0` rather than a named `-o eth0` keeps this independent of the node's NIC
|
|
# name. Each rule is add-if-absent so a pod killed without PostDown does not
|
|
# accumulate duplicates in the host's tables across restarts.
|
|
echo "PostUp = iptables -t nat -C POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || iptables -t nat -A POSTROUTING -s $net ! -o wg0 -j MASQUERADE"
|
|
echo 'PostUp = iptables -C FORWARD -i wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -i wg0 -j ACCEPT'
|
|
echo 'PostUp = iptables -C FORWARD -o wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -o wg0 -j ACCEPT'
|
|
echo "PostDown = iptables -t nat -D POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || true"
|
|
echo 'PostDown = iptables -D FORWARD -i wg0 -j ACCEPT 2>/dev/null || true'
|
|
echo 'PostDown = iptables -D FORWARD -o wg0 -j ACCEPT 2>/dev/null || true'
|
|
echo ''
|
|
} > /config/wg_confs/wg0.conf
|
|
|
|
cat /run/wg-peers/peers.conf >> /config/wg_confs/wg0.conf
|
|
chmod 600 /config/wg_confs/wg0.conf
|
|
|
|
# Report the SHAPE of the result, never its content — the file now holds the server's
|
|
# private key, and a log line is a transcript (CLAUDE.md §6).
|
|
echo "wg0.conf assembled: $(grep -c '^\[Peer\]' /config/wg_confs/wg0.conf || true) peer(s), server $base.1/$mask, udp/51820."
|
|
securityContext:
|
|
runAsUser: 0
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
resources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 16Mi
|
|
limits:
|
|
memory: 64Mi
|
|
volumeMounts:
|
|
- { name: wg-config, mountPath: /config }
|
|
- { name: wg-server-key, mountPath: /run/wg-server-key, readOnly: true }
|
|
- { name: wg-peers, mountPath: /run/wg-peers, readOnly: true }
|
|
containers:
|
|
- name: wireguard
|
|
# linuxserver/wireguard — provides wg, wg-quick and iptables on a maintained, multi-arch
|
|
# base. Pinned to the full upstream+build tag (1.0.20260223, published 2026-10-01) rather
|
|
# than `latest`, per the crafty/oauth2-proxy convention; Renovate maintains it.
|
|
#
|
|
# NO `PEERS` ENV IS SET, DELIBERATELY. With PEERS the image runs in "server mode" and
|
|
# GENERATES keypairs at runtime — which would put the authoritative peer list on a
|
|
# container filesystem instead of in git, and (worse) print client private keys and QR
|
|
# codes into the POD LOG, where they would be readable by anyone with log access and
|
|
# shipped to Loki. Supplying our own /config/wg_confs/wg0.conf selects the image's
|
|
# bring-up-what-you-are-given path instead.
|
|
image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124
|
|
imagePullPolicy: IfNotPresent
|
|
env:
|
|
- name: TZ
|
|
value: "America/Chicago"
|
|
ports:
|
|
# hostNetwork, so containerPort IS the node port. Declared for documentation and so
|
|
# `kubectl get pod -o wide` shows what this listener owns.
|
|
- { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
|
|
securityContext:
|
|
# NET_ADMIN is the whole privilege budget: create the wg0 link and write the netfilter
|
|
# rules above. (It does NOT let the pod set a host sysctl — /proc/sys is read-only under
|
|
# hostNetwork — which is why nothing here tries.) Everything else is dropped, and the
|
|
# container is
|
|
# NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
|
|
# SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
|
|
# (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
|
|
# never needs to load a kernel module. If the module is missing the pod CrashLoops with
|
|
# a clear wg-quick error rather than quietly falling back to slow userspace WireGuard.
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
add:
|
|
- NET_ADMIN
|
|
# readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
|
|
# /var during startup. Same deferral as crafty/headlamp.
|
|
#
|
|
# EXPECTED, HARMLESS STARTUP NOISE — do not "fix" it. Because capabilities are dropped
|
|
# to NET_ADMIN only, the image's s6 init cannot chown /config (no CAP_CHOWN) and logs a
|
|
# run of `chown: ... Operation not permitted` plus "Permissions could not be set. This
|
|
# is probably because your volume mounts are remote or read-only" and a scary "we will
|
|
# not provide support for it". It is wrong about the cause and it does not matter:
|
|
# /config is a writable tmpfs emptyDir, everything here runs as root (uid 0) anyway, and
|
|
# wg-quick proceeds normally — verified on prod 2026-10-06, where the tunnel's only
|
|
# failure was the sysctl above, well past these warnings. Granting CAP_CHOWN to silence
|
|
# log noise would widen the privilege set of an internet-facing pod for nothing.
|
|
resources:
|
|
requests:
|
|
cpu: 25m
|
|
memory: 32Mi
|
|
limits:
|
|
# Kernel WireGuard does its crypto in the KERNEL, not in this container, so throughput
|
|
# does not consume the container's budget. The limit covers s6 plus the tmpfs holding
|
|
# wg0.conf (a memory-medium emptyDir counts against this).
|
|
memory: 128Mi
|
|
# `wg show wg0` exits non-zero when the interface is absent, which is the one failure that
|
|
# matters and the one that a port probe cannot see: a crashed or never-created wg0 leaves
|
|
# udp/51820 silent, and WireGuard is silent to unauthenticated traffic ANYWAY, so "nothing
|
|
# answers" is indistinguishable from "working" from the outside. This probe is the only
|
|
# thing that tells the difference.
|
|
startupProbe:
|
|
exec:
|
|
command: ["sh", "-c", "wg show wg0 >/dev/null"]
|
|
periodSeconds: 5
|
|
failureThreshold: 24
|
|
livenessProbe:
|
|
exec:
|
|
command: ["sh", "-c", "wg show wg0 >/dev/null"]
|
|
periodSeconds: 30
|
|
failureThreshold: 3
|
|
volumeMounts:
|
|
- { name: wg-config, mountPath: /config }
|
|
# Read-only so the image can SEE whether the kernel module is present and fail with a
|
|
# real message. It cannot load one (no SYS_MODULE) — that is the node's job.
|
|
- { name: lib-modules, mountPath: /lib/modules, readOnly: true }
|
|
volumes:
|
|
# tmpfs. The assembled wg0.conf contains the server's private key, so it must never land on
|
|
# a node's disk — `medium: Memory` guarantees that and guarantees it is gone when the pod is.
|
|
- name: wg-config
|
|
emptyDir:
|
|
medium: Memory
|
|
# 8Mi, not 1Mi: /config is this image's whole config ROOT, not just our one file — its
|
|
# s6 init also writes templates and state there, and a tmpfs that small would ENOSPC
|
|
# during startup rather than at a point anyone would connect to the volume size. Still
|
|
# counted against the container's 128Mi limit, and still far more than wg0.conf needs.
|
|
sizeLimit: 8Mi
|
|
# The ONLY secret in this component. SOPS-encrypted IN THIS TREE (R-SEC-01, spec docs/07) as
|
|
# wg-server-key.sops.yaml, delivered by Flux, referenced here by name only (R-IDP-04).
|
|
#
|
|
# It ships committed with a REPLACE-ME placeholder: an agent must not mint a long-lived
|
|
# private key (CLAUDE.md §6), so the operator generates it and `sops set`s it in
|
|
# (runbooks/vpn-remote-access.md step 2). Shipping the encrypted placeholder rather than no
|
|
# file at all is what lets the resource line and both overlay decryption patches be present
|
|
# and GREEN from the first commit — the alternative, a commented-out resource to uncomment
|
|
# later, is exactly the "file in git with no generator entry: reviewed, merged, apparently
|
|
# done, still broken" shape this repo keeps getting bitten by.
|
|
#
|
|
# DR: in git means a cluster rebuild restores it with everything else — the property an
|
|
# out-of-band `kubectl create secret` would NOT have had, and losing this key means
|
|
# re-enrolling every device because each client profile pins its public half. Velero also
|
|
# captures this namespace (its `daily` schedule excludes only `monitoring` and `unifi`).
|
|
# Keep a password-manager copy too: that is the one that survives losing cluster AND backups.
|
|
- name: wg-server-key
|
|
secret:
|
|
secretName: wg-server-key
|
|
defaultMode: 0400
|
|
items:
|
|
- { key: privatekey, path: privatekey }
|
|
- name: wg-peers
|
|
configMap:
|
|
name: wg-peers
|
|
- name: lib-modules
|
|
hostPath:
|
|
path: /lib/modules
|
|
type: Directory
|