From f92707f38cf02106774a79e649df2ea393119074 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 5 Oct 2026 22:06:18 -0500 Subject: [PATCH] feat(vpn-ingress): WireGuard remote access on udp/51820 (hostNetwork, declarative peers) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus internet egress out the house WAN. Opposite direction to platform/vpn-egress, which sends pods OUT through PIA; the two share nothing but the word "VPN". WireGuard because it is the only option with a first-party client on all four named platforms, and because its handshake MAC is keyed with the responder's own public key: an unauthenticated packet gets NO reply, so udp/51820 is indistinguishable from a closed port. That, not "it is encrypted", is what makes an unfiltered WAN port defensible, and why there is no source allowlist. hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT clients to its own pod IP, so every remote client would carry that pod's Cilium identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. The obvious build would have shipped a healthy-looking tunnel on which every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve. In the host netns traffic is SNATed to the node and arrives as host/remote-node, exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the L2-announcement lease lottery, and double encapsulation (hence MTU 1420). Permitted by restrict-host-namespaces' own carve-out for platform namespaces. The cost is stated in networkpolicy.yaml rather than papered over: no NetworkPolicy constrains a hostNetwork pod. A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule; all three nodes answer the same keys, so failover is a one-line router change. Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my house" a reviewable PR diff. The only secret -- the server's private key -- is an out-of-band Secret: an agent must not mint a long-lived private key, and the two halves are joined only at pod start into a tmpfs that dies with the pod. Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain (kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md. Touches clusters/** , so ci/approval-policy.sh will block pending the operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1). Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 3 +- clusters/base/cluster-vars.yaml | 26 ++ clusters/base/kustomization.yaml | 7 + clusters/base/vpn-ingress.yaml | 56 ++++ .../2026-10-05-vpn-remote-access-wireguard.md | 225 +++++++++++++ .../roles/node_baseline/defaults/main.yml | 11 + infra/vpn-ingress/new-peer.sh | 195 ++++++++++++ platform/vpn-ingress/README.md | 199 ++++++++++++ platform/vpn-ingress/daemonset.yaml | 295 ++++++++++++++++++ platform/vpn-ingress/kustomization.yaml | 30 ++ platform/vpn-ingress/namespace.yaml | 26 ++ platform/vpn-ingress/networkpolicy.yaml | 56 ++++ platform/vpn-ingress/peers.yaml | 73 +++++ runbooks/README.md | 3 + runbooks/kustomization.yaml | 5 + runbooks/vpn-remote-access.md | 203 ++++++++++++ 16 files changed, 1412 insertions(+), 1 deletion(-) create mode 100644 clusters/base/vpn-ingress.yaml create mode 100644 decisions/2026-10-05-vpn-remote-access-wireguard.md create mode 100755 infra/vpn-ingress/new-peer.sh create mode 100644 platform/vpn-ingress/README.md create mode 100644 platform/vpn-ingress/daemonset.yaml create mode 100644 platform/vpn-ingress/kustomization.yaml create mode 100644 platform/vpn-ingress/namespace.yaml create mode 100644 platform/vpn-ingress/networkpolicy.yaml create mode 100644 platform/vpn-ingress/peers.yaml create mode 100644 runbooks/vpn-remote-access.md diff --git a/README.md b/README.md index 4d3672d..39c2afc 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,8 @@ infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium, longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns, - vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt + vpn-egress, vpn-ingress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt + (vpn-egress = pods OUT via PIA; vpn-ingress = remote clients IN via WireGuard) platform-config/ platform CONFIGS — the custom resources of the CRDs installed above (ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs). Split out because CRs and the CRDs they use cannot apply in one pass. diff --git a/clusters/base/cluster-vars.yaml b/clusters/base/cluster-vars.yaml index 26482bf..b7dbddf 100644 --- a/clusters/base/cluster-vars.yaml +++ b/clusters/base/cluster-vars.yaml @@ -102,6 +102,7 @@ data: # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers; # plex-remote + pihole: PLEX_L4_VIP), + # vpn-ingress (WG_CLIENT_CIDR), # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION, # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an # UNSUBSTITUTED path would ship the literal string into the cluster. @@ -207,6 +208,31 @@ data: # prod -> 192.168.2.0/24 (prod LAN; nodes .10/.11/.12) LAN_CIDR: "192.168.1.0/24" # ingress scope for the PiHole DNS listener + # --- Remote-access VPN client subnet (platform/vpn-ingress, WireGuard) -------------------------- + # The tunnel subnet handed to REMOTE VPN clients. `.1` is the server (the hostNetwork DaemonSet, + # on every node); `.2`-`.254` are devices, allocated one /32 each in + # platform/vpn-ingress/peers.yaml. Consumed twice in the pod's config-assembly script: to derive + # the server's own address, and as the `-s` match of the MASQUERADE rule that makes LAN replies + # routable. + # + # IT IS A KNOB RATHER THAN A LITERAL FOR ONE REASON: it is the only address range in this cluster + # that must be checked against EVERY OTHER range at once, so it belongs where those are declared. + # 10.90.0.0/24 was chosen to collide with none of them: + # 10.0.0.0/16 Cilium cluster-pool pod CIDR (${CLUSTER_POD_CIDR}) + # 10.43.0.0/16 k3s service CIDR (measured live 2026-10-05: kubernetes svc + # clusterIP 10.43.0.1) + # 10.72.192.0/18 the PIA tunnel's pushed addresses (platform/vpn-egress) + # 192.168.1.0/24 dev LAN / pfSense LAN (${LAN_CIDR} base default) + # 192.168.2.0/24 prod LAN (${LAN_CIDR} on prod) + # + # NO PER-ENVIRONMENT OVERRIDE, and that is deliberate rather than an omission: unlike every VIP + # above, this range is NOT drawn from the LAN. It is private to the tunnel, so the same value is + # correct in both environments and a prod patch would be a no-op that implies otherwise. + # + # IF YOU CHANGE IT, every already-issued client profile breaks (its `Address` is from this range) + # and must be re-rendered — so treat it as fixed once the first device is enrolled. + WG_CLIENT_CIDR: "10.90.0.0/24" # remote VPN clients; .1 = server. NOT a LAN range + # --- Bulk (external NFS) storage host (docs/04 bulk tier) --------------------------------------- # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive diff --git a/clusters/base/kustomization.yaml b/clusters/base/kustomization.yaml index 025ddac..d91cba3 100644 --- a/clusters/base/kustomization.yaml +++ b/clusters/base/kustomization.yaml @@ -71,4 +71,11 @@ resources: # controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy. # Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name). - backup.yaml # Flux Kustomization -> ./platform/velero (dependsOn platform-config) + # WireGuard remote-access VPN (north-south INGRESS, udp/51820) — POST-platform tier + # (dependsOn platform-config, wait:false). NOT the same thing as platform/vpn-egress, which sends + # pods OUT through PIA; this lets remote CLIENTS IN. Non-gating is load-bearing rather than + # conventional here: its DaemonSet cannot start until the operator creates the out-of-band + # `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have + # stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch. + - vpn-ingress.yaml # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config) - cluster-vars.yaml # overlay knob surface (base defaults) diff --git a/clusters/base/vpn-ingress.yaml b/clusters/base/vpn-ingress.yaml new file mode 100644 index 0000000..c34816a --- /dev/null +++ b/clusters/base/vpn-ingress.yaml @@ -0,0 +1,56 @@ +--- +# Flux Kustomization: WireGuard remote-access VPN — reconciles ./platform/vpn-ingress. +# +# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false). Not a line in +# platform/kustomization.yaml, for the standard reason plus one specific to this component: +# +# * STANDARD: a leaf must never sit in the gating `platform` tier. While `platform` is not-Ready +# NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves +# (DECISIONS-LOG: "Headlamp gating stall"). +# * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the +# operator creates the out-of-band `wg-server-key` Secret (runbooks/vpn-remote-access.md +# step 1). That is the correct fail-closed state — an unenrolled VPN must not come up — but in +# the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the +# cause (a missing manual step) invisible from any Flux condition. +# +# WHY dependsOn platform-config AND NOT cilium. This component needs no Gateway, no TLS cert, no +# Keycloak and no LB-IPAM pool — it has no HTTPRoute and no LoadBalancer Service, and hostNetwork +# means it does not even need a Cilium endpoint. It could legally reconcile much earlier. It is +# pinned behind platform-config anyway so that the ONE thing remote clients actually consume +# through it — split-horizon DNS and the shared Gateway — exists by the time the tunnel does. +# A tunnel that comes up before the things on the far side of it is a support call, not a feature. +# +# wait:false: nothing in the graph dependsOn this, and its pods are legitimately not-Ready whenever +# the server key is absent or the `wireguard` kernel module has not been loaded on a node. Neither +# condition should be able to gate anything else, even if a future resource is mistakenly pointed +# at it. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: vpn-ingress + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: false + dependsOn: + - name: platform-config + sourceRef: + kind: GitRepository + name: flux-system + path: ./platform/vpn-ingress + # REQUIRED, not boilerplate: platform/vpn-ingress/daemonset.yaml renders ${WG_CLIENT_CIDR} into + # the config-assembly script. Without this block that token becomes the EMPTY STRING and the + # Kustomization still reports Ready (Flux's substitution fails OPEN — ci: + # flux-postbuild-substitution), leaving an initContainer that derives a server address of ".1/" + # and a MASQUERADE rule with no source. Both fail loudly at pod start, but the Kustomization + # would look healthy. + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars + # NO `decryption:` stanza is needed in the overlays: this path carries no *.sops.yaml. The server + # key is an out-of-band Secret (platform/vpn-ingress/kustomization.yaml says what to do if that + # ever changes). diff --git a/decisions/2026-10-05-vpn-remote-access-wireguard.md b/decisions/2026-10-05-vpn-remote-access-wireguard.md new file mode 100644 index 0000000..6cb4faf --- /dev/null +++ b/decisions/2026-10-05-vpn-remote-access-wireguard.md @@ -0,0 +1,225 @@ +--- +date: 2026-10-05 +branch: feat/vpn-remote-access-wireguard +title: Remote-access VPN — WireGuard on udp/51820, hostNetwork because a pod identity would 403 every Gateway hostname +--- + +## What changed + +New platform component `platform/vpn-ingress/`: a **WireGuard** remote-access VPN so the operator's +Android / Linux / Windows / macOS devices can dial in from anywhere on the internet and behave like +machines on the LAN — full reach to `192.168.2.0/24` plus internet egress out the house WAN. + +This is the **opposite direction** to the existing `platform/vpn-egress` (pods *out* through PIA, +ADR-0015). They share nothing but the word "VPN"; both READMEs now say so, and `README.md`'s layout +block labels the two. + +**Transport: WireGuard, and the security argument is the handshake's failure mode, not "it is +encrypted".** WireGuard's first handshake message carries a MAC keyed with the responder's own +public key, so a sender that does not already know it gets **no reply at all** — no error, no reset, +no banner. From the internet, udp/51820 is indistinguishable from a closed port: nothing to +brute-force, no version string, no amplification. That is what makes an unfiltered WAN port +defensible here, and it is why there is deliberately no `loadBalancerSourceRanges`-style allowlist +(which would also break roaming clients). It also has first-party clients on all four named +platforms, which OpenVPN (needs a CA + per-device certs + a CRL) and IKEv2 do not. + +**`hostNetwork: true`, and this is THE decision — forced by a measured property of this cluster.** +A pod-networked WireGuard server must SNAT client traffic to its own pod IP (Cilium drops a packet +whose source is not the endpoint's address), so every remote client would have arrived wearing *that +pod's* security identity. `decisions/2026-10-04-plex-advertise-plexdirect.md` measured what that +costs: a request to the shared Gateway VIP **from inside a pod returns `403 Access denied` from +Envoy for all 16 Gateway hostnames**, because Cilium's Gateway preserves the original source +identity and enforces the *destination's* ingress policy, and every app admits `reserved:ingress` +plus named peers — never an arbitrary pod. So the obvious build would have produced a tunnel that +came up healthy, showed connected peers, routed packets, and 403'd every cluster web UI — and +`*.home.keithjackson.dev` would not even have resolved, because `apps/wave-0/pihole-dns.yaml`'s +ingress policy deliberately omits the `cluster` entity. In the host netns there is no pod identity +to carry: forwarded traffic is SNATed to the node address and arrives as `host`/`remote-node`, +byte-for-byte what a LAN client looks like. Three further problems vanish with it: `ip_forward` is +already 1 on these nodes (so no `--allowed-unsafe-sysctls` kubelet change — it is *not* a safe +sysctl); no LoadBalancer VIP, so no L2-announcement lease lottery and no `externalTrafficPolicy` +trade-off; and no double encapsulation, which is what makes `MTU = 1420` simply correct. + +`hostNetwork` is permitted by `restrict-host-namespaces`' own stated carve-out (it matches +`app-namespace=true` only, and names Cilium/node-exporter as the precedent); `namespace.yaml` is +deliberately unlabelled. **The cost is stated, not buried:** a hostNetwork pod has no Cilium +endpoint, so **no NetworkPolicy constrains this workload**. `networkpolicy.yaml` says that at +length rather than shipping an `allow world → 51820` rule that would read as the control and +enforce nothing; the real boundary is WireGuard's peer authentication. + +**A DaemonSet, not a Deployment.** The WAN forward names one node address by hand, off-repo (same +shape as the plex tcp/32400 forward). A floating single replica would silently invalidate it on +every reschedule; an identical listener on all three nodes makes **failover a one-line router +change** with no cluster work and no re-issued client profiles. A pinned LoadBalancer VIP was the +considered alternative and was rejected: it reintroduces the L2 lease, SNAT and an extra node hop +into the one path whose failure mode is "I am away from home and cannot get in". + +**Authorisation is split so that it stays reviewable.** Peer **public** keys live in +`platform/vpn-ingress/peers.yaml` in **plaintext** — a public key grants nothing by itself, and this +makes "who can get into my house" a PR diff with `git log` as its audit trail. The only real secret, +the server's private key, is the `wg-server-key` Secret, **created out of band by the operator**: an +agent must not mint a long-lived private key (CLAUDE.md §6), and a key that passed through a session +transcript would have to be treated as compromised. The two halves are joined only at pod start, by +an init container, into a `medium: Memory` emptyDir — so the assembled config holding the key never +touches a node's disk and dies with the pod. + +**There are no peers yet, and that is the correct initial state**, for the same reason. The VPN ships +INERT-BUT-HEALTHY: a zero-peer interface listens and refuses every handshake. + +Files: + +- `platform/vpn-ingress/` (new): `namespace.yaml`, `peers.yaml` (the plaintext peer register), + `daemonset.yaml` (hostNetwork listener + config assembly), `networkpolicy.yaml` (namespace + baseline, with the honest caveat), `kustomization.yaml`, `README.md` (the design record). +- `clusters/base/vpn-ingress.yaml` (new): its own **non-gating post-platform** Flux Kustomization + (`dependsOn platform-config`, `wait:false`). Non-gating is load-bearing rather than conventional + here: the DaemonSet *cannot* start until the operator creates the server key, so in the gating + `platform` tier an unfinished manual step would have stalled every apps wave on merge. +- `clusters/base/kustomization.yaml`: the new tier added. +- `clusters/base/cluster-vars.yaml`: new `WG_CLIENT_CIDR` (`10.90.0.0/24`), declared next to + `LAN_CIDR` because it is the one range that must be collision-checked against every other; the + comment enumerates the five it avoids. **No prod override, deliberately** — unlike every VIP, this + range is private to the tunnel, so the same value is correct in both environments and a prod patch + would imply otherwise. Substitution inventory updated. +- `infra/ansible/roles/node_baseline/defaults/main.yml`: `wireguard` added to + `node_baseline_kernel_modules`. The pod has `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so + it cannot autoload the module — it CrashLoops with a clear error rather than quietly degrading to + userspace wireguard-go (which would need `/dev/net/tun`, a device a hostPath mount cannot actually + grant, as the `/dev/dri` attempt proved). +- `infra/vpn-ingress/new-peer.sh` (new): enrols a device. Generates the keypair on the operator's + machine, writes a 0600 profile **outside** the checkout (it refuses to write into the repo), + prints a QR code for phones, and prints only the public `[Peer]` stanza. It also refuses a + duplicate address, because two devices on one address does not error anywhere — the server routes + to whichever handshook last, so both appear to work intermittently. +- `runbooks/vpn-remote-access.md` (new) + its `kustomization.yaml` generator entry + `README.md` + index line: the three manual steps, the four off-LAN verification checks, revocation, and why no + connection alert exists. +- `README.md`: `vpn-ingress` added to the layout, with the egress/ingress distinction spelled out. + +**THREE MANUAL STEPS REMAIN AND THE VPN DOES NOT WORK UNTIL ALL THREE ARE DONE** (all off-repo by +necessity; `runbooks/vpn-remote-access.md`): run `node_baseline` to load the kernel module, create +the `wg-server-key` Secret, and forward WAN udp/51820 to one node address on the SOHO router. + +**SENSITIVE PATHS — this PR touches `clusters/**` (three files), so `ci/approval-policy.sh` will +BLOCK it.** It needs the operator's `/approve-override ` posted as a PR **review** of type +"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own +PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit. + +## What I verified + +Live prod, read-only, 2026-10-05, plus the offline gate. + +- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with + `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all + **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`) + and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.** + Run from an isolated git worktree with both CLAUDE.md §3 workarounds applied + (`DEV_KUBECONFIG=/dev/null` plus an `ENGINE` shim bind-mounting the main checkout's `.git` and + the sibling spec repo) — which is why `spec-immutability` is **green here and skipped in CI**, + where the runner has no spec checkout. `approval-policy` passing locally is an offline N/A, not a + verdict; the real verdict is the CI run below. +- **The CI gate ran on PR #55 (run 244, head `1a67fc1b`) and `approval-policy` is the ONLY failing + stage** — read from the job log, not from the red tick. Verbatim: "This PR changes the pipeline or + the security posture it rests on, and has NO approving review from anyone other than its author + ('gitea-admin')", naming exactly the three files I predicted: + `clusters/base/cluster-vars.yaml`, `clusters/base/kustomization.yaml`, + `clusters/base/vpn-ingress.yaml`. Every other stage that ran reported green — + `gitea-settings`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. So the branch is blocked + on the operator's `/approve-override ` (posted as a PR **review** of type "Comment") and + on nothing else. **I have not posted it** — CLAUDE.md §1. +- **Branch protection is live, measured by CI's own `gitea-settings` stage** (it has the token this + workstation lacks): `required_approvals = 0`, `enable_status_check = true`, + `status_check_contexts = ["gate / gate (pull_request)"]`, `block_admin_merge_override = true`, + `enable_push = false`, `dismiss_stale_approvals = true`, and **1 runner online**. That is the §2b + re-measurement of CLAUDE.md §2's table for this branch, not a restatement of it. +- **`flux-postbuild-substitution` (L0) actually exercised the new wiring, and caught a real defect + in it.** On the first run it FAILED, naming `platform/vpn-ingress/daemonset.yaml:118` and + attributing it to "Flux Kustomization 'vpn-ingress' (clusters/base/vpn-ingress.yaml, path + ./platform/vpn-ingress)". Two things are proven by that message: the new tier's + `postBuild.substituteFrom` declaration is correctly detected (so `${WG_CLIENT_CIDR}` will be + substituted, not shipped literally), and the check reads comments as well as values — the hazard + was a comment that spelled out the forbidden braced form literally while claiming not to use it. + Fixed; the check now passes 444/444 with 0 hazards. +- **The `wireguard` kernel module is available on these nodes by construction**: all three run + Debian 13 (trixie) on kernel `7.1.8+deb13-amd64` (measured), and WireGuard has been in-tree since + 5.6, so it is present in the distro kernel and only needs loading. +- **`net.ipv4.ip_forward = 1` is already written by `node_baseline`** + (`roles/node_baseline/tasks/main.yml`, `/etc/sysctl.d/99-k3s-cilium.conf`), and **no kubelet + carries `--allowed-unsafe-sysctls`** (grepped across `roles/k3s_server`, `roles/node_baseline` and + `group_vars`: zero hits). That pair is what makes the host netns the only option that does not + require a node-restart change. +- **Cilium's datapath, measured from `cilium-config` rather than from the values file**: + `enable-ipv4-masquerade=true`, `enable-bpf-masquerade=true`, `routing-mode=tunnel`, + `tunnel-protocol=vxlan`, `ipv4-native-routing-cidr` EMPTY, Cilium `v1.20.1`. The vxlan underlay is + where `MTU = 1420` comes from, and the empty native-routing CIDR is why pod→LAN is masqueraded + today. +- **`WG_CLIENT_CIDR` collides with nothing.** Pod CIDR `10.0.0.0/16` (`CLUSTER_POD_CIDR`), service + CIDR `10.43.0.0/16` (measured: `kubernetes` Service clusterIP is `10.43.0.1`), the PIA tunnel's + `10.72.192.0/18` (`platform/vpn-egress/README.md`), and both LANs `192.168.1.0/24` / + `192.168.2.0/24`. `10.90.0.0/24` is outside all five. +- **The egress-class label matters and is set to `internet`.** Read live: + `CiliumEgressGatewayPolicy vpn-egress-steering` selects `podSelector: + homelab.keithjackson.dev/egress-class: vpn` with **no namespace scope**, so `vpn` here would have + steered every remote client's internet traffic out PIA. +- **All three nodes are untainted** (`spec.taints: None` on each), so the DaemonSet covers the + cluster with no tolerations. +- **`new-peer.sh` was run end to end** against a throwaway server key in the scratchpad (deleted + immediately after; it never existed in the repo). It rendered a correct profile — `Address + 10.90.0.7/32`, `DNS 192.168.2.23`, `Endpoint wan.home.keithjackson.dev:51820`, `MTU 1420`, + `PersistentKeepalive 25`, `AllowedIPs 0.0.0.0/0` — at mode 0600, printed the matching public-key + stanza, and both argument-validation paths (missing name, out-of-range octet) refuse correctly. + The private key was redacted from every line I displayed. +- **The endpoint name is resolver-independent, which is the specific trap this avoids.** + `wan.${BASE_DOMAIN}` is an unproxied public Cloudflare A record kept current by + `platform/externaldns/ddns.yaml` (checked every 5 min), so it resolves from a cellular connection + — unlike any name PiHole answers, which a remote client cannot resolve until it is already + connected. (This is the `plex.direct` lesson applied.) +- **Velero does cover this namespace.** `platform/velero/helmrelease.yaml`'s `daily` schedule + excludes only `monitoring` and `unifi`, so `vpn-ingress` and the `wg-server-key` Secret are + captured off-cluster to R2. Read from the manifest, not from a backup listing. + +## What I did NOT verify + +- **THE ACTUAL FIX. Nothing in this branch has been proven to carry a single packet.** The + DaemonSet has never been reconciled, no `wg0` has ever existed, and no handshake has ever been + attempted. Three blockers are off-repo and manual (kernel module, server key, WAN forward), and + this branch removes none of them — it only makes them the *only* remaining work. +- **Outside-in reachability, at all.** This repo has **no off-LAN vantage point**, which is exactly + the gap the plex remote-access work hit. Step 6 of the runbook (a phone on cellular) is the only + honest test and it has not been run. +- **The in-pod `iptables` → host-netfilter interaction, which is my biggest single doubt.** Debian 13 + uses nf_tables; the container writes MASQUERADE/FORWARD rules into the **host's** tables via the + image's own `iptables`. If that binary is the legacy backend while the host is nft, the rules can + apply to a table the kernel does not consult — and the symptom would be the nastiest available: + handshakes succeed, `wg show` looks perfect, and every forwarded connection hangs. **Check this + first if the tunnel comes up but nothing is reachable:** `iptables-save -t nat | grep MASQUERADE` + and `nft list table ip nat` on the node, from the host, not from the pod. +- **Whether `allowPrivilegeEscalation: false` is compatible with the image's s6 init.** It is the + correct posture and I set it, but this repo has already been bitten by `no-new-privileges` + misbehaving (`platform/externaldns/ddns.yaml`'s README note). If the pod fails before + `wg-quick` runs, that flag is the first thing to try flipping. +- **Whether `NET_ADMIN` alone is sufficient.** Reasoned from what `wg-quick` does (create the link, + set a sysctl, write two netfilter rules), not measured. If `iptables` complains, `NET_RAW` is the + likely addition — deliberately not granted pre-emptively. +- **The image's custom-config path.** `lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124` (tag + confirmed to exist and to be the current one, published 2026-10-01) is expected to bring up + whatever is in `/config/wg_confs/` when `PEERS` is unset. That is its documented behaviour; it has + not been run here, and the init container's assembly of that file has never executed. +- **Whether the probe is right.** `wg show wg0` is asserted to exit non-zero when the interface is + absent. Not executed. +- **Cilium's reaction to a new `wg0` in the host netns.** `platform/cilium/values.yaml` does not pin + `devices`, so Cilium auto-detects; whether it attaches to `wg0` — and whether that matters — is + unknown. `wg` interfaces are already filtered out by `node_health_collect.py`, so something in + the estate anticipates them, but that is not evidence about Cilium. +- **Zero peers means the end-to-end authorisation path is untested**, including revocation. The + first enrolment is also the first test of `peers.yaml`'s format being what `wg-quick` accepts. +- **No peer-level observability exists.** An exporter publishing + `wireguard_latest_handshake_seconds` is the top follow-up; until then "who is connected" is an + on-demand `wg show` and nothing alerts on an unexpected session. Pod-down is covered only by + kube-prometheus-stack's generic `KubePodNotReady` / `KubeDeploymentReplicasMismatch`, which I did + not confirm fire for a DaemonSet in this namespace. +- **The security consequence I am most keen for a human to review rather than take from me:** an + enrolled device reaches the three k3s nodes themselves — apiserver 6443, kubelet 10250, SSH 22 — + because "reach the entire local network" includes them. That is stated in the README and the + runbook with the narrowing recipe, but it is a posture decision, not a verified fact, and it is + the right thing to argue about on this PR. diff --git a/infra/ansible/roles/node_baseline/defaults/main.yml b/infra/ansible/roles/node_baseline/defaults/main.yml index d03016c..b6a7f4b 100644 --- a/infra/ansible/roles/node_baseline/defaults/main.yml +++ b/infra/ansible/roles/node_baseline/defaults/main.yml @@ -74,12 +74,23 @@ node_baseline_backports_kernel_packages: # kernel, and how do I undo it?" from the machine itself. node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel +# Modules written to /etc/modules-load.d/k3s-cilium-longhorn.conf (persistent) AND modprobe'd now. node_baseline_kernel_modules: - overlay - br_netfilter - iscsi_tcp - dm_crypt - nfs + # Remote-access VPN (platform/vpn-ingress). The WireGuard DaemonSet creates its wg0 link in the + # HOST netns with CAP_NET_ADMIN only -- it is deliberately NOT granted CAP_SYS_MODULE, so it + # cannot autoload this itself: `ip link add wg0 type wireguard` fails outright when the module is + # absent, and the pod CrashLoops. Loading it here is therefore a PREREQUISITE of that component, + # not a nicety. In-tree since kernel 5.6, so Debian 13's 7.1.8 ships it and there is nothing to + # install. Naming it here also means the kernel does the crypto: without the module a WireGuard + # container's only fallback is slow userspace wireguard-go, which we do not use (and which would + # need /dev/net/tun -- a device a hostPath mount cannot actually grant, as the /dev/dri attempt + # proved: visible but EPERM from the device cgroup). + - wireguard # --- DATA DISKS (tasks/disks.yml, tag `disks`) ---------------------------------------- # The SHAPE of a node's storage is DECLARED per environment, never inferred from whether diff --git a/infra/vpn-ingress/new-peer.sh b/infra/vpn-ingress/new-peer.sh new file mode 100755 index 0000000..31d370b --- /dev/null +++ b/infra/vpn-ingress/new-peer.sh @@ -0,0 +1,195 @@ +#!/usr/bin/env bash +# ================================================================================================= +# new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress). +# +# Generates a WireGuard keypair ON THIS MACHINE, renders the device's client profile, optionally +# prints a QR code for a phone, and prints the one `[Peer]` stanza to paste into +# platform/vpn-ingress/peers.yaml. The private half NEVER leaves this machine: it is not sent to the +# cluster, not committed, and not written to any log. +# +# WHY A SCRIPT AND NOT A RUNBOOK STEP. Three of these fields are easy to get wrong in ways that are +# either silently insecure or silently broken, and all three are mechanical: +# * AllowedIPs on the SERVER side must be a /32. On the server, AllowedIPs is simultaneously the +# cryptokey routing table AND the inbound source filter, so a /24 here lets one device spoof +# every other device's tunnel address. +# * PersistentKeepalive must be set on the CLIENT. Without it the NAT mapping at the client's end +# expires after ~30s of idle and nothing can reach the device until it speaks first — which +# presents as "the VPN works for a minute and then half-works". +# * The endpoint must be the DDNS name, never the current WAN IP (see --endpoint below). +# +# USAGE +# ./infra/vpn-ingress/new-peer.sh [--split] [--out DIR] +# +# short, lowercase, e.g. pixel-8, thinkpad, work-laptop. Used for the filename +# and the comment that ties the key to a human-recognisable device. +# 2-254, the device's address inside WG_CLIENT_CIDR. Must be unused. +# --split split tunnel: route ONLY the LAN and the tunnel subnet. Default is a FULL +# tunnel (0.0.0.0/0) so the device also egresses the house WAN. +# --out DIR where to write the profile (default: a 0700 dir under $TMPDIR). +# +# ENVIRONMENT +# WG_SERVER_KEY_FILE server PRIVATE key (default ~/.config/wireguard-home/server.key); its public +# half is derived from it. Alternatively set WG_SERVER_PUBKEY directly. +# WG_ENDPOINT_HOST default wan.home.keithjackson.dev +# WG_CLIENT_CIDR default 10.90.0.0/24 (must match clusters/base/cluster-vars.yaml) +# WG_LAN_CIDR default 192.168.2.0/24 (prod LAN_CIDR) +# WG_DNS default 192.168.2.23 (prod PIHOLE_DNS_VIP) +# +# REQUIRES: wireguard-tools (`wg`). `qrencode` is optional and only used for the QR code. +# ================================================================================================= +set -euo pipefail + +die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; } + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml" + +# --- defaults, each mirroring a value that is DECLARED elsewhere in this repo ------------------- +# These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to +# run on an operator's laptop with no checkout-shaped assumptions beyond its own location, and a +# silent mis-parse would be worse than a visible default. If you change the knob, change it here. +: "${WG_CLIENT_CIDR:=10.90.0.0/24}" # clusters/base/cluster-vars.yaml +: "${WG_LAN_CIDR:=192.168.2.0/24}" # clusters/prod LAN_CIDR +: "${WG_DNS:=192.168.2.23}" # clusters/prod PIHOLE_DNS_VIP +: "${WG_SERVER_KEY_FILE:=${HOME}/.config/wireguard-home/server.key}" +# wan. is an UNPROXIED A record that platform/externaldns/ddns.yaml keeps pointed at +# the current WAN IP (checked every 5 min). Use it, never a literal address: this house has a +# DYNAMIC WAN IP, and a profile pinned to today's address stops working without warning and with no +# symptom on the server side. It is a PUBLIC Cloudflare record, so it also resolves from a cellular +# connection — unlike any `*.home.keithjackson.dev` name PiHole answers, which a remote client +# cannot resolve until it is already connected. +: "${WG_ENDPOINT_HOST:=wan.home.keithjackson.dev}" +: "${WG_LISTEN_PORT:=51820}" +: "${WG_MTU:=1420}" + +SPLIT=0 +OUT_DIR="" +NAME="" +OCTET="" +while [ $# -gt 0 ]; do + case "$1" in + --split) SPLIT=1; shift ;; + --out) OUT_DIR="${2:-}"; [ -n "$OUT_DIR" ] || die "--out needs a directory"; shift 2 ;; + -h|--help) sed -n '2,40p' "${BASH_SOURCE[0]}"; exit 0 ;; + -*) die "unknown option: $1" ;; + *) if [ -z "$NAME" ]; then NAME="$1"; elif [ -z "$OCTET" ]; then OCTET="$1"; else die "unexpected argument: $1"; fi; shift ;; + esac +done + +[ -n "$NAME" ] || die "need a device name. See --help" +[ -n "$OCTET" ] || die "need a last octet (2-254). See --help" +printf '%s' "$NAME" | grep -Eq '^[a-z0-9][a-z0-9-]{0,30}$' \ + || die "device name must be lowercase letters/digits/hyphens: got '$NAME'" +printf '%s' "$OCTET" | grep -Eq '^[0-9]+$' || die "octet must be a number: got '$OCTET'" +[ "$OCTET" -ge 2 ] && [ "$OCTET" -le 254 ] || die "octet must be 2-254 (.1 is the server): got '$OCTET'" + +command -v wg >/dev/null || die "wireguard-tools not installed ('wg' not on PATH). Debian/Ubuntu: apt install wireguard-tools" + +# --- the server's public key -------------------------------------------------------------------- +if [ -z "${WG_SERVER_PUBKEY:-}" ]; then + [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read server key '$WG_SERVER_KEY_FILE'. + Either point WG_SERVER_KEY_FILE at it, or set WG_SERVER_PUBKEY to the public half. + If the server key does not exist yet, see runbooks/vpn-remote-access.md step 1." + WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)" +fi + +CLIENT_BASE="${WG_CLIENT_CIDR%/*}"; CLIENT_BASE="${CLIENT_BASE%.*}" +CLIENT_ADDR="${CLIENT_BASE}.${OCTET}" + +# --- refuse a collision, because the failure is otherwise invisible ----------------------------- +# Two devices sharing an address does not error anywhere: the server routes that address to +# whichever peer handshook most recently, so BOTH devices appear to work intermittently. +if [ -r "$PEERS_FILE" ] && grep -q "AllowedIPs *= *${CLIENT_ADDR}/32" "$PEERS_FILE"; then + die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet." +fi +if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then + printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2 +fi + +# --- output location: 0700, and never inside the repo ------------------------------------------- +if [ -z "$OUT_DIR" ]; then + OUT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/wg-peer-XXXXXX")" +else + mkdir -p "$OUT_DIR" +fi +chmod 700 "$OUT_DIR" +case "$(cd "$OUT_DIR" && pwd)" in + "$REPO_ROOT"|"$REPO_ROOT"/*) die "refusing to write a private key inside the git checkout ($OUT_DIR)" ;; +esac +PROFILE="${OUT_DIR}/${NAME}.conf" + +umask 077 +PRIVKEY="$(wg genkey)" +PUBKEY="$(printf '%s' "$PRIVKEY" | wg pubkey)" + +if [ "$SPLIT" -eq 1 ]; then + ALLOWED="${WG_LAN_CIDR}, ${WG_CLIENT_CIDR}" + TUNNEL_KIND="split tunnel (LAN only; the device keeps using its own internet)" +else + ALLOWED="0.0.0.0/0" + TUNNEL_KIND="FULL tunnel (all IPv4 traffic egresses the house WAN)" +fi + +cat > "$PROFILE" <, enrolled $(date -u +%Y-%m-%d) + [Peer] + PublicKey = ${PUBKEY} + AllowedIPs = ${CLIENT_ADDR}/32 + + Keep the comment line: \`wg show\` can only print the key, so it is the only record of which + stanza belongs to which device when one has to be revoked. + + STEP 2 — get the profile onto the device: + Android / iOS : scan the QR code below in the WireGuard app ("+" -> Scan from QR code) + Windows / macOS: WireGuard app -> "Import tunnel(s) from file" -> ${NAME}.conf + Linux : install as /etc/wireguard/${NAME}.conf, then + sudo systemctl enable --now wg-quick@${NAME} + + STEP 3 — delete the profile from this machine once the device has it: + shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE} + + The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet + (~1 min). Until then the device handshakes and gets silence, exactly like an unknown key. + ------------------------------------------------------------------------------------------ +EOF + +if command -v qrencode >/dev/null; then + printf ' QR code (this IS the private key — your terminal scrollback now holds a credential):\n\n' + qrencode -t ansiutf8 < "$PROFILE" +else + printf ' qrencode not installed, so no QR code was printed (Debian/Ubuntu: apt install qrencode).\n' + printf ' Transfer %s to the device by another route instead.\n' "$PROFILE" +fi diff --git a/platform/vpn-ingress/README.md b/platform/vpn-ingress/README.md new file mode 100644 index 0000000..4ed1016 --- /dev/null +++ b/platform/vpn-ingress/README.md @@ -0,0 +1,199 @@ +# platform/vpn-ingress — WireGuard remote access (north-south INGRESS) + +Lets the operator's own devices — Android, Linux, Windows, macOS, iOS — dial in from anywhere on +the internet and behave as though plugged into the home LAN: full reach to `192.168.2.0/24` and +egress out the house WAN. + +> **This is not `platform/vpn-egress`.** They are opposite directions and share nothing but the +> word "VPN". `vpn-egress` pushes *pods out* to the internet through a PIA tunnel (ADR-0015, +> R-NET-05/06/07). This component lets *remote clients in*. Changing one never affects the other. + +| | | +|---|---| +| Transport | WireGuard, **udp/51820**, kernel implementation | +| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node | +| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices | +| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect | +| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** | +| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo | +| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) | +| Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) | + +## Why WireGuard + +The requirement names four client platforms. WireGuard is the only option that has a *first-party* +client on all of them (official apps for Android, iOS, Windows and macOS; in-kernel on Linux since +5.6, so a Linux client needs no userspace daemon at all). It is also the smallest thing to expose: +one UDP port, no cipher negotiation, no PKI, no user database, no TLS stack. + +OpenVPN would have needed a CA, per-device certificates and a revocation list — three pieces of +long-lived state this repo would then have to protect and rotate. IKEv2/IPsec has native clients but +is painful to run in a container and brittle behind double NAT. Neither buys anything here. + +### Why this is safe to expose to the whole internet + +The honest answer is not "because it is encrypted" — it is the handshake's failure mode: + +**An unauthenticated packet gets no reply at all.** WireGuard's first handshake message carries a +MAC keyed with the *responder's own public key*. If the sender does not already know it, the packet +is dropped silently — no error, no reset, no banner, nothing a scanner can fingerprint. From the +outside, udp/51820 is indistinguishable from a closed port. There is no login prompt to brute-force, +no version string to match against a CVE, and no amplification to abuse. + +That is the whole security argument, and it is why the exposed port carries no IP allowlist (which +would also have made roaming clients unusable — a phone's address changes with every cell tower). +Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a +resolver answers anybody, so it must be scoped by address. WireGuard answers nobody. + +What it does **not** protect against is a stolen device — anyone holding a device's private key +*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min). + +## Architecture, and the one decision that drives everything + +``` + phone on 5G SOHO router k3s-prod-N (host netns) + ┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐ + │ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │ + │ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │ + └──────────────┘ internet └───────────────┘ │ SNAT to the node's addr │ + └─────────────┬──────────────┘ + │ as `host`/`remote-node` + ┌─────────────────────────────────┼──────────────────────┐ + ▼ ▼ ▼ + LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet + (NAS, router, printers) PiHole DNS VIP .23 +``` + +### `hostNetwork: true` is required, not convenient + +This is the only genuinely interesting decision in the component, and it was forced by a *measured* +property of this cluster rather than chosen for convenience. + +In Cilium, what a packet may reach is decided by the **security identity of its source**. A +WireGuard server running as an ordinary pod must SNAT client traffic to its own pod IP (Cilium drops +a packet whose source is not the endpoint's own address), so every remote client would have arrived +everywhere wearing *that pod's* identity. On 2026-10-04 this cluster measured what that costs +(`decisions/2026-10-04-plex-advertise-plexdirect.md`): a request to the shared Gateway VIP **from +inside a pod returns `403 Access denied` from Envoy for all 16 Gateway hostnames**, because Cilium's +Gateway preserves the original source identity and enforces the *destination's* ingress policy — +and every app admits `reserved:ingress` plus named peers, never an arbitrary pod. LAN browsers work +only because they arrive as `reserved:world` / `reserved:host`. + +So the pod-networked version of this component would have produced a tunnel that came up healthy, +reported connected peers, routed packets — and 403'd **every single cluster web UI**: plex, gitea, +keycloak, nextcloud, grafana, headlamp, the apex. `*.home.keithjackson.dev` would not even have +resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately omits the `cluster` +entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the +manifests would have hinted at it. + +In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the +node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` — +byte-for-byte what a LAN client looks like. Three further problems vanish with it: + +* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes + `/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a + pod-netns build would have needed `--allowed-unsafe-sysctls` on all three kubelets — a + node-restart change to the k3s service, to run a VPN. +* **No LoadBalancer VIP**, so no `CiliumL2AnnouncementPolicy` lease lottery (the backend-blind + announcer that cost this LAN an 8-day Minecraft outage — see + `conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no + `externalTrafficPolicy` trade-off to make at all. +* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat + 1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct. + +`hostNetwork` is *permitted* here, and not by oversight: +`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector: +app-namespace=true` only, and names this exact carve-out in its own comment — "platform components +that do … e.g. Cilium, node-exporter … live in platform namespaces that do NOT carry the +app-namespace label". [`namespace.yaml`](./namespace.yaml) is deliberately unlabelled. + +**The cost, stated rather than buried:** a `hostNetwork` pod has no Cilium endpoint, so **no +NetworkPolicy constrains this workload** — not [`networkpolicy.yaml`](./networkpolicy.yaml), not +anything. That file says so at length instead of implying a protection it cannot provide. The +boundary around this port is WireGuard's peer authentication, full stop. + +### Why a DaemonSet + +The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400 +forward). A single-replica Deployment would float on every reschedule and silently invalidate it. +A DaemonSet instead puts an identical listener on all three nodes — every node answers the same +keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no +re-issued client profiles. The two idle listeners cost ~15 MiB each. + +A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node +replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node +hop into the one path whose failure mode is *"I am away from home and cannot get in"*. Fewer +translations win here. + +### The split that keeps authorisation reviewable + +| half | where | why | +|---|---|---| +| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail | +| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) | + +The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** — +so the assembled config holding the private key never touches any node's disk and dies with the pod. + +Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a +new line granting a new device access to the whole LAN. + +## What a client can reach — and how to narrow it + +By design, **everything a LAN machine can reach**: all of `192.168.2.0/24`, the shared Gateway's 16 +hostnames, PiHole, the NAS, the router's admin UI, and the internet via the house WAN. + +**That includes the three k3s nodes themselves** — `192.168.2.10/.11/.12`, and therefore the +kube-apiserver on 6443, kubelet on 10250 and SSH on 22. An enrolled device is as privileged as a +machine sitting in the house, which is what "reach the entire local network" asks for; it is +recorded here because it is the largest consequence of saying yes to that, and because nobody should +discover it later. **It is also why a lost device must be revoked promptly** (delete its stanza, +merge; ~1 min to take effect). + +To grant less, narrow it at the **client** profile first — `new-peer.sh --split` emits a +split-tunnel profile routing only the LAN and the tunnel subnet, so the device keeps its own +internet path. To enforce less *server-side*, add filtering to the `PostUp` chain in +[`daemonset.yaml`](./daemonset.yaml) (e.g. `iptables -A FORWARD -i wg0 -d 192.168.2.10 -j DROP` +before the accept rules) — the FORWARD chain there is the only enforcement point, since no +NetworkPolicy applies. + +## Operating it + +* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh ` → paste the printed `[Peer]` + stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`. +* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working. +* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready. +* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show` + (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key. +* **Full bring-up and verification:** `runbooks/vpn-remote-access.md`. + +## Prerequisites outside this directory + +1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules` + (`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has + `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it + CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard. +2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start — + the correct fail-closed state for an unenrolled VPN. +3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo, + and the half CI can never check. + +## Not built (deliberate, with reasons) + +* **No metrics / no peer-level alerting.** A WireGuard exporter would expose + `wireguard_latest_handshake_seconds` per peer — genuinely useful for spotting unexpected + activity — but it needs a second `NET_ADMIN` container in the pod that terminates the + internet-facing tunnel, and that trade was not worth making unverified. Pod-down is already + covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from + kube-prometheus-stack. **This is the top follow-up.** +* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise + handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the + reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the + `wg-server-key` Secret. +* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6 + still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point + was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's + `AllowedIPs`; not done because nothing here needs it. +* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN + browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed + into the tunnel. diff --git a/platform/vpn-ingress/daemonset.yaml b/platform/vpn-ingress/daemonset.yaml new file mode 100644 index 0000000..4a72602 --- /dev/null +++ b/platform/vpn-ingress/daemonset.yaml @@ -0,0 +1,295 @@ +--- +# ================================================================================================= +# WireGuard remote-access listener — udp/51820 in the HOST network namespace, on every node. +# ================================================================================================= +# WHY WireGuard AND NOT OpenVPN/IPsec. The requirement is "Android, Linux, Windows and other +# computing clients", from anywhere on the internet, securely. WireGuard is the only option here +# with a first-party client on all four platforms (official apps for Android/iOS/Windows/macOS, and +# in-kernel on Linux since 5.6 — so Linux clients need no userspace daemon at all). It is also the +# smallest attack surface of the three: one UDP port, ~4k lines of kernel code, no cipher +# negotiation, no PKI, no user database, and — the property that actually matters for a port +# exposed to the whole internet — it NEVER REPLIES to an unauthenticated packet. See README. +# +# ------------------------------------------------------------------------------------------------- +# hostNetwork: true — REQUIRED, not convenient. This is THE decision in this file. +# ------------------------------------------------------------------------------------------------- +# A VPN client must behave like a machine plugged into the LAN. In Cilium, what a packet is ALLOWED +# to reach is decided by the SECURITY IDENTITY of its source, and after any in-pod SNAT that +# identity is the VPN pod's — not `world`. That distinction is not academic; it was MEASURED on +# this cluster on 2026-10-04 (decisions/2026-10-04-plex-advertise-plexdirect.md): a curl of the +# shared Gateway VIP from inside a pod returns `403 Access denied` from Envoy for **all 16** +# Gateway hostnames, because Cilium's Gateway preserves the original source identity and enforces +# the DESTINATION's ingress CiliumNetworkPolicy, and every app admits `reserved:ingress` plus named +# peers — never an arbitrary pod. LAN browsers are allowed only because they arrive as +# `reserved:world` / `reserved:host`. +# +# So a pod-networked WireGuard server would have handed remote clients a tunnel on which every +# single cluster web UI returned 403 — plex, gitea, keycloak, nextcloud, grafana, headlamp, the +# apex, all of them — while `wg show` reported a perfectly healthy tunnel. PiHole would have failed +# the same way and worse: apps/wave-0/pihole-dns.yaml's ingress policy deliberately OMITS the +# `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either. +# The VPN would have looked up and been useless, which is the worst failure shape available. +# +# In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the +# NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` — +# exactly what a LAN client looks like. Three further problems disappear with it: +# * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to +# /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward` +# is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have +# required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change). +# * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost +# this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all. +# * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a +# flat 1500-byte LAN path, not Cilium's 1450-byte VXLAN. +# +# hostNetwork is PERMITTED here, and not by accident: platform/kyverno/policies/ +# restrict-host-namespaces.yaml matches `namespaceSelector: app-namespace=true` only, and names +# this exact carve-out in its own comment ("platform components that do -- e.g. Cilium, +# node-exporter -- live in platform namespaces that do NOT carry the app-namespace label"). +# namespace.yaml is deliberately unlabelled. THE COST IS REAL AND IS STATED: a hostNetwork pod +# bypasses the Cilium datapath, so NO NetworkPolicy constrains this workload (see +# networkpolicy.yaml, which says so rather than pretending otherwise). The boundary protecting +# this port is WireGuard's own cryptographic peer authentication, not the CNI. +# +# ------------------------------------------------------------------------------------------------- +# A DaemonSet, not a single-replica Deployment. +# ------------------------------------------------------------------------------------------------- +# The WAN port-forward on the SOHO router points at ONE node address, by hand, off-repo (same shape +# as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to +# whichever node the scheduler picked and silently invalidate that forward on every reschedule. A +# DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of +# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no +# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own +# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each. +# +# All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the +# DaemonSet to cover the cluster; add them if a node is ever tainted. +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: wireguard + namespace: vpn-ingress + labels: + app.kubernetes.io/name: vpn-ingress +spec: + selector: + matchLabels: + app.kubernetes.io/name: vpn-ingress + # One node at a time, so a rollout never takes every listener down at once. The node named by the + # router's forward does drop for a few seconds during its own replacement; clients re-handshake + # automatically (WireGuard retries every 5s, and rekeys every 2 minutes regardless). + updateStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + template: + metadata: + labels: + app.kubernetes.io/name: vpn-ingress + # Declared for documentation and for symmetry with the app tier. It MUST be `internet`, and + # the value is load-bearing rather than cosmetic: platform/vpn-egress/egress-steering.yaml's + # CiliumEgressGatewayPolicy selects `egress-class: vpn` pods CLUSTER-WIDE, with no namespace + # scope. Setting `vpn` here would steer every remote client's internet traffic out the PIA + # tunnel instead of the house WAN — the opposite of the requirement, and it would look like + # a working VPN with a mysterious foreign exit IP. (The selector matches on the pod label, + # so hostNetwork does not exempt this pod from being selected.) + homelab.keithjackson.dev/egress-class: internet + # WireGuard authenticates its own peers by public key; there is no UI and no IdP hop. + homelab.keithjackson.dev/auth-mode: native + spec: + # See the long rationale at the head of this file. ClusterFirstWithHostNet keeps in-cluster + # DNS working for the pod itself despite hostNetwork (without it the pod inherits the NODE's + # resolv.conf, which points at public resolvers). + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + automountServiceAccountToken: false + # ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ── + # The two halves are deliberately stored apart (peers.yaml explains why) and are only ever + # joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private + # key is therefore never written to any node's disk, never committed, and never logged. + initContainers: + - name: assemble-config + image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124 + imagePullPolicy: IfNotPresent + # POSIX sh. Note on $ handling: this script is rendered through Flux's envsubst (the + # `vpn-ingress` Kustomization declares postBuild.substituteFrom), so `${WG_CLIENT_CIDR}` + # below IS substituted on purpose — it is a declared cluster-vars knob. Every SHELL + # variable is written bare (`$net`, `$(cmd)`), which envsubst leaves untouched. There is + # deliberately NO braced shell-variable form anywhere in here: braces are what envsubst + # consumes, so a shell variable written that way silently renders to the empty string. + # (ci: flux-postbuild-substitution enforces this, and caught this very comment when it + # first spelled the hazard out literally — the check reads comments too, correctly.) + command: + - /bin/sh + - -c + - | + set -eu + umask 077 + mkdir -p /config/wg_confs + + # Derive the server's own tunnel address from the single CIDR knob, so there is no + # second knob to keep in agreement: 10.90.0.0/24 -> 10.90.0.1/24. + net=${WG_CLIENT_CIDR} + mask=$(echo "$net" | cut -d/ -f2) + base=$(echo "$net" | cut -d/ -f1 | cut -d. -f1-3) + + # The key arrives as a Secret file; strip any trailing newline the operator's shell + # added, or wg-quick rejects the config with "Key is not the correct length or format". + key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey) + if [ -z "$key" ]; then + echo "FATAL: /run/wg-server-key/privatekey is empty." >&2 + echo " See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2 + exit 1 + fi + + { + echo '[Interface]' + echo "Address = $base.1/$mask" + echo 'ListenPort = 51820' + # 1420 = 1500 (flat LAN/WAN path to the node's own NIC) - 60 bytes of WireGuard + # encapsulation (20 IPv4 + 8 UDP + 32 WireGuard). This is also what wg-quick would + # auto-derive, but it is pinned because the symptom of getting it wrong is the + # nastiest kind: handshakes and SSH work while large packets (HTTPS, file copies) + # hang. Inner packets stay under Cilium's 1450-byte VXLAN MTU too, so reaching a + # cluster pod does not fragment either. + echo 'MTU = 1420' + printf 'PrivateKey = %s\n' "$key" + # ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ── + # ip_forward is already 1 from node_baseline; re-asserted so this component does not + # silently depend on another role's sysctl file surviving. + echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1' + # SNAT client traffic to the node's address. REQUIRED: without it a LAN host would + # reply to 10.90.0.x, find no route, hand the packet to the router, and the router + # would drop it — so every connection would hang rather than fail. Matching on + # `! -o wg0` rather than a named `-o eth0` keeps this independent of the node's NIC + # name. Each rule is add-if-absent so a pod killed without PostDown does not + # accumulate duplicates in the host's tables across restarts. + echo "PostUp = iptables -t nat -C POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || iptables -t nat -A POSTROUTING -s $net ! -o wg0 -j MASQUERADE" + echo 'PostUp = iptables -C FORWARD -i wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -i wg0 -j ACCEPT' + echo 'PostUp = iptables -C FORWARD -o wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -o wg0 -j ACCEPT' + echo "PostDown = iptables -t nat -D POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || true" + echo 'PostDown = iptables -D FORWARD -i wg0 -j ACCEPT 2>/dev/null || true' + echo 'PostDown = iptables -D FORWARD -o wg0 -j ACCEPT 2>/dev/null || true' + echo '' + } > /config/wg_confs/wg0.conf + + cat /run/wg-peers/peers.conf >> /config/wg_confs/wg0.conf + chmod 600 /config/wg_confs/wg0.conf + + # Report the SHAPE of the result, never its content — the file now holds the server's + # private key, and a log line is a transcript (CLAUDE.md §6). + echo "wg0.conf assembled: $(grep -c '^\[Peer\]' /config/wg_confs/wg0.conf || true) peer(s), server $base.1/$mask, udp/51820." + securityContext: + runAsUser: 0 + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + resources: + requests: + cpu: 10m + memory: 16Mi + limits: + memory: 64Mi + volumeMounts: + - { name: wg-config, mountPath: /config } + - { name: wg-server-key, mountPath: /run/wg-server-key, readOnly: true } + - { name: wg-peers, mountPath: /run/wg-peers, readOnly: true } + containers: + - name: wireguard + # linuxserver/wireguard — provides wg, wg-quick and iptables on a maintained, multi-arch + # base. Pinned to the full upstream+build tag (1.0.20260223, published 2026-10-01) rather + # than `latest`, per the crafty/oauth2-proxy convention; Renovate maintains it. + # + # NO `PEERS` ENV IS SET, DELIBERATELY. With PEERS the image runs in "server mode" and + # GENERATES keypairs at runtime — which would put the authoritative peer list on a + # container filesystem instead of in git, and (worse) print client private keys and QR + # codes into the POD LOG, where they would be readable by anyone with log access and + # shipped to Loki. Supplying our own /config/wg_confs/wg0.conf selects the image's + # bring-up-what-you-are-given path instead. + image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124 + imagePullPolicy: IfNotPresent + env: + - name: TZ + value: "America/Chicago" + ports: + # hostNetwork, so containerPort IS the node port. Declared for documentation and so + # `kubectl get pod -o wide` shows what this listener owns. + - { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP } + securityContext: + # NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and + # write the two netfilter rules above. Everything else is dropped, and the container is + # NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule. + # SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node + # (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container + # never needs to load a kernel module. If the module is missing the pod CrashLoops with + # a clear wg-quick error rather than quietly falling back to slow userspace WireGuard. + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + add: + - NET_ADMIN + # readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and + # /var during startup. Same deferral as crafty/headlamp. + resources: + requests: + cpu: 25m + memory: 32Mi + limits: + # Kernel WireGuard does its crypto in the KERNEL, not in this container, so throughput + # does not consume the container's budget. The limit covers s6 plus the tmpfs holding + # wg0.conf (a memory-medium emptyDir counts against this). + memory: 128Mi + # `wg show wg0` exits non-zero when the interface is absent, which is the one failure that + # matters and the one that a port probe cannot see: a crashed or never-created wg0 leaves + # udp/51820 silent, and WireGuard is silent to unauthenticated traffic ANYWAY, so "nothing + # answers" is indistinguishable from "working" from the outside. This probe is the only + # thing that tells the difference. + startupProbe: + exec: + command: ["sh", "-c", "wg show wg0 >/dev/null"] + periodSeconds: 5 + failureThreshold: 24 + livenessProbe: + exec: + command: ["sh", "-c", "wg show wg0 >/dev/null"] + periodSeconds: 30 + failureThreshold: 3 + volumeMounts: + - { name: wg-config, mountPath: /config } + # Read-only so the image can SEE whether the kernel module is present and fail with a + # real message. It cannot load one (no SYS_MODULE) — that is the node's job. + - { name: lib-modules, mountPath: /lib/modules, readOnly: true } + volumes: + # tmpfs. The assembled wg0.conf contains the server's private key, so it must never land on + # a node's disk — `medium: Memory` guarantees that and guarantees it is gone when the pod is. + - name: wg-config + emptyDir: + medium: Memory + # 8Mi, not 1Mi: /config is this image's whole config ROOT, not just our one file — its + # s6 init also writes templates and state there, and a tmpfs that small would ENOSPC + # during startup rather than at a point anyone would connect to the volume size. Still + # counted against the container's 128Mi limit, and still far more than wg0.conf needs. + sizeLimit: 8Mi + # The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by + # name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this + # tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key + # that passed through a session transcript would have to be treated as compromised. + # DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace + # and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy + # in the operator's password manager as well — restoring it is what makes every already + # enrolled client work again without re-enrolling. + - name: wg-server-key + secret: + secretName: wg-server-key + defaultMode: 0400 + items: + - { key: privatekey, path: privatekey } + - name: wg-peers + configMap: + name: wg-peers + - name: lib-modules + hostPath: + path: /lib/modules + type: Directory diff --git a/platform/vpn-ingress/kustomization.yaml b/platform/vpn-ingress/kustomization.yaml new file mode 100644 index 0000000..96a158c --- /dev/null +++ b/platform/vpn-ingress/kustomization.yaml @@ -0,0 +1,30 @@ +--- +# WireGuard remote-access VPN (north-south INGRESS). Reconciled by its OWN non-gating post-platform +# Flux Kustomization (clusters/base/vpn-ingress.yaml, dependsOn platform-config, wait:false) — the +# same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be +# legitimately unhealthy must never sit in the gating `platform` tier, where one component's +# unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating +# stall"). That rationale is unusually concrete here: until the operator creates the `wg-server-key` +# Secret (runbooks/vpn-remote-access.md step 1) this DaemonSet CANNOT start, so putting it in the +# gating tier would have taken the whole platform down on merge. +# +# NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + # The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file). + - peers.yaml + # The listener itself: hostNetwork DaemonSet, udp/51820 on every node. + - daemonset.yaml + # Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file + # says so at length rather than implying a protection it cannot provide. + - networkpolicy.yaml +# +# DELIBERATELY ABSENT: a Secret. The server's private key (`wg-server-key`) is created out of band +# by the operator and referenced BY NAME from daemonset.yaml. There is no `*.sops.yaml` here, so +# this component needs NO SOPS decryption stanza in clusters/dev or clusters/prod — unlike +# crafty/headlamp/velero. If that key is ever migrated into SOPS, BOTH halves must land together: +# the file here, and a `decryption:` patch naming the `vpn-ingress` Kustomization in each overlay. +# Without the patch, kustomize hands Flux the raw ciphertext and the pod mounts a key full of +# ENC[...] — which applies cleanly and fails only at handshake time. diff --git a/platform/vpn-ingress/namespace.yaml b/platform/vpn-ingress/namespace.yaml new file mode 100644 index 0000000..173a763 --- /dev/null +++ b/platform/vpn-ingress/namespace.yaml @@ -0,0 +1,26 @@ +--- +# WireGuard REMOTE-ACCESS VPN (north-south INGRESS) — the counterpart to platform/vpn-egress. +# +# Do not confuse the two. They are opposite directions and share nothing but the word "VPN": +# * platform/vpn-egress — pods OUT to the internet through a PIA tunnel (ADR-0015, R-NET-05/06/07). +# * platform/vpn-ingress — remote CLIENTS IN from the internet to this LAN. THIS directory. +# +# NOTE: intentionally NOT homelab.keithjackson.dev/app-namespace=true. This is PLATFORM network +# infrastructure (like cilium / vpn-egress), not an application, so Kyverno does NOT generate the app +# default-deny baseline here and the app-scoped auth-mode / egress-class / host-namespace policies do +# not apply — every Kyverno policy in platform/kyverno/policies matches namespaceSelector +# app-namespace=true. THAT EXEMPTION IS LOAD-BEARING HERE, not incidental: this workload runs with +# `hostNetwork: true`, which `restrict-host-namespaces` forbids in an app namespace. It is permitted +# here by that policy's OWN stated carve-out — "platform components that do [need host namespaces] -- +# e.g. Cilium, node-exporter -- live in platform namespaces that do NOT carry the app-namespace +# label, so they are out of scope". daemonset.yaml documents why hostNetwork is REQUIRED rather than +# convenient; it is the single most consequential decision in this component. +# +# NO `gateway-access` label. This component serves no HTTP and attaches no HTTPRoute to the shared +# Gateway — it is a raw UDP listener. There is deliberately nothing here for the L7 Gateway to route. +apiVersion: v1 +kind: Namespace +metadata: + name: vpn-ingress + labels: + app.kubernetes.io/name: vpn-ingress diff --git a/platform/vpn-ingress/networkpolicy.yaml b/platform/vpn-ingress/networkpolicy.yaml new file mode 100644 index 0000000..7bf8ed8 --- /dev/null +++ b/platform/vpn-ingress/networkpolicy.yaml @@ -0,0 +1,56 @@ +--- +# ================================================================================================= +# READ THIS BEFORE TRUSTING THIS FILE: it does NOT constrain the WireGuard listener. +# ================================================================================================= +# Every other platform namespace in this repo (crafty, headlamp, homepage, runbooks) ships an +# explicit default-deny baseline as defense in depth, because Kyverno's generator only covers +# `app-namespace=true` namespaces. This file keeps that convention — but it would be dishonest to +# leave it at that, because the convention does not actually bite here: +# +# THE `wireguard` DaemonSet RUNS WITH hostNetwork: true, SO IT HAS NO CILIUM ENDPOINT, AND NO +# NetworkPolicy OR CiliumNetworkPolicy APPLIES TO IT — NEITHER THIS ONE NOR ANY OTHER. +# +# That is the documented reason restrict-host-namespaces forbids hostNetwork in app namespaces at +# all ("a pod on hostNetwork bypasses the CNI (Cilium) datapath and therefore ESCAPES the +# default-deny NetworkPolicy fabric -- it is the most direct way to defeat R-NET-01"). Here that +# escape is the deliberate, required design (daemonset.yaml explains why: without it, remote +# clients carry a pod identity and are 403'd by every Gateway hostname). A CiliumNetworkPolicy +# admitting `world` to udp/51820 would therefore be theatre: it would read as the control that +# protects this port, and it would be enforcing nothing. So there isn't one. +# +# WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not: +# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets +# NO REPLY — see README "Why this is safe to expose". This is the real control. +# 2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node. +# 3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement +# ("clients should reach the entire local network"). README "What a client can reach" states +# the consequence in full, including reach to the nodes' own control plane, and gives the +# narrowing recipe for an operator who wants less than that. +# +# WHAT THIS FILE IS STILL FOR: it is the namespace's standing posture for any FUTURE pod-networked +# workload added here (a metrics exporter is the likely one — README "Not built"). Without it, such +# a pod would be born unrestricted, because Kyverno will not generate a baseline in a namespace +# that is not labelled `app-namespace=true`, and this one deliberately is not. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-deny + namespace: vpn-ingress + labels: + homelab.keithjackson.dev/baseline: default-deny +spec: + podSelector: {} + policyTypes: + - Ingress + - Egress + ingress: [] + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 diff --git a/platform/vpn-ingress/peers.yaml b/platform/vpn-ingress/peers.yaml new file mode 100644 index 0000000..a6d7137 --- /dev/null +++ b/platform/vpn-ingress/peers.yaml @@ -0,0 +1,73 @@ +--- +# ============================================================================================ +# THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose. +# ============================================================================================ +# WireGuard has no usernames, no passwords and no PKI. Authorisation is exactly one thing: a +# device's PUBLIC key appears in this file, or it cannot complete a handshake. An unlisted key +# gets no reply at all — not a rejection, silence (see README "Why this is safe to expose"). +# Revoking a device is therefore deleting its stanza, and nothing else. +# +# WHY PLAINTEXT, AND WHY THIS IS NOT A SECRET LEAK. A WireGuard peer's PUBLIC key is public by +# construction — it is the half you are supposed to hand to the server, and holding it grants +# nothing. The only secret in the whole scheme is the SERVER's private key, which is NOT here: +# it lives in the `wg-server-key` Secret and is assembled into the running config at pod start +# (daemonset.yaml). Keeping the register in cleartext buys the property that matters most for an +# access-control list: **"who can get into my house" is reviewable in a pull-request diff**, and +# its history is `git log` on this file. A SOPS-encrypted register would hide exactly the change +# a reviewer most needs to see — a new line granting a new device entry to the whole LAN. +# +# This is the same division of labour the repo already uses for the opposite case: a Secret's KEY +# NAMES are cleartext while values are encrypted (.sops.yaml `encrypted_regex`), so diffs stay +# readable. Here the whole file is the key-name half. +# +# ADDING A DEVICE — never hand-roll this, and never generate a key in a chat transcript. +# ./infra/vpn-ingress/new-peer.sh +# It generates the keypair ON THE OPERATOR'S MACHINE, writes/renders the client profile (and a +# QR code for phones), prints ONLY the public half for pasting below, and never transmits or +# logs the private half. Then: add the stanza, PR, merge; Flux rolls the DaemonSet. +# +# ADDRESS ALLOCATION. ${WG_CLIENT_CIDR} is the tunnel subnet; .1 is the server (every node). +# Give each device ONE address as a /32 in AllowedIPs. A /32 is not a formality — on the SERVER +# side AllowedIPs is the cryptokey ROUTING table AND the inbound source filter: a peer may only +# send from, and receive traffic for, the addresses listed in its own stanza. A sloppy /24 here +# would let any one device impersonate every other device's tunnel address. +# +# .1 the server (this DaemonSet, on every node) +# .2 - .254 devices, one each, allocated below. Keep this list dense and in order so the +# next free octet is obvious; never re-use an octet after revoking a device +# (an old client profile would silently start working again as someone else). +# +# PRESHARED KEYS are deliberately NOT used. They add a symmetric post-quantum hedge on top of +# the Noise handshake and would be a genuine secret — which would force this register into SOPS +# and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK +# belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken". +apiVersion: v1 +kind: ConfigMap +metadata: + name: wg-peers + namespace: vpn-ingress + labels: + app.kubernetes.io/name: vpn-ingress +data: + # Appended VERBATIM to the generated [Interface] block, in this order, at pod start. + # + # *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. *** + # An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of + # authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the + # DaemonSet will run and listen, and every handshake will be refused because no key is trusted. + # The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible + # (`wg show` reports an interface with no peers). + # + # COPY THIS TEMPLATE, uncomment, one block per device: + # + # # pixel-8 — Keith's phone, enrolled 2026-10-05 + # [Peer] + # PublicKey = + # AllowedIPs = 10.90.0.2/32 + # + # Keep the `# — , enrolled ` comment line. It is the only place a key is + # tied to a human-recognisable device; `wg show` can only ever print the key itself, so without + # it an incident responder cannot tell which stanza to delete. + peers.conf: | + # No peers enrolled yet. See the template in this file's comments and + # ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device. diff --git a/runbooks/README.md b/runbooks/README.md index 3f07cc9..1b9493c 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -120,6 +120,9 @@ Each affected runbook says so on its own page and gives the by-hand equivalent. - **`vpn-egress-stale`** — the tunnel is UP but `egress-class=vpn` pods have no egress: Cilium's cached ifindex outlived a reconnect. Also covers the watchdog for that rule going inert. - **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class. +- **`vpn-remote-access`** — the INBOUND remote-access VPN (WireGuard, udp/51820): standing it up, + enrolling and revoking devices, and the off-LAN checks that are the only honest proof it works. + The opposite direction to the two pages above — do not confuse them. ### Backups and DR - **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index 0cc6e4a..fbf5bba 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -113,3 +113,8 @@ configMapGenerator: - update-rollback=update-rollback.md - vpn-down=vpn-down.md - vpn-egress-stale=vpn-egress-stale.md + # Operator procedure, not an alert page. Served because the component it completes is INERT + # until three off-repo manual steps are done (kernel module, server key, WAN port-forward) — + # and because the one check that actually proves it cannot be run from inside the house. + # Peer-level alerting does not exist yet (it needs an exporter); this page is the substitute. + - vpn-remote-access=vpn-remote-access.md diff --git a/runbooks/vpn-remote-access.md b/runbooks/vpn-remote-access.md new file mode 100644 index 0000000..4f4acd0 --- /dev/null +++ b/runbooks/vpn-remote-access.md @@ -0,0 +1,203 @@ +# vpn-remote-access — standing up the WireGuard remote-access VPN, and proving it + +**Status: the in-cluster half is shipped; the VPN is INERT until three manual steps are done.** +All three are off-repo by necessity — a kernel module, a private key, and a router port-forward. +**Alert:** pod-down is covered generically (`KubePodNotReady` / `KubeDeploymentReplicasMismatch`). +There is **no peer-level alerting** — see "Why there is no connection alert". This is an **operator +procedure**, not an incident page. +**Files:** `platform/vpn-ingress/` (namespace, peer register, DaemonSet, baseline policy), +`clusters/base/vpn-ingress.yaml` (its Flux tier), `clusters/base/cluster-vars.yaml` +(`WG_CLIENT_CIDR`), `infra/ansible/roles/node_baseline/defaults/main.yml` (the `wireguard` module), +`infra/vpn-ingress/new-peer.sh` (device enrolment). +**Design rationale:** `platform/vpn-ingress/README.md` — read "Why `hostNetwork` is required" +before changing anything about how this is wired. +**Not to be confused with** `vpn-down` / `vpn-egress-stale`, which are about `platform/vpn-egress` +(pods *out* through PIA). This page is remote clients *in*. + +## What this gives you + +A device — Android, Linux, Windows, macOS, iOS — dials in over **udp/51820** from anywhere and +behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's web UIs, PiHole +DNS, and internet egress out the house WAN. + +Authorisation is one thing only: the device's **public key is in +`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all. + +> **An enrolled device is as privileged as a machine sitting in the house** — including the three +> k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach +> the entire local network" means. Revoke a lost device promptly: delete its stanza, PR, merge. + +## Step 1 — load the `wireguard` module on every node (once) + +The DaemonSet creates its `wg0` link in the host netns with `CAP_NET_ADMIN` only; it is +deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module. Without it +`wg-quick` fails outright and the pod CrashLoops. + +```bash +# operator action (Ansible is not an agent action) +cd infra/ansible && ./site.sh --tags system --limit k3s_prod +``` + +Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded: + +```bash +# expected: a `wireguard` line, and the modules-load.d file naming it for the next boot +lsmod | grep '^wireguard' +grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf +``` + +## Step 2 — create the server key (once) + +This is the **only** secret in the component, and the one thing that must never be generated in an +agent session or pasted into a transcript. Generate it on your own machine: + +```bash +mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home +umask 077 +wg genkey > ~/.config/wireguard-home/server.key # keep this; see "Back it up" below +``` + +Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that +lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6): + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \ + --from-file=privatekey=/work/.cache/server.key' +``` + +…where `/work/.cache/server.key` is a path you copied the key to **inside the repo's gitignored +cache** (or mount it however you prefer — the requirement is only that the bytes never pass through +a log). Delete that copy afterwards: `shred -u .cache/server.key`. + +Confirm it exists **by key name only**: + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1' +# expected: {"privatekey -- a NAME. If you see base64 payload, you printed the key. Rotate it. +``` + +**Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that +passed through a session transcript would have to be treated as compromised. The trade-off is that +this Secret is **not reconstructible from git** — so: + +**Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only +`monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured +off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work +again after a cluster rebuild — **losing it means re-enrolling every device.** + +## Step 3 — forward udp/51820 on the router (once) + +On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g. +`192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check. + +Pick any node — all three run an identical listener. Note which one you chose here: + +``` +# WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set +``` + +**Node loss = failover is this one line.** Re-point the forward at another node address; every +device keeps working with the same profile, because all three nodes answer the same keys. + +Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is +already silent to unauthenticated probes (README, "Why this is safe to expose"). + +## Step 4 — confirm the listener is healthy + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress get ds wireguard -o wide' +# expected: DESIRED 3 CURRENT 3 READY 3 + +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show' +# expected: `interface: wg0`, `listening port: 51820`, and (before step 5) NO peers. +``` + +A zero-peer interface that is Ready is the **correct** state before any device is enrolled: it +listens, and refuses every handshake because no key is trusted. + +The readiness/liveness probe is `wg show wg0`, and that is load-bearing rather than cosmetic: a +port probe cannot distinguish "wg0 never came up" from "working", because WireGuard is silent to +unauthenticated traffic either way. If the pod is Ready, the interface exists. + +## Step 5 — enrol a device + +```bash +./infra/vpn-ingress/new-peer.sh pixel-8 2 # full tunnel (LAN + WAN egress) +./infra/vpn-ingress/new-peer.sh thinkpad 3 --split # split tunnel (LAN only) +``` + +It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout, +prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then: + +1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the + `# — , enrolled ` comment. `wg show` can only ever print the key, so that + comment is the only record of which stanza belongs to which device. +2. PR → merge. Flux rolls the DaemonSet (~1 min). +3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on + Windows/macOS; `/etc/wireguard/.conf` + `wg-quick@` on Linux). +4. **Delete the profile from your machine**: `shred -u `. It is the only copy of that + device's private key — the server never sees it and it cannot be re-derived. + +Until the PR is merged the device handshakes and gets silence, exactly like an unknown key. + +## Step 6 — verify it end to end (from OFF the LAN) + +**This is the only honest test, and it cannot be run from inside the house.** Put the phone on +cellular, with Wi-Fi off, and check all four — each proves a different hop: + +| # | Check | Proves | +|---|---|---| +| 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza | +| 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path | +| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd | +| 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier | + +Then confirm server-side: + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show' +# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer. +``` + +**If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not +WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up. + +**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60 +bytes of WireGuard encapsulation); drop the client to `1280` and re-test. + +**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside — +by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes +if the router is not delivering the packets, and a `handshake` with no completed session if it is. + +## Revoking a device + +Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the +next reconcile. **Never re-use that address octet**: an old profile would silently start working +again as a different device. + +## Why there is no connection alert + +A WireGuard exporter would publish `wireguard_latest_handshake_seconds` per peer and make +"something connected at 03:00" alertable. It needs a second `NET_ADMIN` container inside the pod +that terminates the internet-facing tunnel, and that was not worth shipping unverified. **It is the +top follow-up for this component.** Until then: + +* pod-down is covered generically by kube-prometheus-stack's `KubePodNotReady` / + `KubeDeploymentReplicasMismatch`; +* "who is connected" is an on-demand question — `wg show`, above; +* **never** `wg showconf` — it prints the server's private key. + +## Known limitations + +* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside + the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point. +* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's + address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would + preserve them — at the cost of another hand-maintained off-repo route. +* **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients + re-handshake automatically (retry every 5s, rekey every 2 min).