Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
f92707f38c |
No files matched your search
@@ -21,7 +21,8 @@ infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible,
|
||||
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
|
||||
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
|
||||
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
|
||||
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
|
||||
vpn-egress, vpn-ingress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
|
||||
(vpn-egress = pods OUT via PIA; vpn-ingress = remote clients IN via WireGuard)
|
||||
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
|
||||
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
|
||||
Split out because CRs and the CRDs they use cannot apply in one pass.
|
||||
|
||||
@@ -102,6 +102,7 @@ data:
|
||||
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
|
||||
# GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
|
||||
# plex-remote + pihole: PLEX_L4_VIP),
|
||||
# vpn-ingress (WG_CLIENT_CIDR),
|
||||
# velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
|
||||
# VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
|
||||
# UNSUBSTITUTED path would ship the literal string into the cluster.
|
||||
@@ -207,6 +208,31 @@ data:
|
||||
# prod -> 192.168.2.0/24 (prod LAN; nodes .10/.11/.12)
|
||||
LAN_CIDR: "192.168.1.0/24" # ingress scope for the PiHole DNS listener
|
||||
|
||||
# --- Remote-access VPN client subnet (platform/vpn-ingress, WireGuard) --------------------------
|
||||
# The tunnel subnet handed to REMOTE VPN clients. `.1` is the server (the hostNetwork DaemonSet,
|
||||
# on every node); `.2`-`.254` are devices, allocated one /32 each in
|
||||
# platform/vpn-ingress/peers.yaml. Consumed twice in the pod's config-assembly script: to derive
|
||||
# the server's own address, and as the `-s` match of the MASQUERADE rule that makes LAN replies
|
||||
# routable.
|
||||
#
|
||||
# IT IS A KNOB RATHER THAN A LITERAL FOR ONE REASON: it is the only address range in this cluster
|
||||
# that must be checked against EVERY OTHER range at once, so it belongs where those are declared.
|
||||
# 10.90.0.0/24 was chosen to collide with none of them:
|
||||
# 10.0.0.0/16 Cilium cluster-pool pod CIDR (${CLUSTER_POD_CIDR})
|
||||
# 10.43.0.0/16 k3s service CIDR (measured live 2026-10-05: kubernetes svc
|
||||
# clusterIP 10.43.0.1)
|
||||
# 10.72.192.0/18 the PIA tunnel's pushed addresses (platform/vpn-egress)
|
||||
# 192.168.1.0/24 dev LAN / pfSense LAN (${LAN_CIDR} base default)
|
||||
# 192.168.2.0/24 prod LAN (${LAN_CIDR} on prod)
|
||||
#
|
||||
# NO PER-ENVIRONMENT OVERRIDE, and that is deliberate rather than an omission: unlike every VIP
|
||||
# above, this range is NOT drawn from the LAN. It is private to the tunnel, so the same value is
|
||||
# correct in both environments and a prod patch would be a no-op that implies otherwise.
|
||||
#
|
||||
# IF YOU CHANGE IT, every already-issued client profile breaks (its `Address` is from this range)
|
||||
# and must be re-rendered — so treat it as fixed once the first device is enrolled.
|
||||
WG_CLIENT_CIDR: "10.90.0.0/24" # remote VPN clients; .1 = server. NOT a LAN range
|
||||
|
||||
# --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
|
||||
# THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
|
||||
# library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
|
||||
|
||||
@@ -71,4 +71,11 @@ resources:
|
||||
# controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
|
||||
# Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
|
||||
- backup.yaml # Flux Kustomization -> ./platform/velero (dependsOn platform-config)
|
||||
# WireGuard remote-access VPN (north-south INGRESS, udp/51820) — POST-platform tier
|
||||
# (dependsOn platform-config, wait:false). NOT the same thing as platform/vpn-egress, which sends
|
||||
# pods OUT through PIA; this lets remote CLIENTS IN. Non-gating is load-bearing rather than
|
||||
# conventional here: its DaemonSet cannot start until the operator creates the out-of-band
|
||||
# `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have
|
||||
# stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch.
|
||||
- vpn-ingress.yaml # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config)
|
||||
- cluster-vars.yaml # overlay knob surface (base defaults)
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
# Flux Kustomization: WireGuard remote-access VPN — reconciles ./platform/vpn-ingress.
|
||||
#
|
||||
# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false). Not a line in
|
||||
# platform/kustomization.yaml, for the standard reason plus one specific to this component:
|
||||
#
|
||||
# * STANDARD: a leaf must never sit in the gating `platform` tier. While `platform` is not-Ready
|
||||
# NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
|
||||
# (DECISIONS-LOG: "Headlamp gating stall").
|
||||
# * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
|
||||
# operator creates the out-of-band `wg-server-key` Secret (runbooks/vpn-remote-access.md
|
||||
# step 1). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
|
||||
# the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
|
||||
# cause (a missing manual step) invisible from any Flux condition.
|
||||
#
|
||||
# WHY dependsOn platform-config AND NOT cilium. This component needs no Gateway, no TLS cert, no
|
||||
# Keycloak and no LB-IPAM pool — it has no HTTPRoute and no LoadBalancer Service, and hostNetwork
|
||||
# means it does not even need a Cilium endpoint. It could legally reconcile much earlier. It is
|
||||
# pinned behind platform-config anyway so that the ONE thing remote clients actually consume
|
||||
# through it — split-horizon DNS and the shared Gateway — exists by the time the tunnel does.
|
||||
# A tunnel that comes up before the things on the far side of it is a support call, not a feature.
|
||||
#
|
||||
# wait:false: nothing in the graph dependsOn this, and its pods are legitimately not-Ready whenever
|
||||
# the server key is absent or the `wireguard` kernel module has not been loaded on a node. Neither
|
||||
# condition should be able to gate anything else, even if a future resource is mistakenly pointed
|
||||
# at it.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: vpn-ingress
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/vpn-ingress
|
||||
# REQUIRED, not boilerplate: platform/vpn-ingress/daemonset.yaml renders ${WG_CLIENT_CIDR} into
|
||||
# the config-assembly script. Without this block that token becomes the EMPTY STRING and the
|
||||
# Kustomization still reports Ready (Flux's substitution fails OPEN — ci:
|
||||
# flux-postbuild-substitution), leaving an initContainer that derives a server address of ".1/"
|
||||
# and a MASQUERADE rule with no source. Both fail loudly at pod start, but the Kustomization
|
||||
# would look healthy.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# NO `decryption:` stanza is needed in the overlays: this path carries no *.sops.yaml. The server
|
||||
# key is an out-of-band Secret (platform/vpn-ingress/kustomization.yaml says what to do if that
|
||||
# ever changes).
|
||||
@@ -0,0 +1,225 @@
|
||||
---
|
||||
date: 2026-10-05
|
||||
branch: feat/vpn-remote-access-wireguard
|
||||
title: Remote-access VPN — WireGuard on udp/51820, hostNetwork because a pod identity would 403 every Gateway hostname
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
New platform component `platform/vpn-ingress/`: a **WireGuard** remote-access VPN so the operator's
|
||||
Android / Linux / Windows / macOS devices can dial in from anywhere on the internet and behave like
|
||||
machines on the LAN — full reach to `192.168.2.0/24` plus internet egress out the house WAN.
|
||||
|
||||
This is the **opposite direction** to the existing `platform/vpn-egress` (pods *out* through PIA,
|
||||
ADR-0015). They share nothing but the word "VPN"; both READMEs now say so, and `README.md`'s layout
|
||||
block labels the two.
|
||||
|
||||
**Transport: WireGuard, and the security argument is the handshake's failure mode, not "it is
|
||||
encrypted".** WireGuard's first handshake message carries a MAC keyed with the responder's own
|
||||
public key, so a sender that does not already know it gets **no reply at all** — no error, no reset,
|
||||
no banner. From the internet, udp/51820 is indistinguishable from a closed port: nothing to
|
||||
brute-force, no version string, no amplification. That is what makes an unfiltered WAN port
|
||||
defensible here, and it is why there is deliberately no `loadBalancerSourceRanges`-style allowlist
|
||||
(which would also break roaming clients). It also has first-party clients on all four named
|
||||
platforms, which OpenVPN (needs a CA + per-device certs + a CRL) and IKEv2 do not.
|
||||
|
||||
**`hostNetwork: true`, and this is THE decision — forced by a measured property of this cluster.**
|
||||
A pod-networked WireGuard server must SNAT client traffic to its own pod IP (Cilium drops a packet
|
||||
whose source is not the endpoint's address), so every remote client would have arrived wearing *that
|
||||
pod's* security identity. `decisions/2026-10-04-plex-advertise-plexdirect.md` measured what that
|
||||
costs: a request to the shared Gateway VIP **from inside a pod returns `403 Access denied` from
|
||||
Envoy for all 16 Gateway hostnames**, because Cilium's Gateway preserves the original source
|
||||
identity and enforces the *destination's* ingress policy, and every app admits `reserved:ingress`
|
||||
plus named peers — never an arbitrary pod. So the obvious build would have produced a tunnel that
|
||||
came up healthy, showed connected peers, routed packets, and 403'd every cluster web UI — and
|
||||
`*.home.keithjackson.dev` would not even have resolved, because `apps/wave-0/pihole-dns.yaml`'s
|
||||
ingress policy deliberately omits the `cluster` entity. In the host netns there is no pod identity
|
||||
to carry: forwarded traffic is SNATed to the node address and arrives as `host`/`remote-node`,
|
||||
byte-for-byte what a LAN client looks like. Three further problems vanish with it: `ip_forward` is
|
||||
already 1 on these nodes (so no `--allowed-unsafe-sysctls` kubelet change — it is *not* a safe
|
||||
sysctl); no LoadBalancer VIP, so no L2-announcement lease lottery and no `externalTrafficPolicy`
|
||||
trade-off; and no double encapsulation, which is what makes `MTU = 1420` simply correct.
|
||||
|
||||
`hostNetwork` is permitted by `restrict-host-namespaces`' own stated carve-out (it matches
|
||||
`app-namespace=true` only, and names Cilium/node-exporter as the precedent); `namespace.yaml` is
|
||||
deliberately unlabelled. **The cost is stated, not buried:** a hostNetwork pod has no Cilium
|
||||
endpoint, so **no NetworkPolicy constrains this workload**. `networkpolicy.yaml` says that at
|
||||
length rather than shipping an `allow world → 51820` rule that would read as the control and
|
||||
enforce nothing; the real boundary is WireGuard's peer authentication.
|
||||
|
||||
**A DaemonSet, not a Deployment.** The WAN forward names one node address by hand, off-repo (same
|
||||
shape as the plex tcp/32400 forward). A floating single replica would silently invalidate it on
|
||||
every reschedule; an identical listener on all three nodes makes **failover a one-line router
|
||||
change** with no cluster work and no re-issued client profiles. A pinned LoadBalancer VIP was the
|
||||
considered alternative and was rejected: it reintroduces the L2 lease, SNAT and an extra node hop
|
||||
into the one path whose failure mode is "I am away from home and cannot get in".
|
||||
|
||||
**Authorisation is split so that it stays reviewable.** Peer **public** keys live in
|
||||
`platform/vpn-ingress/peers.yaml` in **plaintext** — a public key grants nothing by itself, and this
|
||||
makes "who can get into my house" a PR diff with `git log` as its audit trail. The only real secret,
|
||||
the server's private key, is the `wg-server-key` Secret, **created out of band by the operator**: an
|
||||
agent must not mint a long-lived private key (CLAUDE.md §6), and a key that passed through a session
|
||||
transcript would have to be treated as compromised. The two halves are joined only at pod start, by
|
||||
an init container, into a `medium: Memory` emptyDir — so the assembled config holding the key never
|
||||
touches a node's disk and dies with the pod.
|
||||
|
||||
**There are no peers yet, and that is the correct initial state**, for the same reason. The VPN ships
|
||||
INERT-BUT-HEALTHY: a zero-peer interface listens and refuses every handshake.
|
||||
|
||||
Files:
|
||||
|
||||
- `platform/vpn-ingress/` (new): `namespace.yaml`, `peers.yaml` (the plaintext peer register),
|
||||
`daemonset.yaml` (hostNetwork listener + config assembly), `networkpolicy.yaml` (namespace
|
||||
baseline, with the honest caveat), `kustomization.yaml`, `README.md` (the design record).
|
||||
- `clusters/base/vpn-ingress.yaml` (new): its own **non-gating post-platform** Flux Kustomization
|
||||
(`dependsOn platform-config`, `wait:false`). Non-gating is load-bearing rather than conventional
|
||||
here: the DaemonSet *cannot* start until the operator creates the server key, so in the gating
|
||||
`platform` tier an unfinished manual step would have stalled every apps wave on merge.
|
||||
- `clusters/base/kustomization.yaml`: the new tier added.
|
||||
- `clusters/base/cluster-vars.yaml`: new `WG_CLIENT_CIDR` (`10.90.0.0/24`), declared next to
|
||||
`LAN_CIDR` because it is the one range that must be collision-checked against every other; the
|
||||
comment enumerates the five it avoids. **No prod override, deliberately** — unlike every VIP, this
|
||||
range is private to the tunnel, so the same value is correct in both environments and a prod patch
|
||||
would imply otherwise. Substitution inventory updated.
|
||||
- `infra/ansible/roles/node_baseline/defaults/main.yml`: `wireguard` added to
|
||||
`node_baseline_kernel_modules`. The pod has `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so
|
||||
it cannot autoload the module — it CrashLoops with a clear error rather than quietly degrading to
|
||||
userspace wireguard-go (which would need `/dev/net/tun`, a device a hostPath mount cannot actually
|
||||
grant, as the `/dev/dri` attempt proved).
|
||||
- `infra/vpn-ingress/new-peer.sh` (new): enrols a device. Generates the keypair on the operator's
|
||||
machine, writes a 0600 profile **outside** the checkout (it refuses to write into the repo),
|
||||
prints a QR code for phones, and prints only the public `[Peer]` stanza. It also refuses a
|
||||
duplicate address, because two devices on one address does not error anywhere — the server routes
|
||||
to whichever handshook last, so both appear to work intermittently.
|
||||
- `runbooks/vpn-remote-access.md` (new) + its `kustomization.yaml` generator entry + `README.md`
|
||||
index line: the three manual steps, the four off-LAN verification checks, revocation, and why no
|
||||
connection alert exists.
|
||||
- `README.md`: `vpn-ingress` added to the layout, with the egress/ingress distinction spelled out.
|
||||
|
||||
**THREE MANUAL STEPS REMAIN AND THE VPN DOES NOT WORK UNTIL ALL THREE ARE DONE** (all off-repo by
|
||||
necessity; `runbooks/vpn-remote-access.md`): run `node_baseline` to load the kernel module, create
|
||||
the `wg-server-key` Secret, and forward WAN udp/51820 to one node address on the SOHO router.
|
||||
|
||||
**SENSITIVE PATHS — this PR touches `clusters/**` (three files), so `ci/approval-policy.sh` will
|
||||
BLOCK it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
|
||||
"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
|
||||
PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
|
||||
|
||||
## What I verified
|
||||
|
||||
Live prod, read-only, 2026-10-05, plus the offline gate.
|
||||
|
||||
- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
|
||||
`spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
|
||||
**pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
|
||||
and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
|
||||
Run from an isolated git worktree with both CLAUDE.md §3 workarounds applied
|
||||
(`DEV_KUBECONFIG=/dev/null` plus an `ENGINE` shim bind-mounting the main checkout's `.git` and
|
||||
the sibling spec repo) — which is why `spec-immutability` is **green here and skipped in CI**,
|
||||
where the runner has no spec checkout. `approval-policy` passing locally is an offline N/A, not a
|
||||
verdict; the real verdict is the CI run below.
|
||||
- **The CI gate ran on PR #55 (run 244, head `1a67fc1b`) and `approval-policy` is the ONLY failing
|
||||
stage** — read from the job log, not from the red tick. Verbatim: "This PR changes the pipeline or
|
||||
the security posture it rests on, and has NO approving review from anyone other than its author
|
||||
('gitea-admin')", naming exactly the three files I predicted:
|
||||
`clusters/base/cluster-vars.yaml`, `clusters/base/kustomization.yaml`,
|
||||
`clusters/base/vpn-ingress.yaml`. Every other stage that ran reported green —
|
||||
`gitea-settings`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. So the branch is blocked
|
||||
on the operator's `/approve-override <reason>` (posted as a PR **review** of type "Comment") and
|
||||
on nothing else. **I have not posted it** — CLAUDE.md §1.
|
||||
- **Branch protection is live, measured by CI's own `gitea-settings` stage** (it has the token this
|
||||
workstation lacks): `required_approvals = 0`, `enable_status_check = true`,
|
||||
`status_check_contexts = ["gate / gate (pull_request)"]`, `block_admin_merge_override = true`,
|
||||
`enable_push = false`, `dismiss_stale_approvals = true`, and **1 runner online**. That is the §2b
|
||||
re-measurement of CLAUDE.md §2's table for this branch, not a restatement of it.
|
||||
- **`flux-postbuild-substitution` (L0) actually exercised the new wiring, and caught a real defect
|
||||
in it.** On the first run it FAILED, naming `platform/vpn-ingress/daemonset.yaml:118` and
|
||||
attributing it to "Flux Kustomization 'vpn-ingress' (clusters/base/vpn-ingress.yaml, path
|
||||
./platform/vpn-ingress)". Two things are proven by that message: the new tier's
|
||||
`postBuild.substituteFrom` declaration is correctly detected (so `${WG_CLIENT_CIDR}` will be
|
||||
substituted, not shipped literally), and the check reads comments as well as values — the hazard
|
||||
was a comment that spelled out the forbidden braced form literally while claiming not to use it.
|
||||
Fixed; the check now passes 444/444 with 0 hazards.
|
||||
- **The `wireguard` kernel module is available on these nodes by construction**: all three run
|
||||
Debian 13 (trixie) on kernel `7.1.8+deb13-amd64` (measured), and WireGuard has been in-tree since
|
||||
5.6, so it is present in the distro kernel and only needs loading.
|
||||
- **`net.ipv4.ip_forward = 1` is already written by `node_baseline`**
|
||||
(`roles/node_baseline/tasks/main.yml`, `/etc/sysctl.d/99-k3s-cilium.conf`), and **no kubelet
|
||||
carries `--allowed-unsafe-sysctls`** (grepped across `roles/k3s_server`, `roles/node_baseline` and
|
||||
`group_vars`: zero hits). That pair is what makes the host netns the only option that does not
|
||||
require a node-restart change.
|
||||
- **Cilium's datapath, measured from `cilium-config` rather than from the values file**:
|
||||
`enable-ipv4-masquerade=true`, `enable-bpf-masquerade=true`, `routing-mode=tunnel`,
|
||||
`tunnel-protocol=vxlan`, `ipv4-native-routing-cidr` EMPTY, Cilium `v1.20.1`. The vxlan underlay is
|
||||
where `MTU = 1420` comes from, and the empty native-routing CIDR is why pod→LAN is masqueraded
|
||||
today.
|
||||
- **`WG_CLIENT_CIDR` collides with nothing.** Pod CIDR `10.0.0.0/16` (`CLUSTER_POD_CIDR`), service
|
||||
CIDR `10.43.0.0/16` (measured: `kubernetes` Service clusterIP is `10.43.0.1`), the PIA tunnel's
|
||||
`10.72.192.0/18` (`platform/vpn-egress/README.md`), and both LANs `192.168.1.0/24` /
|
||||
`192.168.2.0/24`. `10.90.0.0/24` is outside all five.
|
||||
- **The egress-class label matters and is set to `internet`.** Read live:
|
||||
`CiliumEgressGatewayPolicy vpn-egress-steering` selects `podSelector:
|
||||
homelab.keithjackson.dev/egress-class: vpn` with **no namespace scope**, so `vpn` here would have
|
||||
steered every remote client's internet traffic out PIA.
|
||||
- **All three nodes are untainted** (`spec.taints: None` on each), so the DaemonSet covers the
|
||||
cluster with no tolerations.
|
||||
- **`new-peer.sh` was run end to end** against a throwaway server key in the scratchpad (deleted
|
||||
immediately after; it never existed in the repo). It rendered a correct profile — `Address
|
||||
10.90.0.7/32`, `DNS 192.168.2.23`, `Endpoint wan.home.keithjackson.dev:51820`, `MTU 1420`,
|
||||
`PersistentKeepalive 25`, `AllowedIPs 0.0.0.0/0` — at mode 0600, printed the matching public-key
|
||||
stanza, and both argument-validation paths (missing name, out-of-range octet) refuse correctly.
|
||||
The private key was redacted from every line I displayed.
|
||||
- **The endpoint name is resolver-independent, which is the specific trap this avoids.**
|
||||
`wan.${BASE_DOMAIN}` is an unproxied public Cloudflare A record kept current by
|
||||
`platform/externaldns/ddns.yaml` (checked every 5 min), so it resolves from a cellular connection
|
||||
— unlike any name PiHole answers, which a remote client cannot resolve until it is already
|
||||
connected. (This is the `plex.direct` lesson applied.)
|
||||
- **Velero does cover this namespace.** `platform/velero/helmrelease.yaml`'s `daily` schedule
|
||||
excludes only `monitoring` and `unifi`, so `vpn-ingress` and the `wg-server-key` Secret are
|
||||
captured off-cluster to R2. Read from the manifest, not from a backup listing.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **THE ACTUAL FIX. Nothing in this branch has been proven to carry a single packet.** The
|
||||
DaemonSet has never been reconciled, no `wg0` has ever existed, and no handshake has ever been
|
||||
attempted. Three blockers are off-repo and manual (kernel module, server key, WAN forward), and
|
||||
this branch removes none of them — it only makes them the *only* remaining work.
|
||||
- **Outside-in reachability, at all.** This repo has **no off-LAN vantage point**, which is exactly
|
||||
the gap the plex remote-access work hit. Step 6 of the runbook (a phone on cellular) is the only
|
||||
honest test and it has not been run.
|
||||
- **The in-pod `iptables` → host-netfilter interaction, which is my biggest single doubt.** Debian 13
|
||||
uses nf_tables; the container writes MASQUERADE/FORWARD rules into the **host's** tables via the
|
||||
image's own `iptables`. If that binary is the legacy backend while the host is nft, the rules can
|
||||
apply to a table the kernel does not consult — and the symptom would be the nastiest available:
|
||||
handshakes succeed, `wg show` looks perfect, and every forwarded connection hangs. **Check this
|
||||
first if the tunnel comes up but nothing is reachable:** `iptables-save -t nat | grep MASQUERADE`
|
||||
and `nft list table ip nat` on the node, from the host, not from the pod.
|
||||
- **Whether `allowPrivilegeEscalation: false` is compatible with the image's s6 init.** It is the
|
||||
correct posture and I set it, but this repo has already been bitten by `no-new-privileges`
|
||||
misbehaving (`platform/externaldns/ddns.yaml`'s README note). If the pod fails before
|
||||
`wg-quick` runs, that flag is the first thing to try flipping.
|
||||
- **Whether `NET_ADMIN` alone is sufficient.** Reasoned from what `wg-quick` does (create the link,
|
||||
set a sysctl, write two netfilter rules), not measured. If `iptables` complains, `NET_RAW` is the
|
||||
likely addition — deliberately not granted pre-emptively.
|
||||
- **The image's custom-config path.** `lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124` (tag
|
||||
confirmed to exist and to be the current one, published 2026-10-01) is expected to bring up
|
||||
whatever is in `/config/wg_confs/` when `PEERS` is unset. That is its documented behaviour; it has
|
||||
not been run here, and the init container's assembly of that file has never executed.
|
||||
- **Whether the probe is right.** `wg show wg0` is asserted to exit non-zero when the interface is
|
||||
absent. Not executed.
|
||||
- **Cilium's reaction to a new `wg0` in the host netns.** `platform/cilium/values.yaml` does not pin
|
||||
`devices`, so Cilium auto-detects; whether it attaches to `wg0` — and whether that matters — is
|
||||
unknown. `wg` interfaces are already filtered out by `node_health_collect.py`, so something in
|
||||
the estate anticipates them, but that is not evidence about Cilium.
|
||||
- **Zero peers means the end-to-end authorisation path is untested**, including revocation. The
|
||||
first enrolment is also the first test of `peers.yaml`'s format being what `wg-quick` accepts.
|
||||
- **No peer-level observability exists.** An exporter publishing
|
||||
`wireguard_latest_handshake_seconds` is the top follow-up; until then "who is connected" is an
|
||||
on-demand `wg show` and nothing alerts on an unexpected session. Pod-down is covered only by
|
||||
kube-prometheus-stack's generic `KubePodNotReady` / `KubeDeploymentReplicasMismatch`, which I did
|
||||
not confirm fire for a DaemonSet in this namespace.
|
||||
- **The security consequence I am most keen for a human to review rather than take from me:** an
|
||||
enrolled device reaches the three k3s nodes themselves — apiserver 6443, kubelet 10250, SSH 22 —
|
||||
because "reach the entire local network" includes them. That is stated in the README and the
|
||||
runbook with the narrowing recipe, but it is a posture decision, not a verified fact, and it is
|
||||
the right thing to argue about on this PR.
|
||||
@@ -74,12 +74,23 @@ node_baseline_backports_kernel_packages:
|
||||
# kernel, and how do I undo it?" from the machine itself.
|
||||
node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel
|
||||
|
||||
# Modules written to /etc/modules-load.d/k3s-cilium-longhorn.conf (persistent) AND modprobe'd now.
|
||||
node_baseline_kernel_modules:
|
||||
- overlay
|
||||
- br_netfilter
|
||||
- iscsi_tcp
|
||||
- dm_crypt
|
||||
- nfs
|
||||
# Remote-access VPN (platform/vpn-ingress). The WireGuard DaemonSet creates its wg0 link in the
|
||||
# HOST netns with CAP_NET_ADMIN only -- it is deliberately NOT granted CAP_SYS_MODULE, so it
|
||||
# cannot autoload this itself: `ip link add wg0 type wireguard` fails outright when the module is
|
||||
# absent, and the pod CrashLoops. Loading it here is therefore a PREREQUISITE of that component,
|
||||
# not a nicety. In-tree since kernel 5.6, so Debian 13's 7.1.8 ships it and there is nothing to
|
||||
# install. Naming it here also means the kernel does the crypto: without the module a WireGuard
|
||||
# container's only fallback is slow userspace wireguard-go, which we do not use (and which would
|
||||
# need /dev/net/tun -- a device a hostPath mount cannot actually grant, as the /dev/dri attempt
|
||||
# proved: visible but EPERM from the device cgroup).
|
||||
- wireguard
|
||||
|
||||
# --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
|
||||
# The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
|
||||
|
||||
Executable
+195
@@ -0,0 +1,195 @@
|
||||
#!/usr/bin/env bash
|
||||
# =================================================================================================
|
||||
# new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
|
||||
#
|
||||
# Generates a WireGuard keypair ON THIS MACHINE, renders the device's client profile, optionally
|
||||
# prints a QR code for a phone, and prints the one `[Peer]` stanza to paste into
|
||||
# platform/vpn-ingress/peers.yaml. The private half NEVER leaves this machine: it is not sent to the
|
||||
# cluster, not committed, and not written to any log.
|
||||
#
|
||||
# WHY A SCRIPT AND NOT A RUNBOOK STEP. Three of these fields are easy to get wrong in ways that are
|
||||
# either silently insecure or silently broken, and all three are mechanical:
|
||||
# * AllowedIPs on the SERVER side must be a /32. On the server, AllowedIPs is simultaneously the
|
||||
# cryptokey routing table AND the inbound source filter, so a /24 here lets one device spoof
|
||||
# every other device's tunnel address.
|
||||
# * PersistentKeepalive must be set on the CLIENT. Without it the NAT mapping at the client's end
|
||||
# expires after ~30s of idle and nothing can reach the device until it speaks first — which
|
||||
# presents as "the VPN works for a minute and then half-works".
|
||||
# * The endpoint must be the DDNS name, never the current WAN IP (see --endpoint below).
|
||||
#
|
||||
# USAGE
|
||||
# ./infra/vpn-ingress/new-peer.sh <device-name> <last-octet> [--split] [--out DIR]
|
||||
#
|
||||
# <device-name> short, lowercase, e.g. pixel-8, thinkpad, work-laptop. Used for the filename
|
||||
# and the comment that ties the key to a human-recognisable device.
|
||||
# <last-octet> 2-254, the device's address inside WG_CLIENT_CIDR. Must be unused.
|
||||
# --split split tunnel: route ONLY the LAN and the tunnel subnet. Default is a FULL
|
||||
# tunnel (0.0.0.0/0) so the device also egresses the house WAN.
|
||||
# --out DIR where to write the profile (default: a 0700 dir under $TMPDIR).
|
||||
#
|
||||
# ENVIRONMENT
|
||||
# WG_SERVER_KEY_FILE server PRIVATE key (default ~/.config/wireguard-home/server.key); its public
|
||||
# half is derived from it. Alternatively set WG_SERVER_PUBKEY directly.
|
||||
# WG_ENDPOINT_HOST default wan.home.keithjackson.dev
|
||||
# WG_CLIENT_CIDR default 10.90.0.0/24 (must match clusters/base/cluster-vars.yaml)
|
||||
# WG_LAN_CIDR default 192.168.2.0/24 (prod LAN_CIDR)
|
||||
# WG_DNS default 192.168.2.23 (prod PIHOLE_DNS_VIP)
|
||||
#
|
||||
# REQUIRES: wireguard-tools (`wg`). `qrencode` is optional and only used for the QR code.
|
||||
# =================================================================================================
|
||||
set -euo pipefail
|
||||
|
||||
die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
|
||||
|
||||
# --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
|
||||
# These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
|
||||
# run on an operator's laptop with no checkout-shaped assumptions beyond its own location, and a
|
||||
# silent mis-parse would be worse than a visible default. If you change the knob, change it here.
|
||||
: "${WG_CLIENT_CIDR:=10.90.0.0/24}" # clusters/base/cluster-vars.yaml
|
||||
: "${WG_LAN_CIDR:=192.168.2.0/24}" # clusters/prod LAN_CIDR
|
||||
: "${WG_DNS:=192.168.2.23}" # clusters/prod PIHOLE_DNS_VIP
|
||||
: "${WG_SERVER_KEY_FILE:=${HOME}/.config/wireguard-home/server.key}"
|
||||
# wan.<base-domain> is an UNPROXIED A record that platform/externaldns/ddns.yaml keeps pointed at
|
||||
# the current WAN IP (checked every 5 min). Use it, never a literal address: this house has a
|
||||
# DYNAMIC WAN IP, and a profile pinned to today's address stops working without warning and with no
|
||||
# symptom on the server side. It is a PUBLIC Cloudflare record, so it also resolves from a cellular
|
||||
# connection — unlike any `*.home.keithjackson.dev` name PiHole answers, which a remote client
|
||||
# cannot resolve until it is already connected.
|
||||
: "${WG_ENDPOINT_HOST:=wan.home.keithjackson.dev}"
|
||||
: "${WG_LISTEN_PORT:=51820}"
|
||||
: "${WG_MTU:=1420}"
|
||||
|
||||
SPLIT=0
|
||||
OUT_DIR=""
|
||||
NAME=""
|
||||
OCTET=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--split) SPLIT=1; shift ;;
|
||||
--out) OUT_DIR="${2:-}"; [ -n "$OUT_DIR" ] || die "--out needs a directory"; shift 2 ;;
|
||||
-h|--help) sed -n '2,40p' "${BASH_SOURCE[0]}"; exit 0 ;;
|
||||
-*) die "unknown option: $1" ;;
|
||||
*) if [ -z "$NAME" ]; then NAME="$1"; elif [ -z "$OCTET" ]; then OCTET="$1"; else die "unexpected argument: $1"; fi; shift ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[ -n "$NAME" ] || die "need a device name. See --help"
|
||||
[ -n "$OCTET" ] || die "need a last octet (2-254). See --help"
|
||||
printf '%s' "$NAME" | grep -Eq '^[a-z0-9][a-z0-9-]{0,30}$' \
|
||||
|| die "device name must be lowercase letters/digits/hyphens: got '$NAME'"
|
||||
printf '%s' "$OCTET" | grep -Eq '^[0-9]+$' || die "octet must be a number: got '$OCTET'"
|
||||
[ "$OCTET" -ge 2 ] && [ "$OCTET" -le 254 ] || die "octet must be 2-254 (.1 is the server): got '$OCTET'"
|
||||
|
||||
command -v wg >/dev/null || die "wireguard-tools not installed ('wg' not on PATH). Debian/Ubuntu: apt install wireguard-tools"
|
||||
|
||||
# --- the server's public key --------------------------------------------------------------------
|
||||
if [ -z "${WG_SERVER_PUBKEY:-}" ]; then
|
||||
[ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read server key '$WG_SERVER_KEY_FILE'.
|
||||
Either point WG_SERVER_KEY_FILE at it, or set WG_SERVER_PUBKEY to the public half.
|
||||
If the server key does not exist yet, see runbooks/vpn-remote-access.md step 1."
|
||||
WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
|
||||
fi
|
||||
|
||||
CLIENT_BASE="${WG_CLIENT_CIDR%/*}"; CLIENT_BASE="${CLIENT_BASE%.*}"
|
||||
CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
|
||||
|
||||
# --- refuse a collision, because the failure is otherwise invisible -----------------------------
|
||||
# Two devices sharing an address does not error anywhere: the server routes that address to
|
||||
# whichever peer handshook most recently, so BOTH devices appear to work intermittently.
|
||||
if [ -r "$PEERS_FILE" ] && grep -q "AllowedIPs *= *${CLIENT_ADDR}/32" "$PEERS_FILE"; then
|
||||
die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
|
||||
fi
|
||||
if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
|
||||
printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
|
||||
fi
|
||||
|
||||
# --- output location: 0700, and never inside the repo -------------------------------------------
|
||||
if [ -z "$OUT_DIR" ]; then
|
||||
OUT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/wg-peer-XXXXXX")"
|
||||
else
|
||||
mkdir -p "$OUT_DIR"
|
||||
fi
|
||||
chmod 700 "$OUT_DIR"
|
||||
case "$(cd "$OUT_DIR" && pwd)" in
|
||||
"$REPO_ROOT"|"$REPO_ROOT"/*) die "refusing to write a private key inside the git checkout ($OUT_DIR)" ;;
|
||||
esac
|
||||
PROFILE="${OUT_DIR}/${NAME}.conf"
|
||||
|
||||
umask 077
|
||||
PRIVKEY="$(wg genkey)"
|
||||
PUBKEY="$(printf '%s' "$PRIVKEY" | wg pubkey)"
|
||||
|
||||
if [ "$SPLIT" -eq 1 ]; then
|
||||
ALLOWED="${WG_LAN_CIDR}, ${WG_CLIENT_CIDR}"
|
||||
TUNNEL_KIND="split tunnel (LAN only; the device keeps using its own internet)"
|
||||
else
|
||||
ALLOWED="0.0.0.0/0"
|
||||
TUNNEL_KIND="FULL tunnel (all IPv4 traffic egresses the house WAN)"
|
||||
fi
|
||||
|
||||
cat > "$PROFILE" <<EOF
|
||||
# ${NAME} — remote-access VPN profile for home.keithjackson.dev
|
||||
# Generated $(date -u +%Y-%m-%dT%H:%M:%SZ) by infra/vpn-ingress/new-peer.sh
|
||||
# ${TUNNEL_KIND}
|
||||
#
|
||||
# THIS FILE CONTAINS A PRIVATE KEY. Move it to the device, then delete it from here.
|
||||
# It is the ONLY copy: the server never sees it and it cannot be recovered or re-derived.
|
||||
[Interface]
|
||||
PrivateKey = ${PRIVKEY}
|
||||
Address = ${CLIENT_ADDR}/32
|
||||
DNS = ${WG_DNS}
|
||||
MTU = ${WG_MTU}
|
||||
|
||||
[Peer]
|
||||
PublicKey = ${WG_SERVER_PUBKEY}
|
||||
Endpoint = ${WG_ENDPOINT_HOST}:${WG_LISTEN_PORT}
|
||||
AllowedIPs = ${ALLOWED}
|
||||
# Hold the NAT mapping open from the client side. Required for the server to be able to reach the
|
||||
# device at all once it goes idle; harmless otherwise (one small packet every 25s).
|
||||
PersistentKeepalive = 25
|
||||
EOF
|
||||
chmod 600 "$PROFILE"
|
||||
|
||||
cat <<EOF
|
||||
|
||||
Device : ${NAME}
|
||||
Address : ${CLIENT_ADDR}/32
|
||||
Routing : ${TUNNEL_KIND}
|
||||
DNS : ${WG_DNS} (PiHole — gives the device LAN answers for *.home.keithjackson.dev)
|
||||
Profile : ${PROFILE} (0600; contains the private key)
|
||||
|
||||
------------------------------------------------------------------------------------------
|
||||
STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
|
||||
|
||||
# ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
|
||||
[Peer]
|
||||
PublicKey = ${PUBKEY}
|
||||
AllowedIPs = ${CLIENT_ADDR}/32
|
||||
|
||||
Keep the comment line: \`wg show\` can only print the key, so it is the only record of which
|
||||
stanza belongs to which device when one has to be revoked.
|
||||
|
||||
STEP 2 — get the profile onto the device:
|
||||
Android / iOS : scan the QR code below in the WireGuard app ("+" -> Scan from QR code)
|
||||
Windows / macOS: WireGuard app -> "Import tunnel(s) from file" -> ${NAME}.conf
|
||||
Linux : install as /etc/wireguard/${NAME}.conf, then
|
||||
sudo systemctl enable --now wg-quick@${NAME}
|
||||
|
||||
STEP 3 — delete the profile from this machine once the device has it:
|
||||
shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}
|
||||
|
||||
The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
|
||||
(~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
|
||||
------------------------------------------------------------------------------------------
|
||||
EOF
|
||||
|
||||
if command -v qrencode >/dev/null; then
|
||||
printf ' QR code (this IS the private key — your terminal scrollback now holds a credential):\n\n'
|
||||
qrencode -t ansiutf8 < "$PROFILE"
|
||||
else
|
||||
printf ' qrencode not installed, so no QR code was printed (Debian/Ubuntu: apt install qrencode).\n'
|
||||
printf ' Transfer %s to the device by another route instead.\n' "$PROFILE"
|
||||
fi
|
||||
@@ -0,0 +1,199 @@
|
||||
# platform/vpn-ingress — WireGuard remote access (north-south INGRESS)
|
||||
|
||||
Lets the operator's own devices — Android, Linux, Windows, macOS, iOS — dial in from anywhere on
|
||||
the internet and behave as though plugged into the home LAN: full reach to `192.168.2.0/24` and
|
||||
egress out the house WAN.
|
||||
|
||||
> **This is not `platform/vpn-egress`.** They are opposite directions and share nothing but the
|
||||
> word "VPN". `vpn-egress` pushes *pods out* to the internet through a PIA tunnel (ADR-0015,
|
||||
> R-NET-05/06/07). This component lets *remote clients in*. Changing one never affects the other.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Transport | WireGuard, **udp/51820**, kernel implementation |
|
||||
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
|
||||
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
|
||||
| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
|
||||
| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** |
|
||||
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
|
||||
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
|
||||
| Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
|
||||
|
||||
## Why WireGuard
|
||||
|
||||
The requirement names four client platforms. WireGuard is the only option that has a *first-party*
|
||||
client on all of them (official apps for Android, iOS, Windows and macOS; in-kernel on Linux since
|
||||
5.6, so a Linux client needs no userspace daemon at all). It is also the smallest thing to expose:
|
||||
one UDP port, no cipher negotiation, no PKI, no user database, no TLS stack.
|
||||
|
||||
OpenVPN would have needed a CA, per-device certificates and a revocation list — three pieces of
|
||||
long-lived state this repo would then have to protect and rotate. IKEv2/IPsec has native clients but
|
||||
is painful to run in a container and brittle behind double NAT. Neither buys anything here.
|
||||
|
||||
### Why this is safe to expose to the whole internet
|
||||
|
||||
The honest answer is not "because it is encrypted" — it is the handshake's failure mode:
|
||||
|
||||
**An unauthenticated packet gets no reply at all.** WireGuard's first handshake message carries a
|
||||
MAC keyed with the *responder's own public key*. If the sender does not already know it, the packet
|
||||
is dropped silently — no error, no reset, no banner, nothing a scanner can fingerprint. From the
|
||||
outside, udp/51820 is indistinguishable from a closed port. There is no login prompt to brute-force,
|
||||
no version string to match against a CVE, and no amplification to abuse.
|
||||
|
||||
That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
|
||||
would also have made roaming clients unusable — a phone's address changes with every cell tower).
|
||||
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a
|
||||
resolver answers anybody, so it must be scoped by address. WireGuard answers nobody.
|
||||
|
||||
What it does **not** protect against is a stolen device — anyone holding a device's private key
|
||||
*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).
|
||||
|
||||
## Architecture, and the one decision that drives everything
|
||||
|
||||
```
|
||||
phone on 5G SOHO router k3s-prod-N (host netns)
|
||||
┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐
|
||||
│ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │
|
||||
│ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │
|
||||
└──────────────┘ internet └───────────────┘ │ SNAT to the node's addr │
|
||||
└─────────────┬──────────────┘
|
||||
│ as `host`/`remote-node`
|
||||
┌─────────────────────────────────┼──────────────────────┐
|
||||
▼ ▼ ▼
|
||||
LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet
|
||||
(NAS, router, printers) PiHole DNS VIP .23
|
||||
```
|
||||
|
||||
### `hostNetwork: true` is required, not convenient
|
||||
|
||||
This is the only genuinely interesting decision in the component, and it was forced by a *measured*
|
||||
property of this cluster rather than chosen for convenience.
|
||||
|
||||
In Cilium, what a packet may reach is decided by the **security identity of its source**. A
|
||||
WireGuard server running as an ordinary pod must SNAT client traffic to its own pod IP (Cilium drops
|
||||
a packet whose source is not the endpoint's own address), so every remote client would have arrived
|
||||
everywhere wearing *that pod's* identity. On 2026-10-04 this cluster measured what that costs
|
||||
(`decisions/2026-10-04-plex-advertise-plexdirect.md`): a request to the shared Gateway VIP **from
|
||||
inside a pod returns `403 Access denied` from Envoy for all 16 Gateway hostnames**, because Cilium's
|
||||
Gateway preserves the original source identity and enforces the *destination's* ingress policy —
|
||||
and every app admits `reserved:ingress` plus named peers, never an arbitrary pod. LAN browsers work
|
||||
only because they arrive as `reserved:world` / `reserved:host`.
|
||||
|
||||
So the pod-networked version of this component would have produced a tunnel that came up healthy,
|
||||
reported connected peers, routed packets — and 403'd **every single cluster web UI**: plex, gitea,
|
||||
keycloak, nextcloud, grafana, headlamp, the apex. `*.home.keithjackson.dev` would not even have
|
||||
resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately omits the `cluster`
|
||||
entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
|
||||
manifests would have hinted at it.
|
||||
|
||||
In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the
|
||||
node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
|
||||
byte-for-byte what a LAN client looks like. Three further problems vanish with it:
|
||||
|
||||
* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
|
||||
`/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
|
||||
pod-netns build would have needed `--allowed-unsafe-sysctls` on all three kubelets — a
|
||||
node-restart change to the k3s service, to run a VPN.
|
||||
* **No LoadBalancer VIP**, so no `CiliumL2AnnouncementPolicy` lease lottery (the backend-blind
|
||||
announcer that cost this LAN an 8-day Minecraft outage — see
|
||||
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
|
||||
`externalTrafficPolicy` trade-off to make at all.
|
||||
* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
|
||||
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct.
|
||||
|
||||
`hostNetwork` is *permitted* here, and not by oversight:
|
||||
`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
|
||||
app-namespace=true` only, and names this exact carve-out in its own comment — "platform components
|
||||
that do … e.g. Cilium, node-exporter … live in platform namespaces that do NOT carry the
|
||||
app-namespace label". [`namespace.yaml`](./namespace.yaml) is deliberately unlabelled.
|
||||
|
||||
**The cost, stated rather than buried:** a `hostNetwork` pod has no Cilium endpoint, so **no
|
||||
NetworkPolicy constrains this workload** — not [`networkpolicy.yaml`](./networkpolicy.yaml), not
|
||||
anything. That file says so at length instead of implying a protection it cannot provide. The
|
||||
boundary around this port is WireGuard's peer authentication, full stop.
|
||||
|
||||
### Why a DaemonSet
|
||||
|
||||
The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
|
||||
forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
|
||||
A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
|
||||
keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
|
||||
re-issued client profiles. The two idle listeners cost ~15 MiB each.
|
||||
|
||||
A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
|
||||
replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
|
||||
hop into the one path whose failure mode is *"I am away from home and cannot get in"*. Fewer
|
||||
translations win here.
|
||||
|
||||
### The split that keeps authorisation reviewable
|
||||
|
||||
| half | where | why |
|
||||
|---|---|---|
|
||||
| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
|
||||
| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) |
|
||||
|
||||
The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
|
||||
so the assembled config holding the private key never touches any node's disk and dies with the pod.
|
||||
|
||||
Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
|
||||
new line granting a new device access to the whole LAN.
|
||||
|
||||
## What a client can reach — and how to narrow it
|
||||
|
||||
By design, **everything a LAN machine can reach**: all of `192.168.2.0/24`, the shared Gateway's 16
|
||||
hostnames, PiHole, the NAS, the router's admin UI, and the internet via the house WAN.
|
||||
|
||||
**That includes the three k3s nodes themselves** — `192.168.2.10/.11/.12`, and therefore the
|
||||
kube-apiserver on 6443, kubelet on 10250 and SSH on 22. An enrolled device is as privileged as a
|
||||
machine sitting in the house, which is what "reach the entire local network" asks for; it is
|
||||
recorded here because it is the largest consequence of saying yes to that, and because nobody should
|
||||
discover it later. **It is also why a lost device must be revoked promptly** (delete its stanza,
|
||||
merge; ~1 min to take effect).
|
||||
|
||||
To grant less, narrow it at the **client** profile first — `new-peer.sh --split` emits a
|
||||
split-tunnel profile routing only the LAN and the tunnel subnet, so the device keeps its own
|
||||
internet path. To enforce less *server-side*, add filtering to the `PostUp` chain in
|
||||
[`daemonset.yaml`](./daemonset.yaml) (e.g. `iptables -A FORWARD -i wg0 -d 192.168.2.10 -j DROP`
|
||||
before the accept rules) — the FORWARD chain there is the only enforcement point, since no
|
||||
NetworkPolicy applies.
|
||||
|
||||
## Operating it
|
||||
|
||||
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
|
||||
stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
|
||||
* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
|
||||
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
|
||||
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
|
||||
(`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
|
||||
* **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
|
||||
|
||||
## Prerequisites outside this directory
|
||||
|
||||
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
|
||||
(`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
|
||||
`NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
|
||||
CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
|
||||
2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start —
|
||||
the correct fail-closed state for an unenrolled VPN.
|
||||
3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
|
||||
and the half CI can never check.
|
||||
|
||||
## Not built (deliberate, with reasons)
|
||||
|
||||
* **No metrics / no peer-level alerting.** A WireGuard exporter would expose
|
||||
`wireguard_latest_handshake_seconds` per peer — genuinely useful for spotting unexpected
|
||||
activity — but it needs a second `NET_ADMIN` container in the pod that terminates the
|
||||
internet-facing tunnel, and that trade was not worth making unverified. Pod-down is already
|
||||
covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
|
||||
kube-prometheus-stack. **This is the top follow-up.**
|
||||
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
|
||||
handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
|
||||
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
|
||||
`wg-server-key` Secret.
|
||||
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
|
||||
still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point
|
||||
was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's
|
||||
`AllowedIPs`; not done because nothing here needs it.
|
||||
* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
|
||||
browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
|
||||
into the tunnel.
|
||||
@@ -0,0 +1,295 @@
|
||||
---
|
||||
# =================================================================================================
|
||||
# WireGuard remote-access listener — udp/51820 in the HOST network namespace, on every node.
|
||||
# =================================================================================================
|
||||
# WHY WireGuard AND NOT OpenVPN/IPsec. The requirement is "Android, Linux, Windows and other
|
||||
# computing clients", from anywhere on the internet, securely. WireGuard is the only option here
|
||||
# with a first-party client on all four platforms (official apps for Android/iOS/Windows/macOS, and
|
||||
# in-kernel on Linux since 5.6 — so Linux clients need no userspace daemon at all). It is also the
|
||||
# smallest attack surface of the three: one UDP port, ~4k lines of kernel code, no cipher
|
||||
# negotiation, no PKI, no user database, and — the property that actually matters for a port
|
||||
# exposed to the whole internet — it NEVER REPLIES to an unauthenticated packet. See README.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# hostNetwork: true — REQUIRED, not convenient. This is THE decision in this file.
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# A VPN client must behave like a machine plugged into the LAN. In Cilium, what a packet is ALLOWED
|
||||
# to reach is decided by the SECURITY IDENTITY of its source, and after any in-pod SNAT that
|
||||
# identity is the VPN pod's — not `world`. That distinction is not academic; it was MEASURED on
|
||||
# this cluster on 2026-10-04 (decisions/2026-10-04-plex-advertise-plexdirect.md): a curl of the
|
||||
# shared Gateway VIP from inside a pod returns `403 Access denied` from Envoy for **all 16**
|
||||
# Gateway hostnames, because Cilium's Gateway preserves the original source identity and enforces
|
||||
# the DESTINATION's ingress CiliumNetworkPolicy, and every app admits `reserved:ingress` plus named
|
||||
# peers — never an arbitrary pod. LAN browsers are allowed only because they arrive as
|
||||
# `reserved:world` / `reserved:host`.
|
||||
#
|
||||
# So a pod-networked WireGuard server would have handed remote clients a tunnel on which every
|
||||
# single cluster web UI returned 403 — plex, gitea, keycloak, nextcloud, grafana, headlamp, the
|
||||
# apex, all of them — while `wg show` reported a perfectly healthy tunnel. PiHole would have failed
|
||||
# the same way and worse: apps/wave-0/pihole-dns.yaml's ingress policy deliberately OMITS the
|
||||
# `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either.
|
||||
# The VPN would have looked up and been useless, which is the worst failure shape available.
|
||||
#
|
||||
# In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the
|
||||
# NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
|
||||
# exactly what a LAN client looks like. Three further problems disappear with it:
|
||||
# * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to
|
||||
# /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
|
||||
# is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
|
||||
# required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
|
||||
# * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
|
||||
# this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
|
||||
# * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
|
||||
# flat 1500-byte LAN path, not Cilium's 1450-byte VXLAN.
|
||||
#
|
||||
# hostNetwork is PERMITTED here, and not by accident: platform/kyverno/policies/
|
||||
# restrict-host-namespaces.yaml matches `namespaceSelector: app-namespace=true` only, and names
|
||||
# this exact carve-out in its own comment ("platform components that do -- e.g. Cilium,
|
||||
# node-exporter -- live in platform namespaces that do NOT carry the app-namespace label").
|
||||
# namespace.yaml is deliberately unlabelled. THE COST IS REAL AND IS STATED: a hostNetwork pod
|
||||
# bypasses the Cilium datapath, so NO NetworkPolicy constrains this workload (see
|
||||
# networkpolicy.yaml, which says so rather than pretending otherwise). The boundary protecting
|
||||
# this port is WireGuard's own cryptographic peer authentication, not the CNI.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# A DaemonSet, not a single-replica Deployment.
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# The WAN port-forward on the SOHO router points at ONE node address, by hand, off-repo (same shape
|
||||
# as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to
|
||||
# whichever node the scheduler picked and silently invalidate that forward on every reschedule. A
|
||||
# DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of
|
||||
# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no
|
||||
# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own
|
||||
# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each.
|
||||
#
|
||||
# All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the
|
||||
# DaemonSet to cover the cluster; add them if a node is ever tainted.
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: wireguard
|
||||
namespace: vpn-ingress
|
||||
labels:
|
||||
app.kubernetes.io/name: vpn-ingress
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: vpn-ingress
|
||||
# One node at a time, so a rollout never takes every listener down at once. The node named by the
|
||||
# router's forward does drop for a few seconds during its own replacement; clients re-handshake
|
||||
# automatically (WireGuard retries every 5s, and rekeys every 2 minutes regardless).
|
||||
updateStrategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxUnavailable: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: vpn-ingress
|
||||
# Declared for documentation and for symmetry with the app tier. It MUST be `internet`, and
|
||||
# the value is load-bearing rather than cosmetic: platform/vpn-egress/egress-steering.yaml's
|
||||
# CiliumEgressGatewayPolicy selects `egress-class: vpn` pods CLUSTER-WIDE, with no namespace
|
||||
# scope. Setting `vpn` here would steer every remote client's internet traffic out the PIA
|
||||
# tunnel instead of the house WAN — the opposite of the requirement, and it would look like
|
||||
# a working VPN with a mysterious foreign exit IP. (The selector matches on the pod label,
|
||||
# so hostNetwork does not exempt this pod from being selected.)
|
||||
homelab.keithjackson.dev/egress-class: internet
|
||||
# WireGuard authenticates its own peers by public key; there is no UI and no IdP hop.
|
||||
homelab.keithjackson.dev/auth-mode: native
|
||||
spec:
|
||||
# See the long rationale at the head of this file. ClusterFirstWithHostNet keeps in-cluster
|
||||
# DNS working for the pod itself despite hostNetwork (without it the pod inherits the NODE's
|
||||
# resolv.conf, which points at public resolvers).
|
||||
hostNetwork: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
automountServiceAccountToken: false
|
||||
# ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
|
||||
# The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
|
||||
# joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
|
||||
# key is therefore never written to any node's disk, never committed, and never logged.
|
||||
initContainers:
|
||||
- name: assemble-config
|
||||
image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124
|
||||
imagePullPolicy: IfNotPresent
|
||||
# POSIX sh. Note on $ handling: this script is rendered through Flux's envsubst (the
|
||||
# `vpn-ingress` Kustomization declares postBuild.substituteFrom), so `${WG_CLIENT_CIDR}`
|
||||
# below IS substituted on purpose — it is a declared cluster-vars knob. Every SHELL
|
||||
# variable is written bare (`$net`, `$(cmd)`), which envsubst leaves untouched. There is
|
||||
# deliberately NO braced shell-variable form anywhere in here: braces are what envsubst
|
||||
# consumes, so a shell variable written that way silently renders to the empty string.
|
||||
# (ci: flux-postbuild-substitution enforces this, and caught this very comment when it
|
||||
# first spelled the hazard out literally — the check reads comments too, correctly.)
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
set -eu
|
||||
umask 077
|
||||
mkdir -p /config/wg_confs
|
||||
|
||||
# Derive the server's own tunnel address from the single CIDR knob, so there is no
|
||||
# second knob to keep in agreement: 10.90.0.0/24 -> 10.90.0.1/24.
|
||||
net=${WG_CLIENT_CIDR}
|
||||
mask=$(echo "$net" | cut -d/ -f2)
|
||||
base=$(echo "$net" | cut -d/ -f1 | cut -d. -f1-3)
|
||||
|
||||
# The key arrives as a Secret file; strip any trailing newline the operator's shell
|
||||
# added, or wg-quick rejects the config with "Key is not the correct length or format".
|
||||
key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey)
|
||||
if [ -z "$key" ]; then
|
||||
echo "FATAL: /run/wg-server-key/privatekey is empty." >&2
|
||||
echo " See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
{
|
||||
echo '[Interface]'
|
||||
echo "Address = $base.1/$mask"
|
||||
echo 'ListenPort = 51820'
|
||||
# 1420 = 1500 (flat LAN/WAN path to the node's own NIC) - 60 bytes of WireGuard
|
||||
# encapsulation (20 IPv4 + 8 UDP + 32 WireGuard). This is also what wg-quick would
|
||||
# auto-derive, but it is pinned because the symptom of getting it wrong is the
|
||||
# nastiest kind: handshakes and SSH work while large packets (HTTPS, file copies)
|
||||
# hang. Inner packets stay under Cilium's 1450-byte VXLAN MTU too, so reaching a
|
||||
# cluster pod does not fragment either.
|
||||
echo 'MTU = 1420'
|
||||
printf 'PrivateKey = %s\n' "$key"
|
||||
# ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
|
||||
# ip_forward is already 1 from node_baseline; re-asserted so this component does not
|
||||
# silently depend on another role's sysctl file surviving.
|
||||
echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1'
|
||||
# SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
|
||||
# reply to 10.90.0.x, find no route, hand the packet to the router, and the router
|
||||
# would drop it — so every connection would hang rather than fail. Matching on
|
||||
# `! -o wg0` rather than a named `-o eth0` keeps this independent of the node's NIC
|
||||
# name. Each rule is add-if-absent so a pod killed without PostDown does not
|
||||
# accumulate duplicates in the host's tables across restarts.
|
||||
echo "PostUp = iptables -t nat -C POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || iptables -t nat -A POSTROUTING -s $net ! -o wg0 -j MASQUERADE"
|
||||
echo 'PostUp = iptables -C FORWARD -i wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -i wg0 -j ACCEPT'
|
||||
echo 'PostUp = iptables -C FORWARD -o wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -o wg0 -j ACCEPT'
|
||||
echo "PostDown = iptables -t nat -D POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || true"
|
||||
echo 'PostDown = iptables -D FORWARD -i wg0 -j ACCEPT 2>/dev/null || true'
|
||||
echo 'PostDown = iptables -D FORWARD -o wg0 -j ACCEPT 2>/dev/null || true'
|
||||
echo ''
|
||||
} > /config/wg_confs/wg0.conf
|
||||
|
||||
cat /run/wg-peers/peers.conf >> /config/wg_confs/wg0.conf
|
||||
chmod 600 /config/wg_confs/wg0.conf
|
||||
|
||||
# Report the SHAPE of the result, never its content — the file now holds the server's
|
||||
# private key, and a log line is a transcript (CLAUDE.md §6).
|
||||
echo "wg0.conf assembled: $(grep -c '^\[Peer\]' /config/wg_confs/wg0.conf || true) peer(s), server $base.1/$mask, udp/51820."
|
||||
securityContext:
|
||||
runAsUser: 0
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
memory: 64Mi
|
||||
volumeMounts:
|
||||
- { name: wg-config, mountPath: /config }
|
||||
- { name: wg-server-key, mountPath: /run/wg-server-key, readOnly: true }
|
||||
- { name: wg-peers, mountPath: /run/wg-peers, readOnly: true }
|
||||
containers:
|
||||
- name: wireguard
|
||||
# linuxserver/wireguard — provides wg, wg-quick and iptables on a maintained, multi-arch
|
||||
# base. Pinned to the full upstream+build tag (1.0.20260223, published 2026-10-01) rather
|
||||
# than `latest`, per the crafty/oauth2-proxy convention; Renovate maintains it.
|
||||
#
|
||||
# NO `PEERS` ENV IS SET, DELIBERATELY. With PEERS the image runs in "server mode" and
|
||||
# GENERATES keypairs at runtime — which would put the authoritative peer list on a
|
||||
# container filesystem instead of in git, and (worse) print client private keys and QR
|
||||
# codes into the POD LOG, where they would be readable by anyone with log access and
|
||||
# shipped to Loki. Supplying our own /config/wg_confs/wg0.conf selects the image's
|
||||
# bring-up-what-you-are-given path instead.
|
||||
image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124
|
||||
imagePullPolicy: IfNotPresent
|
||||
env:
|
||||
- name: TZ
|
||||
value: "America/Chicago"
|
||||
ports:
|
||||
# hostNetwork, so containerPort IS the node port. Declared for documentation and so
|
||||
# `kubectl get pod -o wide` shows what this listener owns.
|
||||
- { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
|
||||
securityContext:
|
||||
# NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and
|
||||
# write the two netfilter rules above. Everything else is dropped, and the container is
|
||||
# NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
|
||||
# SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
|
||||
# (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
|
||||
# never needs to load a kernel module. If the module is missing the pod CrashLoops with
|
||||
# a clear wg-quick error rather than quietly falling back to slow userspace WireGuard.
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
add:
|
||||
- NET_ADMIN
|
||||
# readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
|
||||
# /var during startup. Same deferral as crafty/headlamp.
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
# Kernel WireGuard does its crypto in the KERNEL, not in this container, so throughput
|
||||
# does not consume the container's budget. The limit covers s6 plus the tmpfs holding
|
||||
# wg0.conf (a memory-medium emptyDir counts against this).
|
||||
memory: 128Mi
|
||||
# `wg show wg0` exits non-zero when the interface is absent, which is the one failure that
|
||||
# matters and the one that a port probe cannot see: a crashed or never-created wg0 leaves
|
||||
# udp/51820 silent, and WireGuard is silent to unauthenticated traffic ANYWAY, so "nothing
|
||||
# answers" is indistinguishable from "working" from the outside. This probe is the only
|
||||
# thing that tells the difference.
|
||||
startupProbe:
|
||||
exec:
|
||||
command: ["sh", "-c", "wg show wg0 >/dev/null"]
|
||||
periodSeconds: 5
|
||||
failureThreshold: 24
|
||||
livenessProbe:
|
||||
exec:
|
||||
command: ["sh", "-c", "wg show wg0 >/dev/null"]
|
||||
periodSeconds: 30
|
||||
failureThreshold: 3
|
||||
volumeMounts:
|
||||
- { name: wg-config, mountPath: /config }
|
||||
# Read-only so the image can SEE whether the kernel module is present and fail with a
|
||||
# real message. It cannot load one (no SYS_MODULE) — that is the node's job.
|
||||
- { name: lib-modules, mountPath: /lib/modules, readOnly: true }
|
||||
volumes:
|
||||
# tmpfs. The assembled wg0.conf contains the server's private key, so it must never land on
|
||||
# a node's disk — `medium: Memory` guarantees that and guarantees it is gone when the pod is.
|
||||
- name: wg-config
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
# 8Mi, not 1Mi: /config is this image's whole config ROOT, not just our one file — its
|
||||
# s6 init also writes templates and state there, and a tmpfs that small would ENOSPC
|
||||
# during startup rather than at a point anyone would connect to the volume size. Still
|
||||
# counted against the container's 128Mi limit, and still far more than wg0.conf needs.
|
||||
sizeLimit: 8Mi
|
||||
# The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by
|
||||
# name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this
|
||||
# tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key
|
||||
# that passed through a session transcript would have to be treated as compromised.
|
||||
# DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace
|
||||
# and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy
|
||||
# in the operator's password manager as well — restoring it is what makes every already
|
||||
# enrolled client work again without re-enrolling.
|
||||
- name: wg-server-key
|
||||
secret:
|
||||
secretName: wg-server-key
|
||||
defaultMode: 0400
|
||||
items:
|
||||
- { key: privatekey, path: privatekey }
|
||||
- name: wg-peers
|
||||
configMap:
|
||||
name: wg-peers
|
||||
- name: lib-modules
|
||||
hostPath:
|
||||
path: /lib/modules
|
||||
type: Directory
|
||||
@@ -0,0 +1,30 @@
|
||||
---
|
||||
# WireGuard remote-access VPN (north-south INGRESS). Reconciled by its OWN non-gating post-platform
|
||||
# Flux Kustomization (clusters/base/vpn-ingress.yaml, dependsOn platform-config, wait:false) — the
|
||||
# same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be
|
||||
# legitimately unhealthy must never sit in the gating `platform` tier, where one component's
|
||||
# unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating
|
||||
# stall"). That rationale is unusually concrete here: until the operator creates the `wg-server-key`
|
||||
# Secret (runbooks/vpn-remote-access.md step 1) this DaemonSet CANNOT start, so putting it in the
|
||||
# gating tier would have taken the whole platform down on merge.
|
||||
#
|
||||
# NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- namespace.yaml
|
||||
# The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
|
||||
- peers.yaml
|
||||
# The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
|
||||
- daemonset.yaml
|
||||
# Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
|
||||
# says so at length rather than implying a protection it cannot provide.
|
||||
- networkpolicy.yaml
|
||||
#
|
||||
# DELIBERATELY ABSENT: a Secret. The server's private key (`wg-server-key`) is created out of band
|
||||
# by the operator and referenced BY NAME from daemonset.yaml. There is no `*.sops.yaml` here, so
|
||||
# this component needs NO SOPS decryption stanza in clusters/dev or clusters/prod — unlike
|
||||
# crafty/headlamp/velero. If that key is ever migrated into SOPS, BOTH halves must land together:
|
||||
# the file here, and a `decryption:` patch naming the `vpn-ingress` Kustomization in each overlay.
|
||||
# Without the patch, kustomize hands Flux the raw ciphertext and the pod mounts a key full of
|
||||
# ENC[...] — which applies cleanly and fails only at handshake time.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
# WireGuard REMOTE-ACCESS VPN (north-south INGRESS) — the counterpart to platform/vpn-egress.
|
||||
#
|
||||
# Do not confuse the two. They are opposite directions and share nothing but the word "VPN":
|
||||
# * platform/vpn-egress — pods OUT to the internet through a PIA tunnel (ADR-0015, R-NET-05/06/07).
|
||||
# * platform/vpn-ingress — remote CLIENTS IN from the internet to this LAN. THIS directory.
|
||||
#
|
||||
# NOTE: intentionally NOT homelab.keithjackson.dev/app-namespace=true. This is PLATFORM network
|
||||
# infrastructure (like cilium / vpn-egress), not an application, so Kyverno does NOT generate the app
|
||||
# default-deny baseline here and the app-scoped auth-mode / egress-class / host-namespace policies do
|
||||
# not apply — every Kyverno policy in platform/kyverno/policies matches namespaceSelector
|
||||
# app-namespace=true. THAT EXEMPTION IS LOAD-BEARING HERE, not incidental: this workload runs with
|
||||
# `hostNetwork: true`, which `restrict-host-namespaces` forbids in an app namespace. It is permitted
|
||||
# here by that policy's OWN stated carve-out — "platform components that do [need host namespaces] --
|
||||
# e.g. Cilium, node-exporter -- live in platform namespaces that do NOT carry the app-namespace
|
||||
# label, so they are out of scope". daemonset.yaml documents why hostNetwork is REQUIRED rather than
|
||||
# convenient; it is the single most consequential decision in this component.
|
||||
#
|
||||
# NO `gateway-access` label. This component serves no HTTP and attaches no HTTPRoute to the shared
|
||||
# Gateway — it is a raw UDP listener. There is deliberately nothing here for the L7 Gateway to route.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: vpn-ingress
|
||||
labels:
|
||||
app.kubernetes.io/name: vpn-ingress
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
# =================================================================================================
|
||||
# READ THIS BEFORE TRUSTING THIS FILE: it does NOT constrain the WireGuard listener.
|
||||
# =================================================================================================
|
||||
# Every other platform namespace in this repo (crafty, headlamp, homepage, runbooks) ships an
|
||||
# explicit default-deny baseline as defense in depth, because Kyverno's generator only covers
|
||||
# `app-namespace=true` namespaces. This file keeps that convention — but it would be dishonest to
|
||||
# leave it at that, because the convention does not actually bite here:
|
||||
#
|
||||
# THE `wireguard` DaemonSet RUNS WITH hostNetwork: true, SO IT HAS NO CILIUM ENDPOINT, AND NO
|
||||
# NetworkPolicy OR CiliumNetworkPolicy APPLIES TO IT — NEITHER THIS ONE NOR ANY OTHER.
|
||||
#
|
||||
# That is the documented reason restrict-host-namespaces forbids hostNetwork in app namespaces at
|
||||
# all ("a pod on hostNetwork bypasses the CNI (Cilium) datapath and therefore ESCAPES the
|
||||
# default-deny NetworkPolicy fabric -- it is the most direct way to defeat R-NET-01"). Here that
|
||||
# escape is the deliberate, required design (daemonset.yaml explains why: without it, remote
|
||||
# clients carry a pod identity and are 403'd by every Gateway hostname). A CiliumNetworkPolicy
|
||||
# admitting `world` to udp/51820 would therefore be theatre: it would read as the control that
|
||||
# protects this port, and it would be enforcing nothing. So there isn't one.
|
||||
#
|
||||
# WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
|
||||
# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
|
||||
# NO REPLY — see README "Why this is safe to expose". This is the real control.
|
||||
# 2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
|
||||
# 3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement
|
||||
# ("clients should reach the entire local network"). README "What a client can reach" states
|
||||
# the consequence in full, including reach to the nodes' own control plane, and gives the
|
||||
# narrowing recipe for an operator who wants less than that.
|
||||
#
|
||||
# WHAT THIS FILE IS STILL FOR: it is the namespace's standing posture for any FUTURE pod-networked
|
||||
# workload added here (a metrics exporter is the likely one — README "Not built"). Without it, such
|
||||
# a pod would be born unrestricted, because Kyverno will not generate a baseline in a namespace
|
||||
# that is not labelled `app-namespace=true`, and this one deliberately is not.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny
|
||||
namespace: vpn-ingress
|
||||
labels:
|
||||
homelab.keithjackson.dev/baseline: default-deny
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress: []
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
# ============================================================================================
|
||||
# THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
|
||||
# ============================================================================================
|
||||
# WireGuard has no usernames, no passwords and no PKI. Authorisation is exactly one thing: a
|
||||
# device's PUBLIC key appears in this file, or it cannot complete a handshake. An unlisted key
|
||||
# gets no reply at all — not a rejection, silence (see README "Why this is safe to expose").
|
||||
# Revoking a device is therefore deleting its stanza, and nothing else.
|
||||
#
|
||||
# WHY PLAINTEXT, AND WHY THIS IS NOT A SECRET LEAK. A WireGuard peer's PUBLIC key is public by
|
||||
# construction — it is the half you are supposed to hand to the server, and holding it grants
|
||||
# nothing. The only secret in the whole scheme is the SERVER's private key, which is NOT here:
|
||||
# it lives in the `wg-server-key` Secret and is assembled into the running config at pod start
|
||||
# (daemonset.yaml). Keeping the register in cleartext buys the property that matters most for an
|
||||
# access-control list: **"who can get into my house" is reviewable in a pull-request diff**, and
|
||||
# its history is `git log` on this file. A SOPS-encrypted register would hide exactly the change
|
||||
# a reviewer most needs to see — a new line granting a new device entry to the whole LAN.
|
||||
#
|
||||
# This is the same division of labour the repo already uses for the opposite case: a Secret's KEY
|
||||
# NAMES are cleartext while values are encrypted (.sops.yaml `encrypted_regex`), so diffs stay
|
||||
# readable. Here the whole file is the key-name half.
|
||||
#
|
||||
# ADDING A DEVICE — never hand-roll this, and never generate a key in a chat transcript.
|
||||
# ./infra/vpn-ingress/new-peer.sh <device-name> <last-octet>
|
||||
# It generates the keypair ON THE OPERATOR'S MACHINE, writes/renders the client profile (and a
|
||||
# QR code for phones), prints ONLY the public half for pasting below, and never transmits or
|
||||
# logs the private half. Then: add the stanza, PR, merge; Flux rolls the DaemonSet.
|
||||
#
|
||||
# ADDRESS ALLOCATION. ${WG_CLIENT_CIDR} is the tunnel subnet; .1 is the server (every node).
|
||||
# Give each device ONE address as a /32 in AllowedIPs. A /32 is not a formality — on the SERVER
|
||||
# side AllowedIPs is the cryptokey ROUTING table AND the inbound source filter: a peer may only
|
||||
# send from, and receive traffic for, the addresses listed in its own stanza. A sloppy /24 here
|
||||
# would let any one device impersonate every other device's tunnel address.
|
||||
#
|
||||
# .1 the server (this DaemonSet, on every node)
|
||||
# .2 - .254 devices, one each, allocated below. Keep this list dense and in order so the
|
||||
# next free octet is obvious; never re-use an octet after revoking a device
|
||||
# (an old client profile would silently start working again as someone else).
|
||||
#
|
||||
# PRESHARED KEYS are deliberately NOT used. They add a symmetric post-quantum hedge on top of
|
||||
# the Noise handshake and would be a genuine secret — which would force this register into SOPS
|
||||
# and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
|
||||
# belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: wg-peers
|
||||
namespace: vpn-ingress
|
||||
labels:
|
||||
app.kubernetes.io/name: vpn-ingress
|
||||
data:
|
||||
# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
|
||||
#
|
||||
# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
|
||||
# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
|
||||
# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
|
||||
# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
|
||||
# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
|
||||
# (`wg show` reports an interface with no peers).
|
||||
#
|
||||
# COPY THIS TEMPLATE, uncomment, one block per device:
|
||||
#
|
||||
# # pixel-8 — Keith's phone, enrolled 2026-10-05
|
||||
# [Peer]
|
||||
# PublicKey = <the public key new-peer.sh printed>
|
||||
# AllowedIPs = 10.90.0.2/32
|
||||
#
|
||||
# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
|
||||
# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
|
||||
# it an incident responder cannot tell which stanza to delete.
|
||||
peers.conf: |
|
||||
# No peers enrolled yet. See the template in this file's comments and
|
||||
# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
|
||||
@@ -120,6 +120,9 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
- **`vpn-egress-stale`** — the tunnel is UP but `egress-class=vpn` pods have no egress: Cilium's cached
|
||||
ifindex outlived a reconnect. Also covers the watchdog for that rule going inert.
|
||||
- **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.
|
||||
- **`vpn-remote-access`** — the INBOUND remote-access VPN (WireGuard, udp/51820): standing it up,
|
||||
enrolling and revoking devices, and the off-LAN checks that are the only honest proof it works.
|
||||
The opposite direction to the two pages above — do not confuse them.
|
||||
|
||||
### Backups and DR
|
||||
- **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here.
|
||||
|
||||
@@ -113,3 +113,8 @@ configMapGenerator:
|
||||
- update-rollback=update-rollback.md
|
||||
- vpn-down=vpn-down.md
|
||||
- vpn-egress-stale=vpn-egress-stale.md
|
||||
# Operator procedure, not an alert page. Served because the component it completes is INERT
|
||||
# until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
|
||||
# and because the one check that actually proves it cannot be run from inside the house.
|
||||
# Peer-level alerting does not exist yet (it needs an exporter); this page is the substitute.
|
||||
- vpn-remote-access=vpn-remote-access.md
|
||||
@@ -0,0 +1,203 @@
|
||||
# vpn-remote-access — standing up the WireGuard remote-access VPN, and proving it
|
||||
|
||||
**Status: the in-cluster half is shipped; the VPN is INERT until three manual steps are done.**
|
||||
All three are off-repo by necessity — a kernel module, a private key, and a router port-forward.
|
||||
**Alert:** pod-down is covered generically (`KubePodNotReady` / `KubeDeploymentReplicasMismatch`).
|
||||
There is **no peer-level alerting** — see "Why there is no connection alert". This is an **operator
|
||||
procedure**, not an incident page.
|
||||
**Files:** `platform/vpn-ingress/` (namespace, peer register, DaemonSet, baseline policy),
|
||||
`clusters/base/vpn-ingress.yaml` (its Flux tier), `clusters/base/cluster-vars.yaml`
|
||||
(`WG_CLIENT_CIDR`), `infra/ansible/roles/node_baseline/defaults/main.yml` (the `wireguard` module),
|
||||
`infra/vpn-ingress/new-peer.sh` (device enrolment).
|
||||
**Design rationale:** `platform/vpn-ingress/README.md` — read "Why `hostNetwork` is required"
|
||||
before changing anything about how this is wired.
|
||||
**Not to be confused with** `vpn-down` / `vpn-egress-stale`, which are about `platform/vpn-egress`
|
||||
(pods *out* through PIA). This page is remote clients *in*.
|
||||
|
||||
## What this gives you
|
||||
|
||||
A device — Android, Linux, Windows, macOS, iOS — dials in over **udp/51820** from anywhere and
|
||||
behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's web UIs, PiHole
|
||||
DNS, and internet egress out the house WAN.
|
||||
|
||||
Authorisation is one thing only: the device's **public key is in
|
||||
`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
|
||||
|
||||
> **An enrolled device is as privileged as a machine sitting in the house** — including the three
|
||||
> k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
|
||||
> the entire local network" means. Revoke a lost device promptly: delete its stanza, PR, merge.
|
||||
|
||||
## Step 1 — load the `wireguard` module on every node (once)
|
||||
|
||||
The DaemonSet creates its `wg0` link in the host netns with `CAP_NET_ADMIN` only; it is
|
||||
deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module. Without it
|
||||
`wg-quick` fails outright and the pod CrashLoops.
|
||||
|
||||
```bash
|
||||
# operator action (Ansible is not an agent action)
|
||||
cd infra/ansible && ./site.sh --tags system --limit k3s_prod
|
||||
```
|
||||
|
||||
Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:
|
||||
|
||||
```bash
|
||||
# expected: a `wireguard` line, and the modules-load.d file naming it for the next boot
|
||||
lsmod | grep '^wireguard'
|
||||
grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
|
||||
```
|
||||
|
||||
## Step 2 — create the server key (once)
|
||||
|
||||
This is the **only** secret in the component, and the one thing that must never be generated in an
|
||||
agent session or pasted into a transcript. Generate it on your own machine:
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
|
||||
umask 077
|
||||
wg genkey > ~/.config/wireguard-home/server.key # keep this; see "Back it up" below
|
||||
```
|
||||
|
||||
Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that
|
||||
lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6):
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \
|
||||
--from-file=privatekey=/work/.cache/server.key'
|
||||
```
|
||||
|
||||
…where `/work/.cache/server.key` is a path you copied the key to **inside the repo's gitignored
|
||||
cache** (or mount it however you prefer — the requirement is only that the bytes never pass through
|
||||
a log). Delete that copy afterwards: `shred -u .cache/server.key`.
|
||||
|
||||
Confirm it exists **by key name only**:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1'
|
||||
# expected: {"privatekey -- a NAME. If you see base64 payload, you printed the key. Rotate it.
|
||||
```
|
||||
|
||||
**Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that
|
||||
passed through a session transcript would have to be treated as compromised. The trade-off is that
|
||||
this Secret is **not reconstructible from git** — so:
|
||||
|
||||
**Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only
|
||||
`monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured
|
||||
off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work
|
||||
again after a cluster rebuild — **losing it means re-enrolling every device.**
|
||||
|
||||
## Step 3 — forward udp/51820 on the router (once)
|
||||
|
||||
On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
|
||||
`192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.
|
||||
|
||||
Pick any node — all three run an identical listener. Note which one you chose here:
|
||||
|
||||
```
|
||||
# WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set <date>
|
||||
```
|
||||
|
||||
**Node loss = failover is this one line.** Re-point the forward at another node address; every
|
||||
device keeps working with the same profile, because all three nodes answer the same keys.
|
||||
|
||||
Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
|
||||
already silent to unauthenticated probes (README, "Why this is safe to expose").
|
||||
|
||||
## Step 4 — confirm the listener is healthy
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress get ds wireguard -o wide'
|
||||
# expected: DESIRED 3 CURRENT 3 READY 3
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
|
||||
# expected: `interface: wg0`, `listening port: 51820`, and (before step 5) NO peers.
|
||||
```
|
||||
|
||||
A zero-peer interface that is Ready is the **correct** state before any device is enrolled: it
|
||||
listens, and refuses every handshake because no key is trusted.
|
||||
|
||||
The readiness/liveness probe is `wg show wg0`, and that is load-bearing rather than cosmetic: a
|
||||
port probe cannot distinguish "wg0 never came up" from "working", because WireGuard is silent to
|
||||
unauthenticated traffic either way. If the pod is Ready, the interface exists.
|
||||
|
||||
## Step 5 — enrol a device
|
||||
|
||||
```bash
|
||||
./infra/vpn-ingress/new-peer.sh pixel-8 2 # full tunnel (LAN + WAN egress)
|
||||
./infra/vpn-ingress/new-peer.sh thinkpad 3 --split # split tunnel (LAN only)
|
||||
```
|
||||
|
||||
It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout,
|
||||
prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then:
|
||||
|
||||
1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
|
||||
`# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
|
||||
comment is the only record of which stanza belongs to which device.
|
||||
2. PR → merge. Flux rolls the DaemonSet (~1 min).
|
||||
3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
|
||||
Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
|
||||
4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
|
||||
device's private key — the server never sees it and it cannot be re-derived.
|
||||
|
||||
Until the PR is merged the device handshakes and gets silence, exactly like an unknown key.
|
||||
|
||||
## Step 6 — verify it end to end (from OFF the LAN)
|
||||
|
||||
**This is the only honest test, and it cannot be run from inside the house.** Put the phone on
|
||||
cellular, with Wi-Fi off, and check all four — each proves a different hop:
|
||||
|
||||
| # | Check | Proves |
|
||||
|---|---|---|
|
||||
| 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza |
|
||||
| 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path |
|
||||
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd |
|
||||
| 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier |
|
||||
|
||||
Then confirm server-side:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
|
||||
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer.
|
||||
```
|
||||
|
||||
**If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not
|
||||
WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up.
|
||||
|
||||
**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60
|
||||
bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
|
||||
|
||||
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside —
|
||||
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
|
||||
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
|
||||
|
||||
## Revoking a device
|
||||
|
||||
Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
|
||||
next reconcile. **Never re-use that address octet**: an old profile would silently start working
|
||||
again as a different device.
|
||||
|
||||
## Why there is no connection alert
|
||||
|
||||
A WireGuard exporter would publish `wireguard_latest_handshake_seconds` per peer and make
|
||||
"something connected at 03:00" alertable. It needs a second `NET_ADMIN` container inside the pod
|
||||
that terminates the internet-facing tunnel, and that was not worth shipping unverified. **It is the
|
||||
top follow-up for this component.** Until then:
|
||||
|
||||
* pod-down is covered generically by kube-prometheus-stack's `KubePodNotReady` /
|
||||
`KubeDeploymentReplicasMismatch`;
|
||||
* "who is connected" is an on-demand question — `wg show`, above;
|
||||
* **never** `wg showconf` — it prints the server's private key.
|
||||
|
||||
## Known limitations
|
||||
|
||||
* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside
|
||||
the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point.
|
||||
* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's
|
||||
address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
|
||||
preserve them — at the cost of another hand-maintained off-repo route.
|
||||
* **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients
|
||||
re-handshake automatically (retry every 5s, rekey every 2 min).
|
||||
Reference in new issue
Block a user