jakman85andClaude Opus 5
f5353cbd30
A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
holds itself: the announcer does not answer its own ARP request, so on the
lease holder the neighbour entry settles at FAILED and packets to that VIP die
before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes
x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced.
platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and
MASQUERADEs remote clients to the node's own address, so a VPN client takes
the node's LAN path and inherits its ARP failures. The router's udp/51820
forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16
Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering
(its VIP has two Services and k3s-prod-2 held the other lease). The tunnel
handshook, names resolved, every app hung, and nothing reported it.
on k3s-prod-1: ip neigh show 192.168.2.20 -> FAILED (its own VIP)
on prod-2/3: ip neigh show 192.168.2.20 -> lladdr ... (prod-1), 443 OK
Mark the node and exclude it from announcing:
* node_baseline labels the forwarded node (node_baseline_vpn_ingress_node,
k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes
the label from any node that no longer matches so moving it MOVES the label.
* l2-announcement-policy.yaml excludes that label, so the node ARP-resolves
every VIP. Fixes all VIPs for VPN clients, not just the one that collided.
Cost: announcement candidacy drops from 3 nodes to 2.
* VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels
(label added to the kube-state-metrics allowlist), with
VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches,
so a missing label would silently re-arm the whole failure AND leave the
critical alert inert. Four promtool cases, including that inertness.
A node LABEL rather than a cluster-vars knob because the alert must read it
too, and extract_rules.py rightly refuses a ${...} placeholder in an alert
expr -- the rule unit-tested would not be the rule deployed. One source of
truth, and it touches no sensitive path.
Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and
"failover is this one line". Both false; the first caused this.
UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the
handoff entry's third section.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
275 lines
19 KiB
Markdown
275 lines
19 KiB
Markdown
# platform/vpn-ingress — WireGuard remote access (north-south INGRESS)
|
||
|
||
Lets the operator's own devices — Android, Linux, Windows, macOS, iOS — dial in from anywhere on
|
||
the internet and behave as though plugged into the home LAN: full reach to `192.168.2.0/24` and
|
||
egress out the house WAN.
|
||
|
||
> **This is not `platform/vpn-egress`.** They are opposite directions and share nothing but the
|
||
> word "VPN". `vpn-egress` pushes *pods out* to the internet through a PIA tunnel (ADR-0015,
|
||
> R-NET-05/06/07). This component lets *remote clients in*. Changing one never affects the other.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Transport | WireGuard, **udp/51820**, kernel implementation |
|
||
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
|
||
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
|
||
| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
|
||
| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
|
||
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
|
||
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
|
||
| Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
|
||
|
||
## Why WireGuard
|
||
|
||
The requirement names four client platforms. WireGuard is the only option that has a *first-party*
|
||
client on all of them (official apps for Android, iOS, Windows and macOS; in-kernel on Linux since
|
||
5.6, so a Linux client needs no userspace daemon at all). It is also the smallest thing to expose:
|
||
one UDP port, no cipher negotiation, no PKI, no user database, no TLS stack.
|
||
|
||
OpenVPN would have needed a CA, per-device certificates and a revocation list — three pieces of
|
||
long-lived state this repo would then have to protect and rotate. IKEv2/IPsec has native clients but
|
||
is painful to run in a container and brittle behind double NAT. Neither buys anything here.
|
||
|
||
### Why this is safe to expose to the whole internet
|
||
|
||
The honest answer is not "because it is encrypted" — it is the handshake's failure mode:
|
||
|
||
**An unauthenticated packet gets no reply at all.** WireGuard's first handshake message carries a
|
||
MAC keyed with the *responder's own public key*. If the sender does not already know it, the packet
|
||
is dropped silently — no error, no reset, no banner, nothing a scanner can fingerprint. From the
|
||
outside, udp/51820 is indistinguishable from a closed port. There is no login prompt to brute-force,
|
||
no version string to match against a CVE, and no amplification to abuse.
|
||
|
||
That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
|
||
would also have made roaming clients unusable — a phone's address changes with every cell tower).
|
||
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a
|
||
resolver answers anybody, so it must be scoped by address. WireGuard answers nobody.
|
||
|
||
What it does **not** protect against is a stolen device — anyone holding a device's private key
|
||
*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).
|
||
|
||
## Architecture, and the one decision that drives everything
|
||
|
||
```
|
||
phone on 5G SOHO router k3s-prod-N (host netns)
|
||
┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐
|
||
│ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │
|
||
│ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │
|
||
└──────────────┘ internet └───────────────┘ │ SNAT to the node's addr │
|
||
└─────────────┬──────────────┘
|
||
│ as `host`/`remote-node`
|
||
┌─────────────────────────────────┼──────────────────────┐
|
||
▼ ▼ ▼
|
||
LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet
|
||
(NAS, router, printers) PiHole DNS VIP .23
|
||
```
|
||
|
||
### `hostNetwork: true` is required, not convenient
|
||
|
||
This is the only genuinely interesting decision in the component, and it was forced by a *measured*
|
||
property of this cluster rather than chosen for convenience.
|
||
|
||
In Cilium, what a packet may reach is decided by the **security identity of its source**. A
|
||
WireGuard server running as an ordinary pod must SNAT client traffic to its own pod IP (Cilium drops
|
||
a packet whose source is not the endpoint's own address), so every remote client would have arrived
|
||
everywhere wearing *that pod's* identity. On 2026-10-04 this cluster measured what that costs
|
||
(`decisions/2026-10-04-plex-advertise-plexdirect.md`): a request to the shared Gateway VIP **from
|
||
inside a pod returns `403 Access denied` from Envoy for all 16 Gateway hostnames**, because Cilium's
|
||
Gateway preserves the original source identity and enforces the *destination's* ingress policy —
|
||
and every app admits `reserved:ingress` plus named peers, never an arbitrary pod. LAN browsers work
|
||
only because they arrive as `reserved:world` / `reserved:host`.
|
||
|
||
So the pod-networked version of this component would have produced a tunnel that came up healthy,
|
||
reported connected peers, routed packets — and 403'd **every single cluster web UI**: plex, gitea,
|
||
keycloak, nextcloud, grafana, headlamp, the apex. `*.home.keithjackson.dev` would not even have
|
||
resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately omits the `cluster`
|
||
entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
|
||
manifests would have hinted at it.
|
||
|
||
In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the
|
||
node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
|
||
byte-for-byte what a LAN client looks like. Three further problems vanish with it:
|
||
|
||
* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
|
||
`/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
|
||
pod-netns build would have needed `--allowed-unsafe-sysctls` on all three kubelets — a
|
||
node-restart change to the k3s service, to run a VPN.
|
||
* **No LoadBalancer VIP**, so no `CiliumL2AnnouncementPolicy` lease lottery (the backend-blind
|
||
announcer that cost this LAN an 8-day Minecraft outage — see
|
||
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
|
||
`externalTrafficPolicy` trade-off to make at all.
|
||
* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
|
||
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct.
|
||
|
||
`hostNetwork` is *permitted* here, and not by oversight:
|
||
`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
|
||
app-namespace=true` only, and names this exact carve-out in its own comment — "platform components
|
||
that do … e.g. Cilium, node-exporter … live in platform namespaces that do NOT carry the
|
||
app-namespace label". [`namespace.yaml`](./namespace.yaml) is deliberately unlabelled.
|
||
|
||
**The cost, stated rather than buried:** a `hostNetwork` pod has no Cilium endpoint, so **no
|
||
NetworkPolicy constrains this workload** — not [`networkpolicy.yaml`](./networkpolicy.yaml), not
|
||
anything. That file says so at length instead of implying a protection it cannot provide. The
|
||
boundary around this port is WireGuard's peer authentication, full stop.
|
||
|
||
### Why a DaemonSet
|
||
|
||
The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
|
||
forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
|
||
A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
|
||
keys — so losing the forwarded node never costs a re-issued client profile. The two idle listeners
|
||
cost ~15 MiB each.
|
||
|
||
> **This section used to say "losing the forwarded node is a one-line router change, with no cluster
|
||
> work". It was wrong, and it cost two days of silently broken remote access on 2026-10-08.** The
|
||
> nodes are warm, but they are **not interchangeable**: the node the forward points at must be the
|
||
> one labelled `homelab.keithjackson.dev/vpn-ingress=true`, because that is the node
|
||
> `platform-config/cilium/l2-announcement-policy.yaml` excludes from announcing LoadBalancer VIPs.
|
||
> **A node cannot reach a VIP whose Cilium L2 announcement lease it holds itself** (see "Measured on
|
||
> prod, 2026-10-08" below), and since this pod MASQUERADEs clients to the node's own address, a VPN
|
||
> client inherits that. Failover is therefore **two** changes: move
|
||
> `node_baseline_vpn_ingress_node` and re-run that role, then re-point the router —
|
||
> `runbooks/vpn-remote-access.md` step 3.
|
||
|
||
A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
|
||
replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
|
||
hop into the one path whose failure mode is *"I am away from home and cannot get in"*. Fewer
|
||
translations win here.
|
||
|
||
### The split that keeps authorisation reviewable
|
||
|
||
| half | where | why |
|
||
|---|---|---|
|
||
| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
|
||
| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
|
||
|
||
The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
|
||
so the assembled config holding the private key never touches any node's disk and dies with the pod.
|
||
|
||
Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with
|
||
everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in,
|
||
corrected after review — would have put the one credential every enrolled device depends on *outside*
|
||
the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this
|
||
key's public half.
|
||
|
||
Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
|
||
new line granting a new device access to the whole LAN.
|
||
|
||
## What a client can reach — and how to narrow it
|
||
|
||
By design, **everything a LAN machine can reach**: all of `192.168.2.0/24`, the shared Gateway's 16
|
||
hostnames, PiHole, the NAS, the router's admin UI, and the internet via the house WAN.
|
||
|
||
**That includes the three k3s nodes themselves** — `192.168.2.10/.11/.12`, and therefore the
|
||
kube-apiserver on 6443, kubelet on 10250 and SSH on 22. An enrolled device is as privileged as a
|
||
machine sitting in the house, which is what "reach the entire local network" asks for; it is
|
||
recorded here because it is the largest consequence of saying yes to that, and because nobody should
|
||
discover it later. **It is also why a lost device must be revoked promptly** (delete its stanza,
|
||
merge; ~1 min to take effect).
|
||
|
||
To grant less, narrow it at the **client** profile first — `new-peer.sh --split` emits a
|
||
split-tunnel profile routing only the LAN and the tunnel subnet, so the device keeps its own
|
||
internet path. To enforce less *server-side*, add filtering to the `PostUp` chain in
|
||
[`daemonset.yaml`](./daemonset.yaml) (e.g. `iptables -A FORWARD -i wg0 -d 192.168.2.10 -j DROP`
|
||
before the accept rules) — the FORWARD chain there is the only enforcement point, since no
|
||
NetworkPolicy applies.
|
||
|
||
## Operating it
|
||
|
||
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
|
||
stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
|
||
* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
|
||
stops working as soon as the pods roll (the generator's content hash guarantees they do).
|
||
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
|
||
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
|
||
(`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
|
||
* **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
|
||
|
||
## Measured on prod, 2026-10-06 — what is now known rather than assumed
|
||
|
||
The first real reconcile settled four things this component's design rested on. Three went the
|
||
right way; one was a defect in my own manifest.
|
||
|
||
| question | answer |
|
||
|---|---|
|
||
| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
|
||
| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
|
||
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
|
||
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
|
||
|
||
## Measured on prod, 2026-10-08 — why the forwarded node cannot be any node
|
||
|
||
The first device enrolled, handshook, resolved names, and could not open a single connection to any
|
||
cluster web app. The tunnel was not at fault — every part of it measured healthy:
|
||
`net.ipv4.ip_forward=1`, `wg0` up at `10.90.0.1/24` MTU 1420, the peer authorised with real
|
||
bidirectional transfer (346 KiB in / 200 KiB out), `MASQUERADE` present and counting 26 flows,
|
||
`FORWARD -i/-o wg0 ACCEPT` counting 108 / 171 packets, and PiHole DNS answering from the node.
|
||
|
||
**The fault was that a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
|
||
holds itself.** Cilium's announcer answers ARP for the VIP on behalf of other LAN hosts, but not for
|
||
the announcing node's own request, so the neighbour entry settles at `FAILED`:
|
||
|
||
| from | `192.168.2.20` (Gateway) | `192.168.2.22` (crafty) | `192.168.2.24` (plex) | announces |
|
||
|---|---|---|---|---|
|
||
| `k3s-prod-1` | **FAILED** | — | — | `.20` |
|
||
| `k3s-prod-2` | ok | **INCOMPLETE** | ok | `.21`, `.22`, `.23-tcp` |
|
||
| `k3s-prod-3` | ok | ok | **INCOMPLETE** | `.24` |
|
||
|
||
Symmetric across 3 nodes × 5 VIPs: each node reached every VIP **except** the ones it alone
|
||
announced. It is a property of L2 announcement, not of any one Service, and it is harmless for
|
||
ordinary pods — they reach Services through the cluster datapath and never ARP for a VIP. This
|
||
component is the exception, because `hostNetwork` + `MASQUERADE` put remote clients on the node's
|
||
own LAN path.
|
||
|
||
The forward pointed at `k3s-prod-1`, which held the `shared` Gateway lease, so **all 16 Gateway
|
||
hostnames were dead over the VPN**. Two things hid it:
|
||
|
||
* **PiHole survived by accident.** `192.168.2.23` is announced by *two* Services
|
||
(`pihole-dns-udp` on prod-1, `pihole-dns-tcp` on prod-2), so prod-2 answered the ARP prod-1 could
|
||
not. Names resolved perfectly and every app hung — the hardest shape to diagnose from a client.
|
||
* **Nothing reported it.** No probe in this repo compares the forwarded node against the lease
|
||
holders, and from outside WireGuard is silent to unauthenticated traffic anyway.
|
||
|
||
Closed by labelling the forwarded node `homelab.keithjackson.dev/vpn-ingress=true` and excluding
|
||
that label in `platform-config/cilium/l2-announcement-policy.yaml` (the node then ARP-resolves every
|
||
VIP), with `VpnIngressNodeAnnouncesL2VIP` watching the lease holders so a recurrence is loud and
|
||
`VpnIngressNodeLabelMissing` watching for the label going absent, which would silently re-arm the
|
||
whole failure. Cost: VIP announcement candidacy drops from three nodes to two.
|
||
|
||
## Prerequisites outside this directory
|
||
|
||
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
|
||
(`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
|
||
`NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
|
||
CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
|
||
2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a
|
||
`REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring
|
||
is complete and the only outstanding act is replacing the value (runbook step 2). Until then the
|
||
DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the
|
||
value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by
|
||
mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is
|
||
the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves
|
||
every Flux condition green.
|
||
3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
|
||
and the half CI can never check.
|
||
|
||
## Not built (deliberate, with reasons)
|
||
|
||
* **No metrics / no peer-level alerting.** A WireGuard exporter would expose
|
||
`wireguard_latest_handshake_seconds` per peer — genuinely useful for spotting unexpected
|
||
activity — but it needs a second `NET_ADMIN` container in the pod that terminates the
|
||
internet-facing tunnel, and that trade was not worth making unverified. Pod-down is already
|
||
covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
|
||
kube-prometheus-stack. **This is the top follow-up.**
|
||
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
|
||
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
|
||
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
|
||
`wg-server-key` Secret.
|
||
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
|
||
still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point
|
||
was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's
|
||
`AllowedIPs`; not done because nothing here needs it.
|
||
* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
|
||
browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
|
||
into the tunnel.
|