Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(vpn-ingress): WireGuard remote access on udp/51820 — hostNetwork, because a pod identity would 403 every Gateway hostname #55

Merged
jakman85 merged 1 commits from feat/vpn-remote-access-wireguard into main 2026-10-06 03:41:49 +00:00
Conversation 1 Commits 1 Files Changed 16
+1412 -1
gitea-admin commented 2026-10-06 03:13:20 +00:00
Owner
Copy link
Copy source

Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere on the internet and behave like machines on the LAN: full reach to 192.168.2.0/24 plus internet egress out the house WAN.

Not platform/vpn-egress. That sends pods out through PIA (ADR-0015). This lets remote clients in. They share nothing but the word "VPN"; both READMEs and the layout block now say so.

Why WireGuard

Only option with a first-party client on all four named platforms (official apps for Android/iOS/Windows/macOS; in-kernel on Linux since 5.6). OpenVPN would need a CA + per-device certs + a CRL — three pieces of long-lived state to protect and rotate.

The security argument is the handshake's failure mode, not "it is encrypted." WireGuard's first handshake message carries a MAC keyed with the responder's own public key, so a sender that does not already know it gets no reply at all — no error, no reset, no banner. From the internet udp/51820 is indistinguishable from a closed port: nothing to brute-force, no version string, no amplification. That is what makes an unfiltered WAN port defensible, and why there is deliberately no source allowlist (which would also break roaming clients).

hostNetwork: true is required, not convenient — this is the decision to review

A pod-networked WireGuard server must SNAT clients to its own pod IP (Cilium drops a packet whose source is not the endpoint's address), so every remote client would arrive wearing that pod's Cilium identity. This cluster has already measured what that costs (decisions/2026-10-04-plex-advertise-plexdirect.md): a request to the shared Gateway VIP from inside a pod returns 403 Access denied from Envoy for all 16 Gateway hostnames, because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy.

So the obvious build would have shipped a tunnel that came up healthy, showed connected peers, routed packets — and 403'd every cluster web UI, with *.home.keithjackson.dev failing to resolve too (pihole-dns.yaml deliberately omits the cluster entity). A VPN that looks up and is useless, with nothing in the manifests hinting at it.

In the host netns there is no pod identity to carry: traffic is SNATed to the node and arrives as host/remote-node — byte-for-byte what a LAN client looks like. Three more problems vanish: ip_forward is already 1 on these nodes (and is not a kubelet-safe sysctl, so the pod version needed --allowed-unsafe-sysctls on all three kubelets); no LoadBalancer VIP, so no L2-announcement lease lottery; no double encapsulation, which is what makes MTU = 1420 simply correct.

Permitted by restrict-host-namespaces' own carve-out (it matches app-namespace=true only and names Cilium/node-exporter as precedent); namespace.yaml is deliberately unlabelled. The cost is stated, not buried: a hostNetwork pod has no Cilium endpoint, so no NetworkPolicy constrains this workload. networkpolicy.yaml says so at length instead of shipping an allow world → 51820 rule that would read as the control and enforce nothing.

Other choices

  • DaemonSet, not Deployment. The WAN forward names one node address by hand, off-repo; a floating replica would silently invalidate it on every reschedule. All three nodes answer the same keys, so failover is a one-line router change. A pinned LB VIP was considered and rejected: it reintroduces the L2 lease, SNAT and an extra hop into the one path whose failure mode is "I am away from home and cannot get in".
  • Authorisation split to stay reviewable. Peer public keys are plaintext in peers.yaml — a public key grants nothing, and this makes "who can get into my house" a PR diff with git log as its audit trail. The only real secret, the server's private key, is an out-of-band Secret: an agent must not mint a long-lived private key (CLAUDE.md §6). The halves are joined only at pod start into a medium: Memory emptyDir, so the assembled config never touches a node's disk.
  • Ships INERT-BUT-HEALTHY with zero peers — for the same reason. A zero-peer interface listens and refuses every handshake.
  • egress-class: internet is load-bearing: the PIA CiliumEgressGatewayPolicy selects egress-class: vpn cluster-wide with no namespace scope, so vpn here would have sent every client's internet traffic out PIA.

Gate

./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' → rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass. The rc 3 is two environmental skips: gitea-settings (no local SETTINGS_API_TOKEN) and conformance L2/L3 (CLUSTER_AVAILABLE unset). Not green — reported as rc 3, not as pass.

flux-postbuild-substitution (L0) caught a real defect in this branch on its first run, naming platform/vpn-ingress/daemonset.yaml:118 and attributing it to Flux Kustomization 'vpn-ingress' — which also proves the new tier's postBuild.substituteFrom is correctly detected. The hazard was a comment that spelled out the forbidden braced form literally while claiming not to use it. Fixed; 444/444, 0 hazards.

THREE manual steps remain — the VPN does not work until all three are done

All off-repo by necessity (runbooks/vpn-remote-access.md):

  1. Run node_baseline to load the wireguard kernel module on every node.
  2. Create the wg-server-key Secret (and keep a copy in your password manager — losing it means re-enrolling every device).
  3. Forward WAN udp/51820 to one node address on the SOHO router.

Then ./infra/vpn-ingress/new-peer.sh <name> <octet> enrols a device (QR code for phones), and runbook step 6 verifies it from off the LAN — the only honest test, and one this repo has no vantage point for.

What is NOT verified

Nothing here has carried a packet. The DaemonSet has never been reconciled and no wg0 has ever existed. Biggest single doubt, called out in the decisions entry: the in-pod iptables → host-nftables interaction. If the image's binary is the legacy backend while Debian 13 is nft, the rules can land in a table the kernel does not consult — and the symptom is the nasty one: handshakes succeed, wg show looks perfect, every forwarded connection hangs. Check iptables-save -t nat | grep MASQUERADE on the node first.

Worth arguing about on this PR: an enrolled device reaches the three k3s nodes themselves — apiserver 6443, kubelet 10250, SSH 22 — because "reach the entire local network" includes them. Documented with a narrowing recipe, but it is a posture decision, not a verified fact.

Sensitive paths

Touches clusters/** (3 files), so ci/approval-policy.sh will BLOCK this pending /approve-override <reason> posted as a PR review of type "Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own PR, and the agent account holds owner, so the prohibition is a rule, not a limit.

🤖 Generated with Claude Code

Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere on the internet and behave like machines on the LAN: full reach to `192.168.2.0/24` plus internet egress out the house WAN. > **Not `platform/vpn-egress`.** That sends pods **out** through PIA (ADR-0015). This lets remote clients **in**. They share nothing but the word "VPN"; both READMEs and the layout block now say so. ## Why WireGuard Only option with a **first-party client on all four named platforms** (official apps for Android/iOS/Windows/macOS; in-kernel on Linux since 5.6). OpenVPN would need a CA + per-device certs + a CRL — three pieces of long-lived state to protect and rotate. **The security argument is the handshake's failure mode, not "it is encrypted."** WireGuard's first handshake message carries a MAC keyed with the *responder's own public key*, so a sender that does not already know it gets **no reply at all** — no error, no reset, no banner. From the internet udp/51820 is indistinguishable from a closed port: nothing to brute-force, no version string, no amplification. That is what makes an unfiltered WAN port defensible, and why there is deliberately no source allowlist (which would also break roaming clients). ## `hostNetwork: true` is required, not convenient — this is the decision to review A pod-networked WireGuard server must SNAT clients to its own pod IP (Cilium drops a packet whose source is not the endpoint's address), so every remote client would arrive wearing **that pod's** Cilium identity. This cluster has already measured what that costs (`decisions/2026-10-04-plex-advertise-plexdirect.md`): a request to the shared Gateway VIP **from inside a pod returns `403 Access denied` from Envoy for all 16 Gateway hostnames**, because Cilium's Gateway preserves the original source identity and enforces the *destination's* ingress policy. So the obvious build would have shipped a tunnel that came up healthy, showed connected peers, routed packets — and **403'd every cluster web UI**, with `*.home.keithjackson.dev` failing to resolve too (`pihole-dns.yaml` deliberately omits the `cluster` entity). A VPN that looks up and is useless, with nothing in the manifests hinting at it. In the host netns there is no pod identity to carry: traffic is SNATed to the node and arrives as `host`/`remote-node` — byte-for-byte what a LAN client looks like. Three more problems vanish: `ip_forward` is already 1 on these nodes (and is *not* a kubelet-safe sysctl, so the pod version needed `--allowed-unsafe-sysctls` on all three kubelets); no LoadBalancer VIP, so no L2-announcement lease lottery; no double encapsulation, which is what makes `MTU = 1420` simply correct. Permitted by `restrict-host-namespaces`' own carve-out (it matches `app-namespace=true` only and names Cilium/node-exporter as precedent); `namespace.yaml` is deliberately unlabelled. **The cost is stated, not buried:** a hostNetwork pod has no Cilium endpoint, so **no NetworkPolicy constrains this workload**. `networkpolicy.yaml` says so at length instead of shipping an `allow world → 51820` rule that would read as the control and enforce nothing. ## Other choices - **DaemonSet, not Deployment.** The WAN forward names one node address by hand, off-repo; a floating replica would silently invalidate it on every reschedule. All three nodes answer the same keys, so **failover is a one-line router change**. A pinned LB VIP was considered and rejected: it reintroduces the L2 lease, SNAT and an extra hop into the one path whose failure mode is "I am away from home and cannot get in". - **Authorisation split to stay reviewable.** Peer **public** keys are plaintext in `peers.yaml` — a public key grants nothing, and this makes *"who can get into my house"* a PR diff with `git log` as its audit trail. The only real secret, the server's private key, is an out-of-band Secret: an agent must not mint a long-lived private key (CLAUDE.md §6). The halves are joined only at pod start into a `medium: Memory` emptyDir, so the assembled config never touches a node's disk. - **Ships INERT-BUT-HEALTHY with zero peers** — for the same reason. A zero-peer interface listens and refuses every handshake. - **`egress-class: internet`** is load-bearing: the PIA `CiliumEgressGatewayPolicy` selects `egress-class: vpn` **cluster-wide with no namespace scope**, so `vpn` here would have sent every client's internet traffic out PIA. ## Gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'` → **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**. The rc 3 is two environmental skips: `gitea-settings` (no local `SETTINGS_API_TOKEN`) and conformance L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green — reported as rc 3, not as pass.** `flux-postbuild-substitution` (L0) **caught a real defect in this branch** on its first run, naming `platform/vpn-ingress/daemonset.yaml:118` and attributing it to `Flux Kustomization 'vpn-ingress'` — which also proves the new tier's `postBuild.substituteFrom` is correctly detected. The hazard was a comment that spelled out the forbidden braced form literally while claiming not to use it. Fixed; 444/444, 0 hazards. ## THREE manual steps remain — the VPN does not work until all three are done All off-repo by necessity (`runbooks/vpn-remote-access.md`): 1. Run `node_baseline` to load the **`wireguard` kernel module** on every node. 2. Create the **`wg-server-key`** Secret (and keep a copy in your password manager — losing it means re-enrolling every device). 3. Forward **WAN udp/51820** to one node address on the SOHO router. Then `./infra/vpn-ingress/new-peer.sh <name> <octet>` enrols a device (QR code for phones), and runbook step 6 verifies it **from off the LAN** — the only honest test, and one this repo has no vantage point for. ## What is NOT verified Nothing here has carried a packet. The DaemonSet has never been reconciled and no `wg0` has ever existed. Biggest single doubt, called out in the decisions entry: **the in-pod `iptables` → host-nftables interaction.** If the image's binary is the legacy backend while Debian 13 is nft, the rules can land in a table the kernel does not consult — and the symptom is the nasty one: handshakes succeed, `wg show` looks perfect, every forwarded connection hangs. Check `iptables-save -t nat | grep MASQUERADE` on the node first. **Worth arguing about on this PR:** an enrolled device reaches the three k3s nodes themselves — apiserver 6443, kubelet 10250, SSH 22 — because "reach the entire local network" includes them. Documented with a narrowing recipe, but it is a posture decision, not a verified fact. ## Sensitive paths Touches `clusters/**` (3 files), so `ci/approval-policy.sh` will **BLOCK** this pending `/approve-override <reason>` posted as a PR **review** of type "Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
gitea-admin added 1 commit 2026-10-06 03:13:20 +00:00
feat(vpn-ingress): WireGuard remote access on udp/51820 (hostNetwork, declarative peers)
gate / gate (pull_request) Failing after 1m48s
Details
gate / conformance (pull_request) Has been skipped
Details
1a67fc1b3c 
Lets the operator's Android / Linux / Windows / macOS devices dial in from
anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus
internet egress out the house WAN. Opposite direction to platform/vpn-egress,
which sends pods OUT through PIA; the two share nothing but the word "VPN".

WireGuard because it is the only option with a first-party client on all four
named platforms, and because its handshake MAC is keyed with the responder's
own public key: an unauthenticated packet gets NO reply, so udp/51820 is
indistinguishable from a closed port. That, not "it is encrypted", is what
makes an unfiltered WAN port defensible, and why there is no source allowlist.

hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT
clients to its own pod IP, so every remote client would carry that pod's Cilium
identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from
Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway
preserves the original source identity and enforces the destination's ingress
policy. The obvious build would have shipped a healthy-looking tunnel on which
every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve.
In the host netns traffic is SNATed to the node and arrives as host/remote-node,
exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the
L2-announcement lease lottery, and double encapsulation (hence MTU 1420).
Permitted by restrict-host-namespaces' own carve-out for platform namespaces.
The cost is stated in networkpolicy.yaml rather than papered over: no
NetworkPolicy constrains a hostNetwork pod.

A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule;
all three nodes answer the same keys, so failover is a one-line router change.

Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my
house" a reviewable PR diff. The only secret -- the server's private key -- is
an out-of-band Secret: an agent must not mint a long-lived private key, and the
two halves are joined only at pod start into a tmpfs that dies with the pod.

Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain
(kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md.

Touches clusters/** , so ci/approval-policy.sh will block pending the
operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
gitea-admin force-pushed feat/vpn-remote-access-wireguard from 1a67fc1b3c to 7f7b7e8cbb 2026-10-06 03:24:14 +00:00 Compare
gitea-admin force-pushed feat/vpn-remote-access-wireguard from 7f7b7e8cbb to f92707f38c 2026-10-06 03:29:48 +00:00 Compare
jakman85 approved these changes 2026-10-06 03:38:46 +00:00
jakman85 merged commit ad3f56eee8 into main 2026-10-06 03:41:49 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
jakman85
Labels
Clear labels
agent
automated-update
blast-radius
breaking
ci-tooling
dependencies
needs-review
own-image
range-widen
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
gitea-admin jakman85 (Keith Jackson) pm-agent renovate-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#55