From 8f60e6103a460b63a75bef463d5a5b8e88bb8859 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 8 Nov 2026 21:14:37 -0600 Subject: [PATCH] fix(vpn): default client profiles to MTU 1280; correct the VPN runbook new-peer.sh generated MTU 1420, which stalled traffic on mobile data despite a healthy handshake (one phone, one carrier, 2026-11-09). Default to 1280, overridable through WG_MTU. Correct stale claims in the runbook and the DaemonSet header against live BPF conntrack: LoadBalancer VIPs see 10.90.0.x as world, not a SNATed host identity. Add a troubleshooting section. The DaemonSet render is unchanged, so WireGuard pods do not roll. Co-Authored-By: Claude Opus 5.5 --- decisions/2026-11-08-wg-client-mtu-docs.md | 74 ++++++++++++++ infra/vpn-ingress/new-peer.sh | 10 +- platform/vpn-ingress/README.md | 4 +- platform/vpn-ingress/daemonset.yaml | 9 +- runbooks/vpn-remote-access.md | 110 ++++++++++++++++++--- 5 files changed, 190 insertions(+), 17 deletions(-) create mode 100644 decisions/2026-11-08-wg-client-mtu-docs.md diff --git a/decisions/2026-11-08-wg-client-mtu-docs.md b/decisions/2026-11-08-wg-client-mtu-docs.md new file mode 100644 index 0000000..73cb50f --- /dev/null +++ b/decisions/2026-11-08-wg-client-mtu-docs.md @@ -0,0 +1,74 @@ +--- +date: 2026-11-08 +branch: fix/wg-client-mtu-docs +title: WireGuard client profiles default to MTU 1280; VPN runbook corrected and given a troubleshooting section +--- + +## What changed + +- **`infra/vpn-ingress/new-peer.sh`**: `WG_MTU` default 1420 -> 1280, still overridable, reason in a + comment. Only new profiles change; the phone's existing profiles are not touched. +- **`runbooks/vpn-remote-access.md`**: the verification table no longer says the Gateway sees a + `host`-identity source; the limitations no longer say everything arrives SNATed or that IPv6 leaks + on Android; the "if nothing handshakes" advice no longer claims `wg show` separates a bad forward + from a bad key. New "Troubleshooting" section covering the four cases in the brief. +- **`platform/vpn-ingress/daemonset.yaml`**: header comment only, SNAT claim corrected. +- **`platform/vpn-ingress/README.md`**: one sentence, so `MTU = 1420` reads as the server's value. + +Rollout on merge: the WireGuard DaemonSet does **not** roll. The `runbooks` ConfigMap is content +hashed and `vpn-remote-access.md` is in it, so the runbooks site rolls. + +Left unchanged, for the operator: + +- **`platform/vpn-ingress/peers.conf`** still says "No peers enrolled yet". A comment-only edit on a + throwaway copy moved the ConfigMap from `wg-peers-29mf6k867t` to `wg-peers-gggkk5d4tm` and changed + the DaemonSet, so correcting it restarts every `wg0`. Fix it alongside the next peer change. +- **`clusters/base/cluster-vars.yaml`** coverage comment still omits `WG_CLIENT_CIDR` (sensitive path). +- **Server `wg0` MTU stays 1420**, because changing it restarts all WireGuard pods. For TCP a 1280 + client should be enough on its own: it advertises an MSS of 1240, which caps what the far end + sends. That protection does not reach UDP. A server-to-phone datagram with an inner size of + 1281-1420 bytes is still sent as up to 1480 outer bytes and could be lost on the same path; QUIC + should recover by probing, large DNS answers are the likeliest casualty. If that shows up, the + fix is a lower server MTU, as a deliberate rollout. +- **Other stale text, outside this brief:** `platform/vpn-ingress/README.md` still says no peer has + ever completed a handshake, and the runbook's status line still calls the VPN inert. + +## What I verified + +- Upstream `wireguard-android` `GoBackend.java`, master (last commit to that file `ecc3c0a`, + 2026-03-15): `builder.setMtu(config.getInterface().getMtu().orElse(1280))`, and `allowFamily()` is + skipped when `sawDefaultRoute && config.getPeers().size() == 1`. +- Live, read-only, 2026-11-09 UTC, on the `k3s-prod-1` pod and agent: + - `wg0` MTU is 1420 on all three pods. Both peers have handshakes and transfer on prod-1 only; the + other two pods read `0`. + - BPF conntrack holds `UDP SVC 10.90.0.x -> 192.168.2.23:53` and flows to `192.168.2.24:32400`. + Their backend entries carry `SourceSecurityID=2` (`world`). So the source stays `10.90.0.x` + through the load balancer on those two VIPs. + - `/proc/net/nf_conntrack` is readable from the pod, which has no `conntrack` binary. +- Offline, pinned toolbox: `kubectl kustomize platform/vpn-ingress` gives identical ConfigMap and + DaemonSet hashes on `origin/main` and on this branch. +- `new-peer.sh` writes `MTU = 1280` by default and `MTU = 1420` with `WG_MTU=1420`. Run with + throwaway keys and stdout discarded; profiles shredded; nothing enrolled. +- No test or conformance check pins the client MTU (`grep` for `WG_MTU` and `1420`). +- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not green.** + `gitea-settings` is the only stage counted as not run (no token locally). `conformance` (L2/L3) + also skipped as the declared no-cluster gap, so R-ENV-01 and R-OBS-04 are not certified. + `spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1` + passed. No sensitive path is touched. + +## What I did NOT verify + +- **That MTU was the cause.** It is inferred from one before-and-after, on one phone and one carrier, + reported by the operator. The path MTU was never measured. +- **That 1280 is the best value.** It is the value that worked, and the IPv6 minimum. A higher one + may work too. +- That the phone's app is on `GoBackend` rather than the kernel backend, so that a blank field meant + 1280 on this device. Only the upstream default was read. +- That TCP MSS behaves as described, or that large UDP replies are actually lost. No packet capture + was taken. +- That the Gateway VIP `.20` sees `10.90.0.x` / `world`. Conntrack held no tunnel flows to it when + checked. +- Android's IPv6 blocking on a real device; read from source only. +- `wg-quick` IPv6 behaviour on Linux, and any non-Android client at 1280. +- `/proc/net/nf_conntrack` output during a key mismatch. The `[UNREPLIED]` reading comes from the + earlier measurement in the brief; I saw only healthy entries. diff --git a/infra/vpn-ingress/new-peer.sh b/infra/vpn-ingress/new-peer.sh index 494404d..113c010 100755 --- a/infra/vpn-ingress/new-peer.sh +++ b/infra/vpn-ingress/new-peer.sh @@ -53,6 +53,7 @@ # WG_ENDPOINT_HOST default wan.home.keithjackson.dev # WG_CLIENT_CIDR default 10.90.0.0/24 (must match clusters/base/cluster-vars.yaml) # WG_LAN_CIDR default 192.168.2.0/24 (prod LAN_CIDR) +# WG_MTU default 1280 (client tunnel MTU; see the comment on the default below) # WG_DNS default 192.168.2.23 (prod PIHOLE_DNS_VIP) # # REQUIRES: `wg`, `qrencode`, `sops` — all three in the dev container. @@ -80,7 +81,14 @@ PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf" # cannot resolve until it is already connected. : "${WG_ENDPOINT_HOST:=wan.home.keithjackson.dev}" : "${WG_LISTEN_PORT:=51820}" -: "${WG_MTU:=1420}" +# The CLIENT's tunnel MTU. Deliberately lower than the server's 1420 (daemonset.yaml), which only +# describes the 1500-byte path at the node's end. A phone's path is the carrier's, often behind +# carrier NAT, and is not known to carry 1500. Measured 2026-11-09, one Android phone on one +# carrier: at 1420 the handshake was healthy but names and pages failed; clearing the app's MTU +# field (its userspace backend then uses 1280) fixed it. 1280 is also the IPv6 minimum link MTU, so +# it stays valid if the tunnel ever gains IPv6. For a device only ever on a wired or known-1500 +# path, WG_MTU=1420 recovers about 10% of per-packet payload. +: "${WG_MTU:=1280}" SPLIT=0 OUT_DIR="" diff --git a/platform/vpn-ingress/README.md b/platform/vpn-ingress/README.md index 84b23c6..3f942b5 100644 --- a/platform/vpn-ingress/README.md +++ b/platform/vpn-ingress/README.md @@ -112,7 +112,9 @@ no change. Three further problems vanish with the host netns: `conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no `externalTrafficPolicy` trade-off to make at all. * **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat - 1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct. + 1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` correct on the + server's `wg0`. It says nothing about a client's own path: `new-peer.sh` defaults client profiles + to `1280` (see "Troubleshooting" in `runbooks/vpn-remote-access.md`). `hostNetwork` is *permitted* here, and not by oversight: `platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector: diff --git a/platform/vpn-ingress/daemonset.yaml b/platform/vpn-ingress/daemonset.yaml index c1a203d..93d55c4 100644 --- a/platform/vpn-ingress/daemonset.yaml +++ b/platform/vpn-ingress/daemonset.yaml @@ -30,9 +30,12 @@ # `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either. # The VPN would have looked up and been useless, which is the worst failure shape available. # -# In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the -# NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` — -# exactly what a LAN client looks like. Three further problems disappear with it: +# In the host netns there is no pod identity to carry, so a client never looks like a pod. What a +# destination sees depends on how the packet is handled. LAN hosts and the internet: forwarded, +# then MASQUERADEd to the NODE's address by the PostUp rule below. LoadBalancer VIPs: Cilium's load +# balancer takes the packet as it enters from wg0, before that rule, so the source is still +# 10.90.0.x with identity `world`, like a LAN browser (measured 2026-11-09 for PiHole .23 and Plex +# .24; see README). Three further problems disappear with it: # * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to # /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward` # is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have diff --git a/runbooks/vpn-remote-access.md b/runbooks/vpn-remote-access.md index 9f123ab..ce52ceb 100644 --- a/runbooks/vpn-remote-access.md +++ b/runbooks/vpn-remote-access.md @@ -252,7 +252,7 @@ cellular, with Wi-Fi off, and check all four — each proves a different hop: |---|---|---| | 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza | | 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path | -| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd | +| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway admitting a tunnel client, which a pod-networked VPN would have 403'd. Expected source identity is `world`, like a LAN browser: measured 2026-11-09 on VIPs `.23` and `.24`, not yet observed on the Gateway's `.20` | | 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier | Then confirm server-side: @@ -260,18 +260,97 @@ Then confirm server-side: ```bash ./infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show' -# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer. +# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer. `ds/wireguard` +# picks ONE pod; if it reads 0, ask the pod on the node holding the forward (see Troubleshooting). ``` **If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up. -**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60 -bytes of WireGuard encapsulation); drop the client to `1280` and re-test. +**If 1 and 2 pass but pages or large transfers hang**, suspect the client MTU; see +Troubleshooting. -**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside — -by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes -if the router is not delivering the packets, and a `handshake` with no completed session if it is. +**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from the +device, by design. `wg show` cannot tell them apart either: it reads the same in both cases. The +node's conntrack table can; see Troubleshooting. + +## Troubleshooting + +Ask the pod on **the node carrying the traffic**: the node the WAN forward names for `away`, +`k3s-prod-1` for `home`. Sessions are per node, and the other two pods read `0`. + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress get pods -o wide' + +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress exec -c wireguard -- wg show wg0 latest-handshakes' +# likewise: `wg show wg0 endpoints`, `wg show wg0 transfer` +``` + +**Use only those three subcommands to diagnose.** Never `wg showconf`, `wg show wg0 private-key` or +`wg show wg0 preshared-keys`: they print key material into your terminal and transcript. + +| output | reading | +|---|---| +| `latest-handshakes`: ` 0` | no handshake with this peer since the pod started | +| `endpoints`: ` (none)` | the server has not accepted a packet from this peer since the pod started | +| `transfer`: ` ` | bytes, as the server counts them | + +### The app shows TX, no RX and no "Latest handshake" + +**Symptom.** The app's transmit counter climbs and receive stays at zero. On the server, +`endpoints` is `(none)` and `latest-handshakes` is `0` for that peer. + +**Cause.** A key mismatch: the server's public key in the profile, or the device's public key in +`peers.conf`. WireGuard drops an initiation it cannot authenticate and sends nothing back, so +neither end logs an error. Measured 2026-11-09: a mistyped server public key in the phone's profile. + +**Check.** First prove the packets arrive. The pod has no `conntrack` binary, but in `hostNetwork` +it can read the host's table: + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress exec -c wireguard -- grep dport=51820 /proc/net/nf_conntrack' +``` + +- A line whose first tuple is `src= ... dport=51820`, marked + `[UNREPLIED]`: the datagrams reach the node and the server stays silent. It is the keys. +- No such line: the packets are not arriving. Look at the router forward, the DNS name, or the + device's network. + +Then compare the keys **by copy and paste, never by eye and never retyped**: the server's public +half (the first command in Step 2's "Record it in your PASSWORD MANAGER") against the profile's +`PublicKey`, and the device's public key from the app against its stanza in `peers.conf`. + +### Works on Wi-Fi, stalls on mobile data, handshake healthy + +**Symptom.** `latest-handshakes` is recent and some traffic flows, but names or pages fail on +mobile data. The same device is fine on Wi-Fi. + +**Cause.** Most likely a client MTU too large for the carrier's path: a tunnel MTU of `1420` means +outer packets of up to 1480 bytes. This is inferred from one before-and-after, on one phone and one +carrier (2026-11-09): `1420` failed and clearing the field fixed it. The path MTU itself was not +measured. + +**Check.** In the Android WireGuard app, edit the tunnel, clear the `MTU` field or set it to `1280`, +save, then turn the tunnel off and on. A blank field means `1280` in the app's default userspace +backend (`GoBackend`: `getMtu().orElse(1280)`). On other clients set `MTU = 1280` in the profile. + +`new-peer.sh` now generates `MTU = 1280` unless `WG_MTU` overrides it. A profile already on a device +does not change, so edit it by hand. If `1280` does not help, it is not the MTU: go back to +Step 6's check 3. + +### The `away` profile cannot connect from home + +**Symptom.** On the home network, `away` never handshakes. + +**Cause.** Its Endpoint is `wan.home.keithjackson.dev`, the house's WAN address, and the router has +no hairpin NAT. WireGuard cannot use the Cloudflare Tunnel instead. + +**Check.** Turn Wi-Fi off. If `away` handshakes on mobile data, the profile is fine: use the `home` +tunnel indoors, as described in +[Phone profiles: one tunnel for home, one for away](../platform/vpn-ingress/README.md#phone-profiles-one-tunnel-for-home-one-for-away). ## Revoking a device @@ -293,10 +372,17 @@ top follow-up for this component.** Until then: ## Known limitations -* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside - the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point. -* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's - address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would - preserve them — at the cost of another hand-maintained off-repo route. +* **No IPv6 in the tunnel.** What a client's IPv6 does then depends on the client. In the Android + app's default `GoBackend`, a config with one peer and a default route skips `allowFamily()`, so + Android should block IPv6 while the tunnel is up and apps fall back to IPv4. That is read from + upstream source, not observed on a device. A `--split` profile has no default route, so its IPv6 + stays outside the tunnel. Expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside too + (not tested). +* **Client IPs are only partly preserved.** LAN hosts and the internet see the node's address + (MASQUERADE). A static route for `10.90.0.0/24` → the forwarded node, added on the router, would + preserve them there, at the cost of another hand-maintained off-repo route. LoadBalancer VIPs are + different: Cilium handles the packet before MASQUERADE, so the source stays `10.90.0.x`, identity + `world` (BPF conntrack on prod-1, 2026-11-09, VIPs `.23` and `.24`). Any `loadBalancerSourceRanges` + must therefore list the tunnel subnet, as PiHole's do. * **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients re-handshake automatically (retry every 5s, rekey every 2 min).