Merge pull request 'fix(vpn): default client profiles to MTU 1280; correct the VPN runbook' (#159) from fix/wg-client-mtu-docs into main
Reviewed-on: #159
This commit was merged in pull request #159.
This commit is contained in:
jakman85 committed
commit
50c8719b12
5 files changed
+190
-17
No files matched your search
@@ -0,0 +1,74 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: fix/wg-client-mtu-docs
|
||||
title: WireGuard client profiles default to MTU 1280; VPN runbook corrected and given a troubleshooting section
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
- **`infra/vpn-ingress/new-peer.sh`**: `WG_MTU` default 1420 -> 1280, still overridable, reason in a
|
||||
comment. Only new profiles change; the phone's existing profiles are not touched.
|
||||
- **`runbooks/vpn-remote-access.md`**: the verification table no longer says the Gateway sees a
|
||||
`host`-identity source; the limitations no longer say everything arrives SNATed or that IPv6 leaks
|
||||
on Android; the "if nothing handshakes" advice no longer claims `wg show` separates a bad forward
|
||||
from a bad key. New "Troubleshooting" section covering the four cases in the brief.
|
||||
- **`platform/vpn-ingress/daemonset.yaml`**: header comment only, SNAT claim corrected.
|
||||
- **`platform/vpn-ingress/README.md`**: one sentence, so `MTU = 1420` reads as the server's value.
|
||||
|
||||
Rollout on merge: the WireGuard DaemonSet does **not** roll. The `runbooks` ConfigMap is content
|
||||
hashed and `vpn-remote-access.md` is in it, so the runbooks site rolls.
|
||||
|
||||
Left unchanged, for the operator:
|
||||
|
||||
- **`platform/vpn-ingress/peers.conf`** still says "No peers enrolled yet". A comment-only edit on a
|
||||
throwaway copy moved the ConfigMap from `wg-peers-29mf6k867t` to `wg-peers-gggkk5d4tm` and changed
|
||||
the DaemonSet, so correcting it restarts every `wg0`. Fix it alongside the next peer change.
|
||||
- **`clusters/base/cluster-vars.yaml`** coverage comment still omits `WG_CLIENT_CIDR` (sensitive path).
|
||||
- **Server `wg0` MTU stays 1420**, because changing it restarts all WireGuard pods. For TCP a 1280
|
||||
client should be enough on its own: it advertises an MSS of 1240, which caps what the far end
|
||||
sends. That protection does not reach UDP. A server-to-phone datagram with an inner size of
|
||||
1281-1420 bytes is still sent as up to 1480 outer bytes and could be lost on the same path; QUIC
|
||||
should recover by probing, large DNS answers are the likeliest casualty. If that shows up, the
|
||||
fix is a lower server MTU, as a deliberate rollout.
|
||||
- **Other stale text, outside this brief:** `platform/vpn-ingress/README.md` still says no peer has
|
||||
ever completed a handshake, and the runbook's status line still calls the VPN inert.
|
||||
|
||||
## What I verified
|
||||
|
||||
- Upstream `wireguard-android` `GoBackend.java`, master (last commit to that file `ecc3c0a`,
|
||||
2026-03-15): `builder.setMtu(config.getInterface().getMtu().orElse(1280))`, and `allowFamily()` is
|
||||
skipped when `sawDefaultRoute && config.getPeers().size() == 1`.
|
||||
- Live, read-only, 2026-11-09 UTC, on the `k3s-prod-1` pod and agent:
|
||||
- `wg0` MTU is 1420 on all three pods. Both peers have handshakes and transfer on prod-1 only; the
|
||||
other two pods read `0`.
|
||||
- BPF conntrack holds `UDP SVC 10.90.0.x -> 192.168.2.23:53` and flows to `192.168.2.24:32400`.
|
||||
Their backend entries carry `SourceSecurityID=2` (`world`). So the source stays `10.90.0.x`
|
||||
through the load balancer on those two VIPs.
|
||||
- `/proc/net/nf_conntrack` is readable from the pod, which has no `conntrack` binary.
|
||||
- Offline, pinned toolbox: `kubectl kustomize platform/vpn-ingress` gives identical ConfigMap and
|
||||
DaemonSet hashes on `origin/main` and on this branch.
|
||||
- `new-peer.sh` writes `MTU = 1280` by default and `MTU = 1420` with `WG_MTU=1420`. Run with
|
||||
throwaway keys and stdout discarded; profiles shredded; nothing enrolled.
|
||||
- No test or conformance check pins the client MTU (`grep` for `WG_MTU` and `1420`).
|
||||
- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not green.**
|
||||
`gitea-settings` is the only stage counted as not run (no token locally). `conformance` (L2/L3)
|
||||
also skipped as the declared no-cluster gap, so R-ENV-01 and R-OBS-04 are not certified.
|
||||
`spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1`
|
||||
passed. No sensitive path is touched.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That MTU was the cause.** It is inferred from one before-and-after, on one phone and one carrier,
|
||||
reported by the operator. The path MTU was never measured.
|
||||
- **That 1280 is the best value.** It is the value that worked, and the IPv6 minimum. A higher one
|
||||
may work too.
|
||||
- That the phone's app is on `GoBackend` rather than the kernel backend, so that a blank field meant
|
||||
1280 on this device. Only the upstream default was read.
|
||||
- That TCP MSS behaves as described, or that large UDP replies are actually lost. No packet capture
|
||||
was taken.
|
||||
- That the Gateway VIP `.20` sees `10.90.0.x` / `world`. Conntrack held no tunnel flows to it when
|
||||
checked.
|
||||
- Android's IPv6 blocking on a real device; read from source only.
|
||||
- `wg-quick` IPv6 behaviour on Linux, and any non-Android client at 1280.
|
||||
- `/proc/net/nf_conntrack` output during a key mismatch. The `[UNREPLIED]` reading comes from the
|
||||
earlier measurement in the brief; I saw only healthy entries.
|
||||
@@ -53,6 +53,7 @@
|
||||
# WG_ENDPOINT_HOST default wan.home.keithjackson.dev
|
||||
# WG_CLIENT_CIDR default 10.90.0.0/24 (must match clusters/base/cluster-vars.yaml)
|
||||
# WG_LAN_CIDR default 192.168.2.0/24 (prod LAN_CIDR)
|
||||
# WG_MTU default 1280 (client tunnel MTU; see the comment on the default below)
|
||||
# WG_DNS default 192.168.2.23 (prod PIHOLE_DNS_VIP)
|
||||
#
|
||||
# REQUIRES: `wg`, `qrencode`, `sops` — all three in the dev container.
|
||||
@@ -80,7 +81,14 @@ PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
|
||||
# cannot resolve until it is already connected.
|
||||
: "${WG_ENDPOINT_HOST:=wan.home.keithjackson.dev}"
|
||||
: "${WG_LISTEN_PORT:=51820}"
|
||||
: "${WG_MTU:=1420}"
|
||||
# The CLIENT's tunnel MTU. Deliberately lower than the server's 1420 (daemonset.yaml), which only
|
||||
# describes the 1500-byte path at the node's end. A phone's path is the carrier's, often behind
|
||||
# carrier NAT, and is not known to carry 1500. Measured 2026-11-09, one Android phone on one
|
||||
# carrier: at 1420 the handshake was healthy but names and pages failed; clearing the app's MTU
|
||||
# field (its userspace backend then uses 1280) fixed it. 1280 is also the IPv6 minimum link MTU, so
|
||||
# it stays valid if the tunnel ever gains IPv6. For a device only ever on a wired or known-1500
|
||||
# path, WG_MTU=1420 recovers about 10% of per-packet payload.
|
||||
: "${WG_MTU:=1280}"
|
||||
|
||||
SPLIT=0
|
||||
OUT_DIR=""
|
||||
|
||||
@@ -112,7 +112,9 @@ no change. Three further problems vanish with the host netns:
|
||||
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
|
||||
`externalTrafficPolicy` trade-off to make at all.
|
||||
* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
|
||||
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct.
|
||||
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` correct on the
|
||||
server's `wg0`. It says nothing about a client's own path: `new-peer.sh` defaults client profiles
|
||||
to `1280` (see "Troubleshooting" in `runbooks/vpn-remote-access.md`).
|
||||
|
||||
`hostNetwork` is *permitted* here, and not by oversight:
|
||||
`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
|
||||
|
||||
@@ -30,9 +30,12 @@
|
||||
# `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either.
|
||||
# The VPN would have looked up and been useless, which is the worst failure shape available.
|
||||
#
|
||||
# In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the
|
||||
# NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
|
||||
# exactly what a LAN client looks like. Three further problems disappear with it:
|
||||
# In the host netns there is no pod identity to carry, so a client never looks like a pod. What a
|
||||
# destination sees depends on how the packet is handled. LAN hosts and the internet: forwarded,
|
||||
# then MASQUERADEd to the NODE's address by the PostUp rule below. LoadBalancer VIPs: Cilium's load
|
||||
# balancer takes the packet as it enters from wg0, before that rule, so the source is still
|
||||
# 10.90.0.x with identity `world`, like a LAN browser (measured 2026-11-09 for PiHole .23 and Plex
|
||||
# .24; see README). Three further problems disappear with it:
|
||||
# * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to
|
||||
# /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
|
||||
# is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
|
||||
|
||||
@@ -252,7 +252,7 @@ cellular, with Wi-Fi off, and check all four — each proves a different hop:
|
||||
|---|---|---|
|
||||
| 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza |
|
||||
| 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path |
|
||||
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd |
|
||||
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway admitting a tunnel client, which a pod-networked VPN would have 403'd. Expected source identity is `world`, like a LAN browser: measured 2026-11-09 on VIPs `.23` and `.24`, not yet observed on the Gateway's `.20` |
|
||||
| 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier |
|
||||
|
||||
Then confirm server-side:
|
||||
@@ -260,18 +260,97 @@ Then confirm server-side:
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
|
||||
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer.
|
||||
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer. `ds/wireguard`
|
||||
# picks ONE pod; if it reads 0, ask the pod on the node holding the forward (see Troubleshooting).
|
||||
```
|
||||
|
||||
**If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not
|
||||
WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up.
|
||||
|
||||
**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60
|
||||
bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
|
||||
**If 1 and 2 pass but pages or large transfers hang**, suspect the client MTU; see
|
||||
Troubleshooting.
|
||||
|
||||
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside —
|
||||
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
|
||||
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
|
||||
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from the
|
||||
device, by design. `wg show` cannot tell them apart either: it reads the same in both cases. The
|
||||
node's conntrack table can; see Troubleshooting.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
Ask the pod on **the node carrying the traffic**: the node the WAN forward names for `away`,
|
||||
`k3s-prod-1` for `home`. Sessions are per node, and the other two pods read `0`.
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress get pods -o wide'
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- wg show wg0 latest-handshakes'
|
||||
# likewise: `wg show wg0 endpoints`, `wg show wg0 transfer`
|
||||
```
|
||||
|
||||
**Use only those three subcommands to diagnose.** Never `wg showconf`, `wg show wg0 private-key` or
|
||||
`wg show wg0 preshared-keys`: they print key material into your terminal and transcript.
|
||||
|
||||
| output | reading |
|
||||
|---|---|
|
||||
| `latest-handshakes`: `<public key> 0` | no handshake with this peer since the pod started |
|
||||
| `endpoints`: `<public key> (none)` | the server has not accepted a packet from this peer since the pod started |
|
||||
| `transfer`: `<public key> <rx> <tx>` | bytes, as the server counts them |
|
||||
|
||||
### The app shows TX, no RX and no "Latest handshake"
|
||||
|
||||
**Symptom.** The app's transmit counter climbs and receive stays at zero. On the server,
|
||||
`endpoints` is `(none)` and `latest-handshakes` is `0` for that peer.
|
||||
|
||||
**Cause.** A key mismatch: the server's public key in the profile, or the device's public key in
|
||||
`peers.conf`. WireGuard drops an initiation it cannot authenticate and sends nothing back, so
|
||||
neither end logs an error. Measured 2026-11-09: a mistyped server public key in the phone's profile.
|
||||
|
||||
**Check.** First prove the packets arrive. The pod has no `conntrack` binary, but in `hostNetwork`
|
||||
it can read the host's table:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- grep dport=51820 /proc/net/nf_conntrack'
|
||||
```
|
||||
|
||||
- A line whose first tuple is `src=<the phone's public address> ... dport=51820`, marked
|
||||
`[UNREPLIED]`: the datagrams reach the node and the server stays silent. It is the keys.
|
||||
- No such line: the packets are not arriving. Look at the router forward, the DNS name, or the
|
||||
device's network.
|
||||
|
||||
Then compare the keys **by copy and paste, never by eye and never retyped**: the server's public
|
||||
half (the first command in Step 2's "Record it in your PASSWORD MANAGER") against the profile's
|
||||
`PublicKey`, and the device's public key from the app against its stanza in `peers.conf`.
|
||||
|
||||
### Works on Wi-Fi, stalls on mobile data, handshake healthy
|
||||
|
||||
**Symptom.** `latest-handshakes` is recent and some traffic flows, but names or pages fail on
|
||||
mobile data. The same device is fine on Wi-Fi.
|
||||
|
||||
**Cause.** Most likely a client MTU too large for the carrier's path: a tunnel MTU of `1420` means
|
||||
outer packets of up to 1480 bytes. This is inferred from one before-and-after, on one phone and one
|
||||
carrier (2026-11-09): `1420` failed and clearing the field fixed it. The path MTU itself was not
|
||||
measured.
|
||||
|
||||
**Check.** In the Android WireGuard app, edit the tunnel, clear the `MTU` field or set it to `1280`,
|
||||
save, then turn the tunnel off and on. A blank field means `1280` in the app's default userspace
|
||||
backend (`GoBackend`: `getMtu().orElse(1280)`). On other clients set `MTU = 1280` in the profile.
|
||||
|
||||
`new-peer.sh` now generates `MTU = 1280` unless `WG_MTU` overrides it. A profile already on a device
|
||||
does not change, so edit it by hand. If `1280` does not help, it is not the MTU: go back to
|
||||
Step 6's check 3.
|
||||
|
||||
### The `away` profile cannot connect from home
|
||||
|
||||
**Symptom.** On the home network, `away` never handshakes.
|
||||
|
||||
**Cause.** Its Endpoint is `wan.home.keithjackson.dev`, the house's WAN address, and the router has
|
||||
no hairpin NAT. WireGuard cannot use the Cloudflare Tunnel instead.
|
||||
|
||||
**Check.** Turn Wi-Fi off. If `away` handshakes on mobile data, the profile is fine: use the `home`
|
||||
tunnel indoors, as described in
|
||||
[Phone profiles: one tunnel for home, one for away](../platform/vpn-ingress/README.md#phone-profiles-one-tunnel-for-home-one-for-away).
|
||||
|
||||
## Revoking a device
|
||||
|
||||
@@ -293,10 +372,17 @@ top follow-up for this component.** Until then:
|
||||
|
||||
## Known limitations
|
||||
|
||||
* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside
|
||||
the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point.
|
||||
* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's
|
||||
address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
|
||||
preserve them — at the cost of another hand-maintained off-repo route.
|
||||
* **No IPv6 in the tunnel.** What a client's IPv6 does then depends on the client. In the Android
|
||||
app's default `GoBackend`, a config with one peer and a default route skips `allowFamily()`, so
|
||||
Android should block IPv6 while the tunnel is up and apps fall back to IPv4. That is read from
|
||||
upstream source, not observed on a device. A `--split` profile has no default route, so its IPv6
|
||||
stays outside the tunnel. Expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside too
|
||||
(not tested).
|
||||
* **Client IPs are only partly preserved.** LAN hosts and the internet see the node's address
|
||||
(MASQUERADE). A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
|
||||
preserve them there, at the cost of another hand-maintained off-repo route. LoadBalancer VIPs are
|
||||
different: Cilium handles the packet before MASQUERADE, so the source stays `10.90.0.x`, identity
|
||||
`world` (BPF conntrack on prod-1, 2026-11-09, VIPs `.23` and `.24`). Any `loadBalancerSourceRanges`
|
||||
must therefore list the tunnel subnet, as PiHole's do.
|
||||
* **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients
|
||||
re-handshake automatically (retry every 5s, rekey every 2 min).
|
||||
Reference in new issue
Block a user