2026-11-09 03:26:02 +00:00
5 changed files with 190 additions and 17 deletions
@@ -0,0 +1,74 @@
+---
+date: 2026-11-08
+branch: fix/wg-client-mtu-docs
+title: WireGuard client profiles default to MTU 1280; VPN runbook corrected and given a troubleshooting section
+---
+
+## What changed
+
+- **`infra/vpn-ingress/new-peer.sh`**: `WG_MTU` default 1420 -> 1280, still overridable, reason in a
+  comment. Only new profiles change; the phone's existing profiles are not touched.
+- **`runbooks/vpn-remote-access.md`**: the verification table no longer says the Gateway sees a
+  `host`-identity source; the limitations no longer say everything arrives SNATed or that IPv6 leaks
+  on Android; the "if nothing handshakes" advice no longer claims `wg show` separates a bad forward
+  from a bad key. New "Troubleshooting" section covering the four cases in the brief.
+- **`platform/vpn-ingress/daemonset.yaml`**: header comment only, SNAT claim corrected.
+- **`platform/vpn-ingress/README.md`**: one sentence, so `MTU = 1420` reads as the server's value.
+
+Rollout on merge: the WireGuard DaemonSet does **not** roll. The `runbooks` ConfigMap is content
+hashed and `vpn-remote-access.md` is in it, so the runbooks site rolls.
+
+Left unchanged, for the operator:
+
+- **`platform/vpn-ingress/peers.conf`** still says "No peers enrolled yet". A comment-only edit on a
+  throwaway copy moved the ConfigMap from `wg-peers-29mf6k867t` to `wg-peers-gggkk5d4tm` and changed
+  the DaemonSet, so correcting it restarts every `wg0`. Fix it alongside the next peer change.
+- **`clusters/base/cluster-vars.yaml`** coverage comment still omits `WG_CLIENT_CIDR` (sensitive path).
+- **Server `wg0` MTU stays 1420**, because changing it restarts all WireGuard pods. For TCP a 1280
+  client should be enough on its own: it advertises an MSS of 1240, which caps what the far end
+  sends. That protection does not reach UDP. A server-to-phone datagram with an inner size of
+  1281-1420 bytes is still sent as up to 1480 outer bytes and could be lost on the same path; QUIC
+  should recover by probing, large DNS answers are the likeliest casualty. If that shows up, the
+  fix is a lower server MTU, as a deliberate rollout.
+- **Other stale text, outside this brief:** `platform/vpn-ingress/README.md` still says no peer has
+  ever completed a handshake, and the runbook's status line still calls the VPN inert.
+
+## What I verified
+
+- Upstream `wireguard-android` `GoBackend.java`, master (last commit to that file `ecc3c0a`,
+  2026-03-15): `builder.setMtu(config.getInterface().getMtu().orElse(1280))`, and `allowFamily()` is
+  skipped when `sawDefaultRoute && config.getPeers().size() == 1`.
+- Live, read-only, 2026-11-09 UTC, on the `k3s-prod-1` pod and agent:
+  - `wg0` MTU is 1420 on all three pods. Both peers have handshakes and transfer on prod-1 only; the
+    other two pods read `0`.
+  - BPF conntrack holds `UDP SVC 10.90.0.x -> 192.168.2.23:53` and flows to `192.168.2.24:32400`.
+    Their backend entries carry `SourceSecurityID=2` (`world`). So the source stays `10.90.0.x`
+    through the load balancer on those two VIPs.
+  - `/proc/net/nf_conntrack` is readable from the pod, which has no `conntrack` binary.
+- Offline, pinned toolbox: `kubectl kustomize platform/vpn-ingress` gives identical ConfigMap and
+  DaemonSet hashes on `origin/main` and on this branch.
+- `new-peer.sh` writes `MTU = 1280` by default and `MTU = 1420` with `WG_MTU=1420`. Run with
+  throwaway keys and stdout discarded; profiles shredded; nothing enrolled.
+- No test or conformance check pins the client MTU (`grep` for `WG_MTU` and `1420`).
+- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not green.**
+  `gitea-settings` is the only stage counted as not run (no token locally). `conformance` (L2/L3)
+  also skipped as the declared no-cluster gap, so R-ENV-01 and R-OBS-04 are not certified.
+  `spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1`
+  passed. No sensitive path is touched.
+
+## What I did NOT verify
+
+- **That MTU was the cause.** It is inferred from one before-and-after, on one phone and one carrier,
+  reported by the operator. The path MTU was never measured.
+- **That 1280 is the best value.** It is the value that worked, and the IPv6 minimum. A higher one
+  may work too.
+- That the phone's app is on `GoBackend` rather than the kernel backend, so that a blank field meant
+  1280 on this device. Only the upstream default was read.
+- That TCP MSS behaves as described, or that large UDP replies are actually lost. No packet capture
+  was taken.
+- That the Gateway VIP `.20` sees `10.90.0.x` / `world`. Conntrack held no tunnel flows to it when
+  checked.
+- Android's IPv6 blocking on a real device; read from source only.
+- `wg-quick` IPv6 behaviour on Linux, and any non-Android client at 1280.
+- `/proc/net/nf_conntrack` output during a key mismatch. The `[UNREPLIED]` reading comes from the
+  earlier measurement in the brief; I saw only healthy entries.
@@ -53,6 +53,7 @@
 #   WG_ENDPOINT_HOST    default wan.home.keithjackson.dev
 #   WG_CLIENT_CIDR      default 10.90.0.0/24   (must match clusters/base/cluster-vars.yaml)
 #   WG_LAN_CIDR         default 192.168.2.0/24 (prod LAN_CIDR)
+#   WG_MTU              default 1280 (client tunnel MTU; see the comment on the default below)
 #   WG_DNS              default 192.168.2.23   (prod PIHOLE_DNS_VIP)
 #
 # REQUIRES: `wg`, `qrencode`, `sops` — all three in the dev container.
@@ -80,7 +81,14 @@ PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
 # cannot resolve until it is already connected.
 : "${WG_ENDPOINT_HOST:=wan.home.keithjackson.dev}"
 : "${WG_LISTEN_PORT:=51820}"
-: "${WG_MTU:=1420}"
+# The CLIENT's tunnel MTU. Deliberately lower than the server's 1420 (daemonset.yaml), which only
+# describes the 1500-byte path at the node's end. A phone's path is the carrier's, often behind
+# carrier NAT, and is not known to carry 1500. Measured 2026-11-09, one Android phone on one
+# carrier: at 1420 the handshake was healthy but names and pages failed; clearing the app's MTU
+# field (its userspace backend then uses 1280) fixed it. 1280 is also the IPv6 minimum link MTU, so
+# it stays valid if the tunnel ever gains IPv6. For a device only ever on a wired or known-1500
+# path, WG_MTU=1420 recovers about 10% of per-packet payload.
+: "${WG_MTU:=1280}"

 SPLIT=0
 OUT_DIR=""
@@ -112,7 +112,9 @@ no change. Three further problems vanish with the host netns:
  `conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
  `externalTrafficPolicy` trade-off to make at all.
 * **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
-  1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct.
+  1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` correct on the
+  server's `wg0`. It says nothing about a client's own path: `new-peer.sh` defaults client profiles
+  to `1280` (see "Troubleshooting" in `runbooks/vpn-remote-access.md`).

 `hostNetwork` is *permitted* here, and not by oversight:
 `platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
@@ -30,9 +30,12 @@
 # `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either.
 # The VPN would have looked up and been useless, which is the worst failure shape available.
 #
-# In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the
-# NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
-# exactly what a LAN client looks like. Three further problems disappear with it:
+# In the host netns there is no pod identity to carry, so a client never looks like a pod. What a
+# destination sees depends on how the packet is handled. LAN hosts and the internet: forwarded,
+# then MASQUERADEd to the NODE's address by the PostUp rule below. LoadBalancer VIPs: Cilium's load
+# balancer takes the packet as it enters from wg0, before that rule, so the source is still
+# 10.90.0.x with identity `world`, like a LAN browser (measured 2026-11-09 for PiHole .23 and Plex
+# .24; see README). Three further problems disappear with it:
 #   * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to
 #     /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
 #     is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
@@ -252,7 +252,7 @@ cellular, with Wi-Fi off, and check all four — each proves a different hop:
 |---|---|---|
 | 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza |
 | 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path |
-| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd |
+| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway admitting a tunnel client, which a pod-networked VPN would have 403'd. Expected source identity is `world`, like a LAN browser: measured 2026-11-09 on VIPs `.23` and `.24`, not yet observed on the Gateway's `.20` |
 | 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier |

 Then confirm server-side:
@@ -260,18 +260,97 @@ Then confirm server-side:
 ```bash
 ./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
-# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer.
+# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer. `ds/wireguard`
+# picks ONE pod; if it reads 0, ask the pod on the node holding the forward (see Troubleshooting).
 ```

 **If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not
 WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up.

-**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60
-bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
+**If 1 and 2 pass but pages or large transfers hang**, suspect the client MTU; see
+Troubleshooting.

-**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside —
-by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
-if the router is not delivering the packets, and a `handshake` with no completed session if it is.
+**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from the
+device, by design. `wg show` cannot tell them apart either: it reads the same in both cases. The
+node's conntrack table can; see Troubleshooting.
+
+## Troubleshooting
+
+Ask the pod on **the node carrying the traffic**: the node the WAN forward names for `away`,
+`k3s-prod-1` for `home`. Sessions are per node, and the other two pods read `0`.
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress get pods -o wide'
+
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- wg show wg0 latest-handshakes'
+# likewise: `wg show wg0 endpoints`, `wg show wg0 transfer`
+```
+
+**Use only those three subcommands to diagnose.** Never `wg showconf`, `wg show wg0 private-key` or
+`wg show wg0 preshared-keys`: they print key material into your terminal and transcript.
+
+| output | reading |
+|---|---|
+| `latest-handshakes`: `<public key>  0` | no handshake with this peer since the pod started |
+| `endpoints`: `<public key>  (none)` | the server has not accepted a packet from this peer since the pod started |
+| `transfer`: `<public key>  <rx>  <tx>` | bytes, as the server counts them |
+
+### The app shows TX, no RX and no "Latest handshake"
+
+**Symptom.** The app's transmit counter climbs and receive stays at zero. On the server,
+`endpoints` is `(none)` and `latest-handshakes` is `0` for that peer.
+
+**Cause.** A key mismatch: the server's public key in the profile, or the device's public key in
+`peers.conf`. WireGuard drops an initiation it cannot authenticate and sends nothing back, so
+neither end logs an error. Measured 2026-11-09: a mistyped server public key in the phone's profile.
+
+**Check.** First prove the packets arrive. The pod has no `conntrack` binary, but in `hostNetwork`
+it can read the host's table:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- grep dport=51820 /proc/net/nf_conntrack'
+```
+
+- A line whose first tuple is `src=<the phone's public address> ... dport=51820`, marked
+  `[UNREPLIED]`: the datagrams reach the node and the server stays silent. It is the keys.
+- No such line: the packets are not arriving. Look at the router forward, the DNS name, or the
+  device's network.
+
+Then compare the keys **by copy and paste, never by eye and never retyped**: the server's public
+half (the first command in Step 2's "Record it in your PASSWORD MANAGER") against the profile's
+`PublicKey`, and the device's public key from the app against its stanza in `peers.conf`.
+
+### Works on Wi-Fi, stalls on mobile data, handshake healthy
+
+**Symptom.** `latest-handshakes` is recent and some traffic flows, but names or pages fail on
+mobile data. The same device is fine on Wi-Fi.
+
+**Cause.** Most likely a client MTU too large for the carrier's path: a tunnel MTU of `1420` means
+outer packets of up to 1480 bytes. This is inferred from one before-and-after, on one phone and one
+carrier (2026-11-09): `1420` failed and clearing the field fixed it. The path MTU itself was not
+measured.
+
+**Check.** In the Android WireGuard app, edit the tunnel, clear the `MTU` field or set it to `1280`,
+save, then turn the tunnel off and on. A blank field means `1280` in the app's default userspace
+backend (`GoBackend`: `getMtu().orElse(1280)`). On other clients set `MTU = 1280` in the profile.
+
+`new-peer.sh` now generates `MTU = 1280` unless `WG_MTU` overrides it. A profile already on a device
+does not change, so edit it by hand. If `1280` does not help, it is not the MTU: go back to
+Step 6's check 3.
+
+### The `away` profile cannot connect from home
+
+**Symptom.** On the home network, `away` never handshakes.
+
+**Cause.** Its Endpoint is `wan.home.keithjackson.dev`, the house's WAN address, and the router has
+no hairpin NAT. WireGuard cannot use the Cloudflare Tunnel instead.
+
+**Check.** Turn Wi-Fi off. If `away` handshakes on mobile data, the profile is fine: use the `home`
+tunnel indoors, as described in
+[Phone profiles: one tunnel for home, one for away](../platform/vpn-ingress/README.md#phone-profiles-one-tunnel-for-home-one-for-away).

 ## Revoking a device

@@ -293,10 +372,17 @@ top follow-up for this component.** Until then:

 ## Known limitations

-* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside
-  the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point.
-* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's
-  address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
-  preserve them — at the cost of another hand-maintained off-repo route.
+* **No IPv6 in the tunnel.** What a client's IPv6 does then depends on the client. In the Android
+  app's default `GoBackend`, a config with one peer and a default route skips `allowFamily()`, so
+  Android should block IPv6 while the tunnel is up and apps fall back to IPv4. That is read from
+  upstream source, not observed on a device. A `--split` profile has no default route, so its IPv6
+  stays outside the tunnel. Expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside too
+  (not tested).
+* **Client IPs are only partly preserved.** LAN hosts and the internet see the node's address
+  (MASQUERADE). A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
+  preserve them there, at the cost of another hand-maintained off-repo route. LoadBalancer VIPs are
+  different: Cilium handles the packet before MASQUERADE, so the source stays `10.90.0.x`, identity
+  `world` (BPF conntrack on prod-1, 2026-11-09, VIPs `.23` and `.24`). Any `loadBalancerSourceRanges`
+  must therefore list the tunnel subnet, as PiHole's do.
 * **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients
  re-handshake automatically (retry every 5s, rekey every 2 min).