diff --git a/decisions/2026-10-06-vpn-ingress-ip-forward.md b/decisions/2026-10-06-vpn-ingress-ip-forward.md new file mode 100644 index 0000000..32ba577 --- /dev/null +++ b/decisions/2026-10-06-vpn-ingress-ip-forward.md @@ -0,0 +1,113 @@ +--- +date: 2026-10-06 +branch: fix/vpn-ingress-ip-forward +title: The VPN tunnel never came up — a hostNetwork pod cannot write net.ipv4.ip_forward, and wg-quick treats a failed PostUp as fatal +--- + +## What changed + +The remote-access VPN reconciled for the first time with a real server key (#60 then #62) and +**the tunnel did not come up**. One line in my own manifest was the whole cause. + +Measured in the `wireguard` container on prod, 2026-10-06: + +``` +[#] ip link add dev wg0 type wireguard +[#] wg addconf wg0 /dev/fd/63 +[#] ip -4 address add 10.90.0.1/24 dev wg0 +[#] ip link set mtu 1420 up dev wg0 +[#] sysctl -q -w net.ipv4.ip_forward=1 +sysctl: permission denied on key "net.ipv4.ip_forward" +[#] ip link delete dev wg0 +**** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! **** +``` + +Everything up to that point worked. **The cause is not a missing capability.** With +`hostNetwork: true` the runtime mounts `/proc/sys` **read-only** in the container — confirmed in the +pod's own `/proc/mounts`: `proc /proc/sys proc ro,nosuid,nodev,noexec,relatime` — precisely because +those sysctls are the *host's* globals and a container must not rewrite them. `CAP_NET_ADMIN` cannot +lift a read-only mount, so this can never succeed from here. And `wg-quick` treats any failed +`PostUp` as fatal, so it deleted the interface it had just finished building. + +**The line was never needed.** `node_baseline` writes `net.ipv4.ip_forward=1` to +`/etc/sysctl.d/99-k3s-cilium.conf`, and reading the value from inside the pod returns `1`. I had +written the header comment *correctly* — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl +is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so +this component does not silently depend on another role's sysctl file surviving". That defensive +re-assertion is the entire outage. **The lesson is narrow and reusable: in a hostNetwork pod, +verify host state, never write it.** + +Fix: + +- **`platform/vpn-ingress/daemonset.yaml`**: the `PostUp = sysctl …` line is gone, replaced by a + **read** in the init container — `grep -qx 1 /proc/sys/net/ipv4/ip_forward` — which fails closed + with a message naming the `node_baseline` role. Reading `/proc/sys` is permitted; only writing is + not. Fail-closed still matters: a tunnel that comes up but cannot forward would satisfy every + probe in this component and silently drop every client's traffic. +- The header bullet that said "no pod sysctl is needed" now also says **do not attempt one**, with + the mechanism, so the next author does not re-add it for the same reason I did. +- The `securityContext` comment claimed NET_ADMIN was for "create the wg0 link, set the host sysctl, + and write the netfilter rules". The middle clause was false; corrected. +- The image's `chown: … Operation not permitted` startup run — plus its own "Permissions could not + be set … we will not provide support for it" — is now documented as **expected and harmless**, in + both the manifest and the runbook, so nobody chases it. Capabilities are dropped to `NET_ADMIN` + only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root, + and `wg-quick` proceeds past it (as it did here — the failure was 30 lines later). +- **`runbooks/vpn-remote-access.md`**: a "what a healthy start looks like" block with the real log + lines; `ip_forward is 0` added to the init-failure table; and the note that `0/1 Running` with + `Startup probe failed: Unable to access interface: No such device` is the signature of `wg-quick` + tearing the tunnel down — read the `[#]` line it died on. +- **`platform/vpn-ingress/README.md`**: a table of what the first reconcile settled (below). + +No sensitive paths: `platform/vpn-ingress/**`, `runbooks/**` and `decisions/**` only. This merges on +a green gate with no override. + +## What I verified + +Live prod, read-only, 2026-10-06. **Four questions this component's design rested on are now +measured rather than assumed — three went the right way, one was the defect above.** + +- **The GitOps path is proven end to end.** `Kustomization vpn-ingress` `lastAppliedRevision: + main@sha1:cf7d5e4` equals the GitRepository artifact revision. The Secret carries + `kustomize.toolkit.fluxcd.io/name: vpn-ingress`, so Flux delivered it. +- **SOPS decryption works, proven by digest rather than by eye.** The value in + `platform/vpn-ingress/wg-server-key.sops.yaml` on `main`, decrypted, is 44 bytes, is **not** the + placeholder, and **`wg pubkey` accepts it**. Its SHA-256 equals the SHA-256 of the live Secret's + decoded `data.privatekey` (`d7c8f8de6f252649…` both sides). No key material was printed at any + point — lengths, booleans and digests only. +- **The `wg pubkey` guard fired correctly, and its `ENC[` branch was never needed.** During the + kubelet window before the updated Secret propagated, all three pods reported *"It is still the + committed REPLACE-ME PLACEHOLDER"* — the guard distinguishing a stale/unfilled value from + ciphertext, exactly as designed. It cleared on its own once the mount refreshed; **a transient + `REPLACE-ME` CrashLoop for a minute or two after the key PR merges is expected, not a fault.** +- **The kernel module reaches the pod**: `ip link add dev wg0 type wireguard` succeeded and + `/proc/modules` lists `wireguard`. So the `node_baseline` addition works and `CAP_SYS_MODULE` is + correctly not required. +- **`allowPrivilegeEscalation: false` does NOT break the image's s6 init** — the container reaches + `wg-quick` normally. The flag stays; the fear recorded in #55 is closed. +- **The iptables backends MATCH, which was my single biggest documented doubt.** The image ships + `iptables v1.8.13 (nf_tables)` via `/usr/sbin/xtables-nft-multi`, and Debian 13 is nf_tables. The + nightmare case — rules accepted into a table the kernel never consults, so handshakes succeed and + every forwarded connection hangs — cannot occur on this pairing. +- **`net.ipv4.ip_forward` is `1`** in the host netns the pod shares, and `/proc/sys` is `ro` there. + Both read directly from the pod. +- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`, + `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two + environmental skips (`gitea-settings`, conformance L2/L3). **Not green. Reported as rc 3.** + +## What I did NOT verify + +- **That the tunnel now comes up.** This fix has not reconciled. The next reconcile is the test, and + its success criterion is specific: pods `1/1 Running`, and `wg show` reporting `interface: wg0` + with `listening port: 51820`. +- **That the MASQUERADE and FORWARD rules actually apply.** They are the next statements after the + one that failed, so **they have never executed**. The backends matching (above) removes the worst + failure mode but proves nothing about the rules themselves. First check after the pods go Ready: + `iptables -t nat -S POSTROUTING | grep 10.90.0.0/24` on a node. +- **Anything about client traffic.** Still zero peers enrolled, still no WAN port-forward, still no + off-LAN vantage point. The VPN has carried no packet. +- **Whether the transient `REPLACE-ME` window has an upper bound.** It cleared within ~2 minutes + here; I did not determine the kubelet refresh behaviour, so a longer stall after a key change is + uncharacterised. +- **The `sops set` hand-off command**, still. The operator filled the key in by their own route + (#62), so the exact quoting I documented inside `dev.sh run bash -c` remains unexercised. diff --git a/platform/vpn-ingress/README.md b/platform/vpn-ingress/README.md index 36a664d..6ed96e0 100644 --- a/platform/vpn-ingress/README.md +++ b/platform/vpn-ingress/README.md @@ -173,6 +173,18 @@ NetworkPolicy applies. (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key. * **Full bring-up and verification:** `runbooks/vpn-remote-access.md`. +## Measured on prod, 2026-10-06 — what is now known rather than assumed + +The first real reconcile settled four things this component's design rested on. Three went the +right way; one was a defect in my own manifest. + +| question | answer | +|---|---| +| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. | +| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. | +| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. | +| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. | + ## Prerequisites outside this directory 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules` diff --git a/platform/vpn-ingress/daemonset.yaml b/platform/vpn-ingress/daemonset.yaml index 801b4ef..fb6aeb6 100644 --- a/platform/vpn-ingress/daemonset.yaml +++ b/platform/vpn-ingress/daemonset.yaml @@ -37,6 +37,10 @@ # /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward` # is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have # required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change). +# COROLLARY LEARNED THE HARD WAY (prod, 2026-10-06): this pod must not merely avoid NEEDING +# the sysctl, it must not ATTEMPT it. hostNetwork implies /proc/sys is mounted read-only, so +# `sysctl -w` fails with EPERM regardless of capabilities, and wg-quick treats a failed PostUp +# as fatal and deletes the interface. Verify the value; never write it. # * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost # this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all. # * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a @@ -171,6 +175,19 @@ spec: exit 1 fi + # The pod shares the HOST netns, so this reads the node's real setting. Reading + # /proc/sys is permitted even though writing is not (see the PostUp note below). + # Fail closed: a tunnel that comes up but cannot forward would look healthy to every + # probe here and simply drop every client's traffic. + if ! grep -qx 1 /proc/sys/net/ipv4/ip_forward; then + echo "FATAL: net.ipv4.ip_forward is 0 on this node, so forwarded VPN traffic would" >&2 + echo " be dropped. This pod CANNOT set it: with hostNetwork the runtime mounts" >&2 + echo " /proc/sys read-only, because it is the host's global setting." >&2 + echo " Fix on the NODE: run the node_baseline role (it writes" >&2 + echo " /etc/sysctl.d/99-k3s-cilium.conf) — runbooks/vpn-remote-access.md step 1." >&2 + exit 1 + fi + { echo '[Interface]' echo "Address = $base.1/$mask" @@ -184,9 +201,20 @@ spec: echo 'MTU = 1420' printf 'PrivateKey = %s\n' "$key" # ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ── - # ip_forward is already 1 from node_baseline; re-asserted so this component does not - # silently depend on another role's sysctl file surviving. - echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1' + # NO `sysctl -w net.ipv4.ip_forward=1` HERE. It is IMPOSSIBLE from this pod and it was + # the only thing stopping the tunnel coming up — MEASURED on prod 2026-10-06: + # [#] sysctl -q -w net.ipv4.ip_forward=1 + # sysctl: permission denied on key "net.ipv4.ip_forward" + # [#] ip link delete dev wg0 + # and wg-quick then tore the whole tunnel down. The cause is not a missing + # capability: with `hostNetwork: true` the runtime mounts /proc/sys READ-ONLY in the + # container (confirmed in /proc/mounts: `proc /proc/sys proc ro,...`), precisely + # because those sysctls are the HOST's globals and a container must not rewrite + # them. CAP_NET_ADMIN does not and cannot lift a read-only mount. + # It was never needed anyway: node_baseline writes net.ipv4.ip_forward=1 to + # /etc/sysctl.d/99-k3s-cilium.conf, and reading it from this pod returns 1. The + # init container now VERIFIES that (a read, which /proc/sys does allow) and fails + # closed with a pointed message, which is what the write was pretending to achieve. # SNAT client traffic to the node's address. REQUIRED: without it a LAN host would # reply to 10.90.0.x, find no route, hand the packet to the router, and the router # would drop it — so every connection would hang rather than fail. Matching on @@ -246,8 +274,10 @@ spec: # `kubectl get pod -o wide` shows what this listener owns. - { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP } securityContext: - # NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and - # write the two netfilter rules above. Everything else is dropped, and the container is + # NET_ADMIN is the whole privilege budget: create the wg0 link and write the netfilter + # rules above. (It does NOT let the pod set a host sysctl — /proc/sys is read-only under + # hostNetwork — which is why nothing here tries.) Everything else is dropped, and the + # container is # NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule. # SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node # (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container @@ -261,6 +291,16 @@ spec: - NET_ADMIN # readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and # /var during startup. Same deferral as crafty/headlamp. + # + # EXPECTED, HARMLESS STARTUP NOISE — do not "fix" it. Because capabilities are dropped + # to NET_ADMIN only, the image's s6 init cannot chown /config (no CAP_CHOWN) and logs a + # run of `chown: ... Operation not permitted` plus "Permissions could not be set. This + # is probably because your volume mounts are remote or read-only" and a scary "we will + # not provide support for it". It is wrong about the cause and it does not matter: + # /config is a writable tmpfs emptyDir, everything here runs as root (uid 0) anyway, and + # wg-quick proceeds normally — verified on prod 2026-10-06, where the tunnel's only + # failure was the sysctl above, well past these warnings. Granting CAP_CHOWN to silence + # log noise would widen the privilege set of an internet-facing pod for nothing. resources: requests: cpu: 25m diff --git a/runbooks/vpn-remote-access.md b/runbooks/vpn-remote-access.md index 17c95fa..e280402 100644 --- a/runbooks/vpn-remote-access.md +++ b/runbooks/vpn-remote-access.md @@ -134,6 +134,36 @@ specific failure in its log — so you never get a server that silently cannot c | `It is EMPTY` | the value was blanked | | `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green | | `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake | +| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting | + +### What a healthy start looks like + +Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key: + +``` +wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820. +``` + +then the `wireguard` container brings the tunnel up: + +``` +**** Found WG conf /config/wg_confs/wg0.conf, adding to list **** +**** Activating tunnel /config/wg_confs/wg0.conf **** +[#] ip link add dev wg0 type wireguard +[#] wg addconf wg0 /dev/fd/63 +[#] ip -4 address add 10.90.0.1/24 dev wg0 +[#] ip link set mtu 1420 up dev wg0 +``` + +**Ignore the `chown: ... Operation not permitted` run before that**, and the image's own +"Permissions could not be set ... we will not provide support for it". Capabilities are dropped to +`NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs +as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter. + +**A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device` +means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it +died on. That is the shape of a real failure; the probe is doing its job of refusing to call a +server without a `wg0` "ready". ## Step 3 — forward udp/51820 on the router (once)