--- date: 2026-10-06 branch: fix/vpn-ingress-ip-forward title: The VPN tunnel never came up — a hostNetwork pod cannot write net.ipv4.ip_forward, and wg-quick treats a failed PostUp as fatal --- ## What changed The remote-access VPN reconciled for the first time with a real server key (#60 then #62) and **the tunnel did not come up**. One line in my own manifest was the whole cause. Measured in the `wireguard` container on prod, 2026-10-06: ``` [#] ip link add dev wg0 type wireguard [#] wg addconf wg0 /dev/fd/63 [#] ip -4 address add 10.90.0.1/24 dev wg0 [#] ip link set mtu 1420 up dev wg0 [#] sysctl -q -w net.ipv4.ip_forward=1 sysctl: permission denied on key "net.ipv4.ip_forward" [#] ip link delete dev wg0 **** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! **** ``` Everything up to that point worked. **The cause is not a missing capability.** With `hostNetwork: true` the runtime mounts `/proc/sys` **read-only** in the container — confirmed in the pod's own `/proc/mounts`: `proc /proc/sys proc ro,nosuid,nodev,noexec,relatime` — precisely because those sysctls are the *host's* globals and a container must not rewrite them. `CAP_NET_ADMIN` cannot lift a read-only mount, so this can never succeed from here. And `wg-quick` treats any failed `PostUp` as fatal, so it deleted the interface it had just finished building. **The line was never needed.** `node_baseline` writes `net.ipv4.ip_forward=1` to `/etc/sysctl.d/99-k3s-cilium.conf`, and reading the value from inside the pod returns `1`. I had written the header comment *correctly* — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so this component does not silently depend on another role's sysctl file surviving". That defensive re-assertion is the entire outage. **The lesson is narrow and reusable: in a hostNetwork pod, verify host state, never write it.** Fix: - **`platform/vpn-ingress/daemonset.yaml`**: the `PostUp = sysctl …` line is gone, replaced by a **read** in the init container — `grep -qx 1 /proc/sys/net/ipv4/ip_forward` — which fails closed with a message naming the `node_baseline` role. Reading `/proc/sys` is permitted; only writing is not. Fail-closed still matters: a tunnel that comes up but cannot forward would satisfy every probe in this component and silently drop every client's traffic. - The header bullet that said "no pod sysctl is needed" now also says **do not attempt one**, with the mechanism, so the next author does not re-add it for the same reason I did. - The `securityContext` comment claimed NET_ADMIN was for "create the wg0 link, set the host sysctl, and write the netfilter rules". The middle clause was false; corrected. - The image's `chown: … Operation not permitted` startup run — plus its own "Permissions could not be set … we will not provide support for it" — is now documented as **expected and harmless**, in both the manifest and the runbook, so nobody chases it. Capabilities are dropped to `NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root, and `wg-quick` proceeds past it (as it did here — the failure was 30 lines later). - **`runbooks/vpn-remote-access.md`**: a "what a healthy start looks like" block with the real log lines; `ip_forward is 0` added to the init-failure table; and the note that `0/1 Running` with `Startup probe failed: Unable to access interface: No such device` is the signature of `wg-quick` tearing the tunnel down — read the `[#]` line it died on. - **`platform/vpn-ingress/README.md`**: a table of what the first reconcile settled (below). No sensitive paths: `platform/vpn-ingress/**`, `runbooks/**` and `decisions/**` only. This merges on a green gate with no override. ## What I verified Live prod, read-only, 2026-10-06. **Four questions this component's design rested on are now measured rather than assumed — three went the right way, one was the defect above.** - **The GitOps path is proven end to end.** `Kustomization vpn-ingress` `lastAppliedRevision: main@sha1:cf7d5e4` equals the GitRepository artifact revision. The Secret carries `kustomize.toolkit.fluxcd.io/name: vpn-ingress`, so Flux delivered it. - **SOPS decryption works, proven by digest rather than by eye.** The value in `platform/vpn-ingress/wg-server-key.sops.yaml` on `main`, decrypted, is 44 bytes, is **not** the placeholder, and **`wg pubkey` accepts it**. Its SHA-256 equals the SHA-256 of the live Secret's decoded `data.privatekey` (`d7c8f8de6f252649…` both sides). No key material was printed at any point — lengths, booleans and digests only. - **The `wg pubkey` guard fired correctly, and its `ENC[` branch was never needed.** During the kubelet window before the updated Secret propagated, all three pods reported *"It is still the committed REPLACE-ME PLACEHOLDER"* — the guard distinguishing a stale/unfilled value from ciphertext, exactly as designed. It cleared on its own once the mount refreshed; **a transient `REPLACE-ME` CrashLoop for a minute or two after the key PR merges is expected, not a fault.** - **The kernel module reaches the pod**: `ip link add dev wg0 type wireguard` succeeded and `/proc/modules` lists `wireguard`. So the `node_baseline` addition works and `CAP_SYS_MODULE` is correctly not required. - **`allowPrivilegeEscalation: false` does NOT break the image's s6 init** — the container reaches `wg-quick` normally. The flag stays; the fear recorded in #55 is closed. - **The iptables backends MATCH, which was my single biggest documented doubt.** The image ships `iptables v1.8.13 (nf_tables)` via `/usr/sbin/xtables-nft-multi`, and Debian 13 is nf_tables. The nightmare case — rules accepted into a table the kernel never consults, so handshakes succeed and every forwarded connection hangs — cannot occur on this pairing. - **`net.ipv4.ip_forward` is `1`** in the host netns the pod shares, and `/proc/sys` is `ro` there. Both read directly from the pod. - **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two environmental skips (`gitea-settings`, conformance L2/L3). **Not green. Reported as rc 3.** ## What I did NOT verify - **That the tunnel now comes up.** This fix has not reconciled. The next reconcile is the test, and its success criterion is specific: pods `1/1 Running`, and `wg show` reporting `interface: wg0` with `listening port: 51820`. - **That the MASQUERADE and FORWARD rules actually apply.** They are the next statements after the one that failed, so **they have never executed**. The backends matching (above) removes the worst failure mode but proves nothing about the rules themselves. First check after the pods go Ready: `iptables -t nat -S POSTROUTING | grep 10.90.0.0/24` on a node. - **Anything about client traffic.** Still zero peers enrolled, still no WAN port-forward, still no off-LAN vantage point. The VPN has carried no packet. - **Whether the transient `REPLACE-ME` window has an upper bound.** It cleared within ~2 minutes here; I did not determine the kubelet refresh behaviour, so a longer stall after a key change is uncharacterised. - **The `sops set` hand-off command**, still. The operator filled the key in by their own route (#62), so the exact quoting I documented inside `dev.sh run bash -c` remains unexercised.