home-network-k3s-impl/decisions/2026-10-06-vpn-ingress-ip-forward.md

date, branch, title

date	branch	title
2026-10-06	fix/vpn-ingress-ip-forward	The VPN tunnel never came up — a hostNetwork pod cannot write net.ipv4.ip_forward, and wg-quick treats a failed PostUp as fatal

What changed

The remote-access VPN reconciled for the first time with a real server key (#60 then #62) and the tunnel did not come up. One line in my own manifest was the whole cause.

Measured in the wireguard container on prod, 2026-10-06:

[#] ip link add dev wg0 type wireguard
[#] wg addconf wg0 /dev/fd/63
[#] ip -4 address add 10.90.0.1/24 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] sysctl -q -w net.ipv4.ip_forward=1
sysctl: permission denied on key "net.ipv4.ip_forward"
[#] ip link delete dev wg0
**** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! ****

Everything up to that point worked. The cause is not a missing capability. With hostNetwork: true the runtime mounts /proc/sys read-only in the container — confirmed in the pod's own /proc/mounts: proc /proc/sys proc ro,nosuid,nodev,noexec,relatime — precisely because those sysctls are the host's globals and a container must not rewrite them. CAP_NET_ADMIN cannot lift a read-only mount, so this can never succeed from here. And wg-quick treats any failed PostUp as fatal, so it deleted the interface it had just finished building.

The line was never needed. node_baseline writes net.ipv4.ip_forward=1 to /etc/sysctl.d/99-k3s-cilium.conf, and reading the value from inside the pod returns 1. I had written the header comment correctly — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so this component does not silently depend on another role's sysctl file surviving". That defensive re-assertion is the entire outage. The lesson is narrow and reusable: in a hostNetwork pod, verify host state, never write it.

Fix:

• platform/vpn-ingress/daemonset.yaml: the PostUp = sysctl … line is gone, replaced by a read in the init container — grep -qx 1 /proc/sys/net/ipv4/ip_forward — which fails closed with a message naming the node_baseline role. Reading /proc/sys is permitted; only writing is not. Fail-closed still matters: a tunnel that comes up but cannot forward would satisfy every probe in this component and silently drop every client's traffic.
• The header bullet that said "no pod sysctl is needed" now also says do not attempt one, with the mechanism, so the next author does not re-add it for the same reason I did.
• The securityContext comment claimed NET_ADMIN was for "create the wg0 link, set the host sysctl, and write the netfilter rules". The middle clause was false; corrected.
• The image's chown: … Operation not permitted startup run — plus its own "Permissions could not be set … we will not provide support for it" — is now documented as expected and harmless, in both the manifest and the runbook, so nobody chases it. Capabilities are dropped to NET_ADMIN only, so its s6 init has no CAP_CHOWN; /config is a writable tmpfs, everything runs as root, and wg-quick proceeds past it (as it did here — the failure was 30 lines later).
• runbooks/vpn-remote-access.md: a "what a healthy start looks like" block with the real log lines; ip_forward is 0 added to the init-failure table; and the note that 0/1 Running with Startup probe failed: Unable to access interface: No such device is the signature of wg-quick tearing the tunnel down — read the [#] line it died on.
• platform/vpn-ingress/README.md: a table of what the first reconcile settled (below).

No sensitive paths: platform/vpn-ingress/**, runbooks/** and decisions/** only. This merges on a green gate with no override.

What I verified

Live prod, read-only, 2026-10-06. Four questions this component's design rested on are now measured rather than assumed — three went the right way, one was the defect above.

• The GitOps path is proven end to end. Kustomization vpn-ingress lastAppliedRevision: main@sha1:cf7d5e4 equals the GitRepository artifact revision. The Secret carries kustomize.toolkit.fluxcd.io/name: vpn-ingress, so Flux delivered it.
• SOPS decryption works, proven by digest rather than by eye. The value in platform/vpn-ingress/wg-server-key.sops.yaml on main, decrypted, is 44 bytes, is not the placeholder, and wg pubkey accepts it. Its SHA-256 equals the SHA-256 of the live Secret's decoded data.privatekey (d7c8f8de6f252649… both sides). No key material was printed at any point — lengths, booleans and digests only.
• The wg pubkey guard fired correctly, and its ENC[ branch was never needed. During the kubelet window before the updated Secret propagated, all three pods reported "It is still the committed REPLACE-ME PLACEHOLDER" — the guard distinguishing a stale/unfilled value from ciphertext, exactly as designed. It cleared on its own once the mount refreshed; a transient REPLACE-ME CrashLoop for a minute or two after the key PR merges is expected, not a fault.
• The kernel module reaches the pod: ip link add dev wg0 type wireguard succeeded and /proc/modules lists wireguard. So the node_baseline addition works and CAP_SYS_MODULE is correctly not required.
• allowPrivilegeEscalation: false does NOT break the image's s6 init — the container reaches wg-quick normally. The flag stays; the fear recorded in #55 is closed.
• The iptables backends MATCH, which was my single biggest documented doubt. The image ships iptables v1.8.13 (nf_tables) via /usr/sbin/xtables-nft-multi, and Debian 13 is nf_tables. The nightmare case — rules accepted into a table the kernel never consults, so handshakes succeed and every forwarded connection hangs — cannot occur on this pairing.
• net.ipv4.ip_forward is 1 in the host netns the pod shares, and /proc/sys is ro there. Both read directly from the pod.
• Toolbox gate from an isolated worktree: rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass; rc 3 is the two environmental skips (gitea-settings, conformance L2/L3). Not green. Reported as rc 3.

What I did NOT verify

• That the tunnel now comes up. This fix has not reconciled. The next reconcile is the test, and its success criterion is specific: pods 1/1 Running, and wg show reporting interface: wg0 with listening port: 51820.
• That the MASQUERADE and FORWARD rules actually apply. They are the next statements after the one that failed, so they have never executed. The backends matching (above) removes the worst failure mode but proves nothing about the rules themselves. First check after the pods go Ready: iptables -t nat -S POSTROUTING | grep 10.90.0.0/24 on a node.
• Anything about client traffic. Still zero peers enrolled, still no WAN port-forward, still no off-LAN vantage point. The VPN has carried no packet.
• Whether the transient REPLACE-ME window has an upper bound. It cleared within ~2 minutes here; I did not determine the kubelet refresh behaviour, so a longer stall after a key change is uncharacterised.
• The sops set hand-off command, still. The operator filled the key in by their own route (#62), so the exact quoting I documented inside dev.sh run bash -c remains unexercised.