fix(vpn-ingress): stop trying to set net.ipv4.ip_forward — a hostNetwork pod cannot, and wg-quick dies on a failed PostUp
The tunnel never came up on its first reconcile with a real key. One line of my
own manifest was the whole cause. Measured on prod 2026-10-06:
[#] ip link add dev wg0 type wireguard <- fine
[#] wg addconf wg0 /dev/fd/63 <- fine
[#] ip -4 address add 10.90.0.1/24 dev wg0 <- fine
[#] ip link set mtu 1420 up dev wg0 <- fine
[#] sysctl -q -w net.ipv4.ip_forward=1
sysctl: permission denied on key "net.ipv4.ip_forward"
[#] ip link delete dev wg0
**** Tunnel failed, will stop all others! ****
Not a missing capability: with hostNetwork the runtime mounts /proc/sys
READ-ONLY (confirmed in the pod's /proc/mounts), because those sysctls are the
HOST's globals. CAP_NET_ADMIN cannot lift a read-only mount, so the write can
never succeed here -- and wg-quick treats a failed PostUp as fatal, so it
deleted the interface it had just built.
The line was never needed. node_baseline already sets ip_forward=1, and reading
it from the pod returns 1. My header comment said exactly that ("so no pod
sysctl is needed") and then emitted the sysctl anyway, justified as "re-asserted
so this component does not silently depend on another role's sysctl file". That
defensive re-assertion is the entire outage. In a hostNetwork pod: VERIFY host
state, never write it.
Replaced with a read in the init container -- `grep -qx 1
/proc/sys/net/ipv4/ip_forward` -- failing closed with a message naming the
node_baseline role. Still fail-closed on purpose: a tunnel that comes up unable
to forward satisfies every probe here and silently drops all client traffic.
Also documents, in the manifest and the runbook, that the image's `chown:
Operation not permitted` run and its "we will not provide support for it"
warning are expected and harmless -- capabilities are dropped to NET_ADMIN so
its s6 init has no CAP_CHOWN, /config is a writable tmpfs, everything runs as
root, and wg-quick proceeds (it did; the failure was 30 lines later).
Three other unknowns closed by the same reconcile, now recorded in the README:
the kernel module reaches the pod (ip link add type wireguard works, no
CAP_SYS_MODULE needed); allowPrivilegeEscalation:false does NOT break the s6
init; and the image's iptables is nf_tables (xtables-nft-multi) matching Debian
13, which closes the worst documented fear -- rules landing in a table the
kernel never consults.
No sensitive paths; merges on a green gate with no override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
cf7d5e4316
commit
5c43fbac65
4 files changed
+200
-5
No files matched your search
@@ -134,6 +134,36 @@ specific failure in its log — so you never get a server that silently cannot c
|
||||
| `It is EMPTY` | the value was blanked |
|
||||
| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
|
||||
| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
|
||||
| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
|
||||
|
||||
### What a healthy start looks like
|
||||
|
||||
Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key:
|
||||
|
||||
```
|
||||
wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820.
|
||||
```
|
||||
|
||||
then the `wireguard` container brings the tunnel up:
|
||||
|
||||
```
|
||||
**** Found WG conf /config/wg_confs/wg0.conf, adding to list ****
|
||||
**** Activating tunnel /config/wg_confs/wg0.conf ****
|
||||
[#] ip link add dev wg0 type wireguard
|
||||
[#] wg addconf wg0 /dev/fd/63
|
||||
[#] ip -4 address add 10.90.0.1/24 dev wg0
|
||||
[#] ip link set mtu 1420 up dev wg0
|
||||
```
|
||||
|
||||
**Ignore the `chown: ... Operation not permitted` run before that**, and the image's own
|
||||
"Permissions could not be set ... we will not provide support for it". Capabilities are dropped to
|
||||
`NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs
|
||||
as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter.
|
||||
|
||||
**A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device`
|
||||
means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it
|
||||
died on. That is the shape of a real failure; the probe is doing its job of refusing to call a
|
||||
server without a `wg0` "ready".
|
||||
|
||||
## Step 3 — forward udp/51820 on the router (once)
|
||||
|
||||
|
||||
Reference in new issue
Block a user