Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

fix(vpn-ingress): stop trying to set net.ipv4.ip_forward — a hostNetwork pod cannot, and wg-quick dies on a failed PostUp #63

Merged
jakman85 merged 1 commits from fix/vpn-ingress-ip-forward into main 2026-10-06 07:09:34 +00:00
Conversation 0 Commits 1 Files Changed 4
+200 -5
gitea-admin commented 2026-10-06 06:30:44 +00:00
Owner
Copy link
Copy source

The tunnel never came up on its first reconcile with a real key, and one line of my own manifest was the whole cause.

Measured in the wireguard container on prod, 2026-10-06:

[#] ip link add dev wg0 type wireguard        <- fine
[#] wg addconf wg0 /dev/fd/63                 <- fine
[#] ip -4 address add 10.90.0.1/24 dev wg0    <- fine
[#] ip link set mtu 1420 up dev wg0           <- fine
[#] sysctl -q -w net.ipv4.ip_forward=1
sysctl: permission denied on key "net.ipv4.ip_forward"
[#] ip link delete dev wg0
**** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! ****

Not a missing capability. With hostNetwork: true the runtime mounts /proc/sys read-only — confirmed in the pod's own /proc/mounts: proc /proc/sys proc ro,nosuid,nodev,noexec,relatime — because those sysctls are the host's globals. CAP_NET_ADMIN cannot lift a read-only mount, so that write can never succeed from this pod. And wg-quick treats any failed PostUp as fatal, so it deleted the interface it had just finished building.

The line was never needed. node_baseline already sets net.ipv4.ip_forward=1, and reading it from inside the pod returns 1. I had written the header comment correctly — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so this component does not silently depend on another role's sysctl file surviving". That defensive re-assertion is the entire outage. The reusable lesson is narrow: in a hostNetwork pod, verify host state, never write it.

The fix

The PostUp = sysctl … line is gone, replaced by a read in the init container — grep -qx 1 /proc/sys/net/ipv4/ip_forward — which fails closed with a message naming the node_baseline role. Reading /proc/sys is permitted; only writing is not. Still fail-closed deliberately: a tunnel that comes up but cannot forward would satisfy every probe in this component and silently drop every client's traffic.

The header bullet now says do not attempt one, with the mechanism, so the next author doesn't re-add it for the reason I did. The securityContext comment claimed NET_ADMIN was partly for "set the host sysctl" — false, corrected.

Three other unknowns closed by the same reconcile

question answer
Does the kernel module reach the pod? Yes — ip link add dev wg0 type wireguard succeeds and /proc/modules lists wireguard. The node_baseline addition works; CAP_SYS_MODULE correctly not needed.
Does allowPrivilegeEscalation: false break the image's s6 init? No — the container reaches wg-quick normally. The flag stays.
Is the image's iptables backend compatible with the host's? Yes, both nft — image ships iptables v1.8.13 (nf_tables) via xtables-nft-multi; Debian 13 is nf_tables. This was my single biggest documented fear (rules accepted into a table the kernel never consults, so handshakes succeed and every forwarded connection hangs). Closed.

Also now documented as expected and harmless, in both the manifest and the runbook, so nobody chases it: the image's chown: … Operation not permitted run and its own "Permissions could not be set … we will not provide support for it". Capabilities are dropped to NET_ADMIN, so its s6 init has no CAP_CHOWN; /config is a writable tmpfs, everything runs as root, and wg-quick proceeds past it — as it did here, failing 30 lines later.

Verification notes worth keeping

  • The GitOps + SOPS path is proven end to end, by digest rather than by eye. The value in wg-server-key.sops.yaml on main decrypts to 44 bytes, is not the placeholder, and wg pubkey accepts it; its SHA-256 equals the live Secret's decoded data.privatekey (d7c8f8de… both sides). No key material was printed — lengths, booleans and digests only.
  • The wg pubkey guard behaved exactly as designed, and its ENC[ branch was never needed. In the kubelet window before the updated Secret propagated, all three pods reported "still the committed REPLACE-ME PLACEHOLDER" and then cleared on their own. A transient REPLACE-ME CrashLoop for a minute or two after a key PR merges is expected, not a fault — now in the runbook.
  • Gate: rc 3, all runnable stages green, two environmental skips. Not green — reported as rc 3.

Not verified

That the tunnel now comes up — this fix has not reconciled. Success criterion is specific: pods 1/1 Running, and wg show reporting interface: wg0 with listening port: 51820.

That the MASQUERADE/FORWARD rules apply — they are the next statements after the one that failed, so they have never executed. Matching backends removes the worst case but proves nothing about the rules. First check once pods are Ready: iptables -t nat -S POSTROUTING | grep 10.90.0.0/24 on a node.

No sensitive paths — platform/vpn-ingress/**, runbooks/**, decisions/** only. Merges on a green gate, no override needed.

🤖 Generated with Claude Code

**The tunnel never came up on its first reconcile with a real key, and one line of my own manifest was the whole cause.** Measured in the `wireguard` container on prod, 2026-10-06: ``` [#] ip link add dev wg0 type wireguard <- fine [#] wg addconf wg0 /dev/fd/63 <- fine [#] ip -4 address add 10.90.0.1/24 dev wg0 <- fine [#] ip link set mtu 1420 up dev wg0 <- fine [#] sysctl -q -w net.ipv4.ip_forward=1 sysctl: permission denied on key "net.ipv4.ip_forward" [#] ip link delete dev wg0 **** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! **** ``` **Not a missing capability.** With `hostNetwork: true` the runtime mounts `/proc/sys` **read-only** — confirmed in the pod's own `/proc/mounts`: `proc /proc/sys proc ro,nosuid,nodev,noexec,relatime` — because those sysctls are the *host's* globals. `CAP_NET_ADMIN` cannot lift a read-only mount, so that write can never succeed from this pod. And `wg-quick` treats any failed `PostUp` as fatal, so it deleted the interface it had just finished building. **The line was never needed.** `node_baseline` already sets `net.ipv4.ip_forward=1`, and reading it from inside the pod returns `1`. I had written the header comment correctly — *"ip_forward is ALREADY 1 on these nodes … so no pod sysctl is needed"* — and then emitted the sysctl anyway, justified in its own comment as *"re-asserted so this component does not silently depend on another role's sysctl file surviving"*. **That defensive re-assertion is the entire outage.** The reusable lesson is narrow: in a hostNetwork pod, verify host state, never write it. ## The fix The `PostUp = sysctl …` line is gone, replaced by a **read** in the init container — `grep -qx 1 /proc/sys/net/ipv4/ip_forward` — which fails closed with a message naming the `node_baseline` role. Reading `/proc/sys` is permitted; only writing is not. Still fail-closed deliberately: a tunnel that comes up but cannot forward would satisfy every probe in this component and silently drop every client's traffic. The header bullet now says *do not attempt one*, with the mechanism, so the next author doesn't re-add it for the reason I did. The `securityContext` comment claimed `NET_ADMIN` was partly for "set the host sysctl" — false, corrected. ## Three other unknowns closed by the same reconcile | question | answer | |---|---| | Does the **kernel module** reach the pod? | **Yes** — `ip link add dev wg0 type wireguard` succeeds and `/proc/modules` lists `wireguard`. The `node_baseline` addition works; `CAP_SYS_MODULE` correctly not needed. | | Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No** — the container reaches `wg-quick` normally. The flag stays. | | Is the image's **iptables** backend compatible with the host's? | **Yes, both nft** — image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`; Debian 13 is nf_tables. This was my single biggest documented fear (rules accepted into a table the kernel never consults, so handshakes succeed and every forwarded connection hangs). Closed. | Also now documented as **expected and harmless**, in both the manifest and the runbook, so nobody chases it: the image's `chown: … Operation not permitted` run and its own "Permissions could not be set … we will not provide support for it". Capabilities are dropped to `NET_ADMIN`, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root, and `wg-quick` proceeds past it — as it did here, failing 30 lines later. ## Verification notes worth keeping - **The GitOps + SOPS path is proven end to end, by digest rather than by eye.** The value in `wg-server-key.sops.yaml` on `main` decrypts to 44 bytes, is not the placeholder, and `wg pubkey` accepts it; its SHA-256 equals the live Secret's decoded `data.privatekey` (`d7c8f8de…` both sides). No key material was printed — lengths, booleans and digests only. - **The `wg pubkey` guard behaved exactly as designed, and its `ENC[` branch was never needed.** In the kubelet window before the updated Secret propagated, all three pods reported *"still the committed REPLACE-ME PLACEHOLDER"* and then cleared on their own. **A transient `REPLACE-ME` CrashLoop for a minute or two after a key PR merges is expected, not a fault** — now in the runbook. - Gate: **rc 3**, all runnable stages green, two environmental skips. **Not green — reported as rc 3.** ## Not verified **That the tunnel now comes up** — this fix has not reconciled. Success criterion is specific: pods `1/1 Running`, and `wg show` reporting `interface: wg0` with `listening port: 51820`. **That the MASQUERADE/FORWARD rules apply** — they are the next statements after the one that failed, so they have **never executed**. Matching backends removes the worst case but proves nothing about the rules. First check once pods are Ready: `iptables -t nat -S POSTROUTING | grep 10.90.0.0/24` on a node. No sensitive paths — `platform/vpn-ingress/**`, `runbooks/**`, `decisions/**` only. **Merges on a green gate, no override needed.** 🤖 Generated with [Claude Code](https://claude.com/claude-code)
gitea-admin added 1 commit 2026-10-06 06:30:44 +00:00
fix(vpn-ingress): stop trying to set net.ipv4.ip_forward — a hostNetwork pod cannot, and wg-quick dies on a failed PostUp
gate / gate (pull_request) Successful in 1m51s
Details
gate / conformance (pull_request) Has been skipped
Details
5c43fbac65 
The tunnel never came up on its first reconcile with a real key. One line of my
own manifest was the whole cause. Measured on prod 2026-10-06:

    [#] ip link add dev wg0 type wireguard        <- fine
    [#] wg addconf wg0 /dev/fd/63                 <- fine
    [#] ip -4 address add 10.90.0.1/24 dev wg0    <- fine
    [#] ip link set mtu 1420 up dev wg0           <- fine
    [#] sysctl -q -w net.ipv4.ip_forward=1
    sysctl: permission denied on key "net.ipv4.ip_forward"
    [#] ip link delete dev wg0
    **** Tunnel failed, will stop all others! ****

Not a missing capability: with hostNetwork the runtime mounts /proc/sys
READ-ONLY (confirmed in the pod's /proc/mounts), because those sysctls are the
HOST's globals. CAP_NET_ADMIN cannot lift a read-only mount, so the write can
never succeed here -- and wg-quick treats a failed PostUp as fatal, so it
deleted the interface it had just built.

The line was never needed. node_baseline already sets ip_forward=1, and reading
it from the pod returns 1. My header comment said exactly that ("so no pod
sysctl is needed") and then emitted the sysctl anyway, justified as "re-asserted
so this component does not silently depend on another role's sysctl file". That
defensive re-assertion is the entire outage. In a hostNetwork pod: VERIFY host
state, never write it.

Replaced with a read in the init container -- `grep -qx 1
/proc/sys/net/ipv4/ip_forward` -- failing closed with a message naming the
node_baseline role. Still fail-closed on purpose: a tunnel that comes up unable
to forward satisfies every probe here and silently drops all client traffic.

Also documents, in the manifest and the runbook, that the image's `chown:
Operation not permitted` run and its "we will not provide support for it"
warning are expected and harmless -- capabilities are dropped to NET_ADMIN so
its s6 init has no CAP_CHOWN, /config is a writable tmpfs, everything runs as
root, and wg-quick proceeds (it did; the failure was 30 lines later).

Three other unknowns closed by the same reconcile, now recorded in the README:
the kernel module reaches the pod (ip link add type wireguard works, no
CAP_SYS_MODULE needed); allowPrivilegeEscalation:false does NOT break the s6
init; and the image's iptables is nf_tables (xtables-nft-multi) matching Debian
13, which closes the worst documented fear -- rules landing in a table the
kernel never consults.

No sensitive paths; merges on a green gate with no override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85 merged commit d1c07fe021 into main 2026-10-06 07:09:34 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
agent
automated-update
blast-radius
breaking
ci-tooling
dependencies
needs-review
own-image
range-widen
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
gitea-admin jakman85 (Keith Jackson) pm-agent renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#63