home-network-k3s-impl/decisions/2026-10-03-cilium-cgroup-hostroot.md

date, branch, title

date	branch	title
2026-10-03	fix/cilium-cgroup-hostroot	Cilium socket-LB never covered host processes — cgroup.hostRoot pointed at an unmounted path

What changed

cgroup.hostRoot changes from /run/cilium/cgroupv2 to /sys/fs/cgroup in all three parity copies: platform/cilium/helmrelease.yaml (authoritative), platform/cilium/values.yaml (mirror) and infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 (bootstrap). autoMount stays false.

The defect. With autoMount: false, the Cilium chart only bind-mounts whatever exists at hostRoot, and expects the operator to have mounted cgroup2 there (cilium 1.20.1 values.yaml). The repo's rationale was "k3s provides the cgroupv2 mount". That is true only for /sys/fs/cgroup. Nothing mounts cgroup2 at /run/cilium/cgroupv2 on these nodes; it is an empty DirectoryOrCreate hostPath. So the agent mounted cgroup2 there itself, from inside its own private cgroup namespace, and that mount's root is the agent container's own leaf cgroup. The socket-LB programs (cil_sock4_connect etc.) were attached to it and covered only the agent.

The consequence. Every other host-netns process got no Service/LB translation:

• Service ClusterIPs were unreachable from every node's host.
• The node holding the Gateway's L2 lease could not reach the Gateway VIP (no route to host). Other nodes reached it only because their packets crossed the wire to the lease holder, where the tc datapath handles them.
• k3s-prod-1 holds both the Gateway lease and the kube-vip API VIP. Its apiserver could not reach the OIDC issuer, logging oidc: authenticator not initialized since at least 2026-09-12 (899 occurrences by 2026-10-03). kubectl SSO through the VIP therefore failed, and in-cluster OIDC clients failed on whichever calls landed on that apiserver.
• Webhooks and aggregated APIs were unaffected only because k3s runs --enable-aggregator-routing=true (endpoint IPs, not ClusterIPs).

Cilium's own source (v1.20.1 bpf/bpf_sock.c, __sock4_xlate_fwd) handles exactly this case when the hook does run: a host-netns connect to an L7-LB Service (the Gateway) is rewritten to 127.0.0.1:<l7_lb_proxy_port>, the local Envoy listener. The fix is to make the hook cover the host.

What I verified

• cilium-dbg status --verbose on k3s-prod-1: Socket LB: Enabled, Coverage: Hostns-only. The live cilium-config shows bpf-lb-sock=false, but socket LB is implied by kube-proxy replacement. The Gateway Service frontends 192.168.2.20:{80,443} and 10.43.225.79:{80,443} map to 127.0.0.1:14337.
• Host: findmnt /run/cilium/cgroupv2 is empty on all three nodes, and /sys/fs/cgroup is cgroup2.
• In the agent: /run/cilium/cgroupv2 is a cgroup2 none mount, the agent's own cgroup reads 0::/, the mount has 0 child cgroups, and bpftool cgroup show /run/cilium/cgroupv2 lists all cil_sock* programs. k3s runs in 0::/system.slice/k3s.service, and the agent container in /kubepods.slice/…/cri-containerd-3ad04cde….scope.
• The decisive test, run on k3s-prod-1 in the same netns: raw TCP connect to 192.168.2.20:443, 10.43.225.79:443 and 10.43.59.141:8080 CONNECTED from inside the agent container (covered cgroup) and FAILED from the host (No route to host / timeout).
• Blast radius: host-network pods are cilium, cilium-envoy, cilium-operator (API via 192.168.2.9:6443, not a Service), kube-vip (127.0.0.1:6443, svc_enable=false) and node-exporter (no outbound). Nothing depends on host Service translation being absent; the change only makes Service/LB addresses reachable from the host where they currently fail.
• helm template cilium/cilium --version 1.20.1 with the new values, in the toolbox: the DaemonSet bind-mounts hostPath /sys/fs/cgroup into cilium-agent and clean-cilium-state, the cilium-config shows cgroup-root: "/sys/fs/cgroup", and the init containers are unchanged (no mount-cgroup).
• Toolbox gate: see the PR description.

What I did NOT verify

• The rollout itself. After merge, the agent DaemonSet rolls one node at a time (maxUnavailable: 1). On each node, afterwards: host TCP connects to the three targets above must succeed. On k3s-prod-1, the apiserver's background OIDC init retry must stop logging authenticator not initialized with no k3s restart.
• That the old programs, attached to the previous agent container's cgroup, disappear with it. Expected, because cgroup BPF attachments die with the cgroup, but not observed.
• An end-to-end kubectl OIDC login (needs a browser).
• No regression test was added. A natural one is an L2 conformance check that a host-netns connect to the Gateway VIP succeeds on the lease holder, but conformance/** is a sensitive path. Left as a follow-up.