fix(cilium): point cgroup.hostRoot at the host cgroup2 root (socket-LB never covered host processes) #42

2026-10-03T23:08:45Z

gitea-admin commented

2026-10-03 23:08:45 +00:00

Cilium's socket-LB never covered host processes. This changes cgroup.hostRoot from /run/cilium/cgroupv2 to /sys/fs/cgroup in all three parity copies: the HelmRelease, the values.yaml mirror and the Ansible bootstrap template.

Root cause

With autoMount: false, the chart expects cgroup2 to already be mounted at hostRoot. Nothing mounts it at /run/cilium/cgroupv2, so the agent mounted cgroup2 itself from inside its private cgroup namespace. That mount's root is the agent container's own cgroup. The cil_sock* programs were attached there and covered only the agent.

Every other host process (the k3s apiserver, anything over SSH) got no Service or load-balancer translation:

ClusterIPs time out from every node.
The node holding the Gateway's L2 lease (k3s-prod-1) gets no route to host to 192.168.2.20.
k3s-prod-1's apiserver has logged oidc: authenticator not initialized since at least 2026-09-12 (899 times). It also holds the kube-vip API address, so kubectl SSO through the VIP fails.

Proof (k3s-prod-1, same netns)

target	agent container (hooked)	host (not hooked)
`192.168.2.20:443` (Gateway VIP)	CONNECTED	No route to host
`10.43.225.79:443` (Gateway ClusterIP)	CONNECTED	FAILED
`10.43.59.141:8080` (Keycloak ClusterIP)	CONNECTED	FAILED

/sys/fs/cgroup is cgroup2 in the host's root namespace on all three nodes. The Cilium v1.20.1 source (bpf_sock.c) translates host-netns connects to the Gateway, an L7-LB Service, to the local Envoy at 127.0.0.1:<proxy port>, so hooking the real root fixes this.

Blast radius

Rollout: the agent DaemonSet rolls one node at a time (maxUnavailable: 1). helm template (1.20.1) shows the only DaemonSet change is the cgroup hostPath/mount and cgroup-root. No new init containers.
Host-network pods: cilium-operator uses 192.168.2.9:6443, kube-vip uses 127.0.0.1:6443 with svc_enable=false, and node-exporter makes no outbound calls. None depends on host Service translation being absent.
Webhooks: unaffected either way, because k3s runs enable-aggregator-routing=true.

Gate

Toolbox rc=3: spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass. gitea-settings was skipped (no token). No sensitive paths are touched.

After merge I will verify

Flux applies it, and the cilium DaemonSet rolls 3/3.
On every node, host TCP connects to the three targets above succeed.
k3s-prod-1's apiserver stops logging authenticator not initialized without a k3s restart (its OIDC init retries in the background).

🤖 Generated with Claude Code

Cilium's socket-LB never covered host processes. This changes `cgroup.hostRoot` from `/run/cilium/cgroupv2` to `/sys/fs/cgroup` in all three parity copies: the HelmRelease, the `values.yaml` mirror and the Ansible bootstrap template. ## Root cause With `autoMount: false`, the chart expects cgroup2 to already be mounted at `hostRoot`. Nothing mounts it at `/run/cilium/cgroupv2`, so the agent mounted cgroup2 itself from inside its **private cgroup namespace**. That mount's root is the agent container's own cgroup. The `cil_sock*` programs were attached there and covered only the agent. Every other host process (the k3s apiserver, anything over SSH) got no Service or load-balancer translation: - ClusterIPs time out from every node. - The node holding the Gateway's L2 lease (k3s-prod-1) gets `no route to host` to `192.168.2.20`. - k3s-prod-1's apiserver has logged `oidc: authenticator not initialized` since at least 2026-09-12 (899 times). It also holds the kube-vip API address, so **kubectl SSO through the VIP fails**. ## Proof (k3s-prod-1, same netns) | target | agent container (hooked) | host (not hooked) | |---|---|---| | `192.168.2.20:443` (Gateway VIP) | CONNECTED | No route to host | | `10.43.225.79:443` (Gateway ClusterIP) | CONNECTED | FAILED | | `10.43.59.141:8080` (Keycloak ClusterIP) | CONNECTED | FAILED | `/sys/fs/cgroup` is cgroup2 in the host's root namespace on all three nodes. The Cilium v1.20.1 source (`bpf_sock.c`) translates host-netns connects to the Gateway, an L7-LB Service, to the local Envoy at `127.0.0.1:<proxy port>`, so hooking the real root fixes this. ## Blast radius - **Rollout:** the agent DaemonSet rolls one node at a time (`maxUnavailable: 1`). `helm template` (1.20.1) shows the only DaemonSet change is the cgroup hostPath/mount and `cgroup-root`. No new init containers. - **Host-network pods:** cilium-operator uses `192.168.2.9:6443`, kube-vip uses `127.0.0.1:6443` with `svc_enable=false`, and node-exporter makes no outbound calls. None depends on host Service translation being absent. - **Webhooks:** unaffected either way, because k3s runs `enable-aggregator-routing=true`. ## Gate Toolbox `rc=3`: spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass. `gitea-settings` was skipped (no token). No sensitive paths are touched. ## After merge I will verify - Flux applies it, and the cilium DaemonSet rolls 3/3. - On every node, host TCP connects to the three targets above succeed. - k3s-prod-1's apiserver stops logging `authenticator not initialized` without a k3s restart (its OIDC init retries in the background). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-03 23:08:45 +00:00

fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root

gate / gate (pull_request) Successful in 1m47s

Details

gate / conformance (pull_request) Has been skipped

Details

9a69f6906a

hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing
mounts cgroup2 there. The agent mounted one from inside its own private
cgroup namespace, so socket-LB attached to the agent's own cgroup and
host processes got no Service/LB translation. k3s-prod-1 (Gateway L2
lease holder + kube-vip VIP) therefore could not reach the Gateway VIP,
and its apiserver's OIDC authenticator has failed since 2026-09-12.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jakman85 merged commit c95b1c7121 into main

2026-10-04 07:10:31 +00:00

jakman85 referenced this issue from a commit

2026-10-04 07:10:32 +00:00

Merge pull request 'fix(cilium): point cgroup.hostRoot at the host cgroup2 root (socket-LB never covered host processes)' (#42) from fix/cilium-cgroup-hostroot into main

gitea-admin referenced this issue from a commit

2026-10-06 02:19:15 +00:00

feat(cilium): roll the agents when cilium-config changes (merged config was silently inert)

gitea-admin referenced this pull request

2026-10-06 02:19:41 +00:00

feat(cilium): roll the agents when cilium-config changes (merged config was silently inert) #53

Sign in to join this conversation.