fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root
gate / gate (pull_request) Successful in 1m47s
gate / conformance (pull_request) Has been skipped

hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing
mounts cgroup2 there. The agent mounted one from inside its own private
cgroup namespace, so socket-LB attached to the agent's own cgroup and
host processes got no Service/LB translation. k3s-prod-1 (Gateway L2
lease holder + kube-vip VIP) therefore could not reach the Gateway VIP,
and its apiserver's OIDC authenticator has failed since 2026-09-12.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-10-03 18:07:18 -05:00
1 parent 7ef2af205d
commit 9a69f6906a
4 files changed
+109 -9

No files matched your search

@@ -0,0 +1,76 @@
---
date: 2026-10-03
branch: fix/cilium-cgroup-hostroot
title: Cilium socket-LB never covered host processes — cgroup.hostRoot pointed at an unmounted path
---
## What changed
`cgroup.hostRoot` changes from `/run/cilium/cgroupv2` to `/sys/fs/cgroup` in all three parity
copies: `platform/cilium/helmrelease.yaml` (authoritative), `platform/cilium/values.yaml`
(mirror) and `infra/ansible/roles/cilium/templates/cilium-values.yaml.j2` (bootstrap).
`autoMount` stays `false`.
**The defect.** With `autoMount: false`, the Cilium chart only bind-mounts whatever exists at
`hostRoot`, and expects the operator to have mounted cgroup2 there (cilium 1.20.1 values.yaml).
The repo's rationale was "k3s provides the cgroupv2 mount". That is true only for
`/sys/fs/cgroup`. Nothing mounts cgroup2 at `/run/cilium/cgroupv2` on these nodes; it is an
empty `DirectoryOrCreate` hostPath. So the agent mounted cgroup2 there itself, from inside its
own private cgroup namespace, and that mount's root is the agent container's own leaf cgroup.
The socket-LB programs (`cil_sock4_connect` etc.) were attached to it and covered only the
agent.
**The consequence.** Every other host-netns process got no Service/LB translation:
- Service ClusterIPs were unreachable from every node's host.
- The node holding the Gateway's L2 lease could not reach the Gateway VIP (`no route to host`).
Other nodes reached it only because their packets crossed the wire to the lease holder, where
the tc datapath handles them.
- k3s-prod-1 holds both the Gateway lease and the kube-vip API VIP. Its apiserver could not reach
the OIDC issuer, logging `oidc: authenticator not initialized` since at least 2026-09-12
(899 occurrences by 2026-10-03). kubectl SSO through the VIP therefore failed, and in-cluster
OIDC clients failed on whichever calls landed on that apiserver.
- Webhooks and aggregated APIs were unaffected only because k3s runs
`--enable-aggregator-routing=true` (endpoint IPs, not ClusterIPs).
Cilium's own source (v1.20.1 `bpf/bpf_sock.c`, `__sock4_xlate_fwd`) handles exactly this case
when the hook does run: a host-netns connect to an L7-LB Service (the Gateway) is rewritten to
`127.0.0.1:<l7_lb_proxy_port>`, the local Envoy listener. The fix is to make the hook cover the
host.
## What I verified
- `cilium-dbg status --verbose` on k3s-prod-1: `Socket LB: Enabled`, `Coverage: Hostns-only`.
The live cilium-config shows `bpf-lb-sock=false`, but socket LB is implied by kube-proxy
replacement. The Gateway Service frontends `192.168.2.20:{80,443}` and
`10.43.225.79:{80,443}` map to `127.0.0.1:14337`.
- Host: `findmnt /run/cilium/cgroupv2` is empty on all three nodes, and `/sys/fs/cgroup` is
`cgroup2`.
- In the agent: `/run/cilium/cgroupv2` is a `cgroup2 none` mount, the agent's own cgroup reads
`0::/`, the mount has 0 child cgroups, and `bpftool cgroup show /run/cilium/cgroupv2` lists all
`cil_sock*` programs. k3s runs in `0::/system.slice/k3s.service`, and the agent container in
`/kubepods.slice/…/cri-containerd-3ad04cde….scope`.
- **The decisive test**, run on k3s-prod-1 in the same netns: raw TCP connect to
`192.168.2.20:443`, `10.43.225.79:443` and `10.43.59.141:8080` CONNECTED from inside the agent
container (covered cgroup) and FAILED from the host (`No route to host` / timeout).
- Blast radius: host-network pods are cilium, cilium-envoy, cilium-operator (API via
`192.168.2.9:6443`, not a Service), kube-vip (`127.0.0.1:6443`, `svc_enable=false`) and
node-exporter (no outbound). Nothing depends on host Service translation being absent; the
change only makes Service/LB addresses reachable from the host where they currently fail.
- `helm template cilium/cilium --version 1.20.1` with the new values, in the toolbox: the
DaemonSet bind-mounts hostPath `/sys/fs/cgroup` into `cilium-agent` and `clean-cilium-state`,
the cilium-config shows `cgroup-root: "/sys/fs/cgroup"`, and the init containers are unchanged
(no `mount-cgroup`).
- Toolbox gate: see the PR description.
## What I did NOT verify
- **The rollout itself.** After merge, the agent DaemonSet rolls one node at a time
(`maxUnavailable: 1`). On each node, afterwards: host TCP connects to the three targets above
must succeed. On k3s-prod-1, the apiserver's background OIDC init retry must stop logging
`authenticator not initialized` with no k3s restart.
- That the old programs, attached to the previous agent container's cgroup, disappear with it.
Expected, because cgroup BPF attachments die with the cgroup, but not observed.
- An end-to-end kubectl OIDC login (needs a browser).
- No regression test was added. A natural one is an L2 conformance check that a host-netns
connect to the Gateway VIP succeeds on the lease holder, but `conformance/**` is a sensitive
path. Left as a follow-up.