---
date: 2026-10-03
branch: fix/cilium-cgroup-hostroot
title: Cilium socket-LB never covered host processes — cgroup.hostRoot pointed at an unmounted path
---

## What changed

`cgroup.hostRoot` changes from `/run/cilium/cgroupv2` to `/sys/fs/cgroup` in all three parity
copies: `platform/cilium/helmrelease.yaml` (authoritative), `platform/cilium/values.yaml`
(mirror) and `infra/ansible/roles/cilium/templates/cilium-values.yaml.j2` (bootstrap).
`autoMount` stays `false`.

**The defect.** With `autoMount: false`, the Cilium chart only bind-mounts whatever exists at
`hostRoot`, and expects the operator to have mounted cgroup2 there (cilium 1.20.1 values.yaml).
The repo's rationale was "k3s provides the cgroupv2 mount". That is true only for
`/sys/fs/cgroup`. Nothing mounts cgroup2 at `/run/cilium/cgroupv2` on these nodes; it is an
empty `DirectoryOrCreate` hostPath. So the agent mounted cgroup2 there itself, from inside its
own private cgroup namespace, and that mount's root is the agent container's own leaf cgroup.
The socket-LB programs (`cil_sock4_connect` etc.) were attached to it and covered only the
agent.

**The consequence.** Every other host-netns process got no Service/LB translation:
- Service ClusterIPs were unreachable from every node's host.
- The node holding the Gateway's L2 lease could not reach the Gateway VIP (`no route to host`).
  Other nodes reached it only because their packets crossed the wire to the lease holder, where
  the tc datapath handles them.
- k3s-prod-1 holds both the Gateway lease and the kube-vip API VIP. Its apiserver could not reach
  the OIDC issuer, logging `oidc: authenticator not initialized` since at least 2026-09-12
  (899 occurrences by 2026-10-03). kubectl SSO through the VIP therefore failed, and in-cluster
  OIDC clients failed on whichever calls landed on that apiserver.
- Webhooks and aggregated APIs were unaffected only because k3s runs
  `--enable-aggregator-routing=true` (endpoint IPs, not ClusterIPs).

Cilium's own source (v1.20.1 `bpf/bpf_sock.c`, `__sock4_xlate_fwd`) handles exactly this case
when the hook does run: a host-netns connect to an L7-LB Service (the Gateway) is rewritten to
`127.0.0.1:<l7_lb_proxy_port>`, the local Envoy listener. The fix is to make the hook cover the
host.

## What I verified

- `cilium-dbg status --verbose` on k3s-prod-1: `Socket LB: Enabled`, `Coverage: Hostns-only`.
  The live cilium-config shows `bpf-lb-sock=false`, but socket LB is implied by kube-proxy
  replacement. The Gateway Service frontends `192.168.2.20:{80,443}` and
  `10.43.225.79:{80,443}` map to `127.0.0.1:14337`.
- Host: `findmnt /run/cilium/cgroupv2` is empty on all three nodes, and `/sys/fs/cgroup` is
  `cgroup2`.
- In the agent: `/run/cilium/cgroupv2` is a `cgroup2 none` mount, the agent's own cgroup reads
  `0::/`, the mount has 0 child cgroups, and `bpftool cgroup show /run/cilium/cgroupv2` lists all
  `cil_sock*` programs. k3s runs in `0::/system.slice/k3s.service`, and the agent container in
  `/kubepods.slice/…/cri-containerd-3ad04cde….scope`.
- **The decisive test**, run on k3s-prod-1 in the same netns: raw TCP connect to
  `192.168.2.20:443`, `10.43.225.79:443` and `10.43.59.141:8080` CONNECTED from inside the agent
  container (covered cgroup) and FAILED from the host (`No route to host` / timeout).
- Blast radius: host-network pods are cilium, cilium-envoy, cilium-operator (API via
  `192.168.2.9:6443`, not a Service), kube-vip (`127.0.0.1:6443`, `svc_enable=false`) and
  node-exporter (no outbound). Nothing depends on host Service translation being absent; the
  change only makes Service/LB addresses reachable from the host where they currently fail.
- `helm template cilium/cilium --version 1.20.1` with the new values, in the toolbox: the
  DaemonSet bind-mounts hostPath `/sys/fs/cgroup` into `cilium-agent` and `clean-cilium-state`,
  the cilium-config shows `cgroup-root: "/sys/fs/cgroup"`, and the init containers are unchanged
  (no `mount-cgroup`).
- Toolbox gate: see the PR description.

## What I did NOT verify

- **The rollout itself.** After merge, the agent DaemonSet rolls one node at a time
  (`maxUnavailable: 1`). On each node, afterwards: host TCP connects to the three targets above
  must succeed. On k3s-prod-1, the apiserver's background OIDC init retry must stop logging
  `authenticator not initialized` with no k3s restart.
- That the old programs, attached to the previous agent container's cgroup, disappear with it.
  Expected, because cgroup BPF attachments die with the cgroup, but not observed.
- An end-to-end kubectl OIDC login (needs a browser).
- No regression test was added. A natural one is an L2 conformance check that a host-netns
  connect to the Gateway VIP succeeds on the lease holder, but `conformance/**` is a sensitive
  path. Left as a follow-up.