jakman85andClaude Opus 5.5
9a69f6906a
hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing mounts cgroup2 there. The agent mounted one from inside its own private cgroup namespace, so socket-LB attached to the agent's own cgroup and host processes got no Service/LB translation. k3s-prod-1 (Gateway L2 lease holder + kube-vip VIP) therefore could not reach the Gateway VIP, and its apiserver's OIDC authenticator has failed since 2026-09-12. Co-Authored-By: Claude Opus 5.5 <[email protected]>
77 lines
4.7 KiB
Markdown
77 lines
4.7 KiB
Markdown
---
|
|
date: 2026-10-03
|
|
branch: fix/cilium-cgroup-hostroot
|
|
title: Cilium socket-LB never covered host processes — cgroup.hostRoot pointed at an unmounted path
|
|
---
|
|
|
|
## What changed
|
|
|
|
`cgroup.hostRoot` changes from `/run/cilium/cgroupv2` to `/sys/fs/cgroup` in all three parity
|
|
copies: `platform/cilium/helmrelease.yaml` (authoritative), `platform/cilium/values.yaml`
|
|
(mirror) and `infra/ansible/roles/cilium/templates/cilium-values.yaml.j2` (bootstrap).
|
|
`autoMount` stays `false`.
|
|
|
|
**The defect.** With `autoMount: false`, the Cilium chart only bind-mounts whatever exists at
|
|
`hostRoot`, and expects the operator to have mounted cgroup2 there (cilium 1.20.1 values.yaml).
|
|
The repo's rationale was "k3s provides the cgroupv2 mount". That is true only for
|
|
`/sys/fs/cgroup`. Nothing mounts cgroup2 at `/run/cilium/cgroupv2` on these nodes; it is an
|
|
empty `DirectoryOrCreate` hostPath. So the agent mounted cgroup2 there itself, from inside its
|
|
own private cgroup namespace, and that mount's root is the agent container's own leaf cgroup.
|
|
The socket-LB programs (`cil_sock4_connect` etc.) were attached to it and covered only the
|
|
agent.
|
|
|
|
**The consequence.** Every other host-netns process got no Service/LB translation:
|
|
- Service ClusterIPs were unreachable from every node's host.
|
|
- The node holding the Gateway's L2 lease could not reach the Gateway VIP (`no route to host`).
|
|
Other nodes reached it only because their packets crossed the wire to the lease holder, where
|
|
the tc datapath handles them.
|
|
- k3s-prod-1 holds both the Gateway lease and the kube-vip API VIP. Its apiserver could not reach
|
|
the OIDC issuer, logging `oidc: authenticator not initialized` since at least 2026-09-12
|
|
(899 occurrences by 2026-10-03). kubectl SSO through the VIP therefore failed, and in-cluster
|
|
OIDC clients failed on whichever calls landed on that apiserver.
|
|
- Webhooks and aggregated APIs were unaffected only because k3s runs
|
|
`--enable-aggregator-routing=true` (endpoint IPs, not ClusterIPs).
|
|
|
|
Cilium's own source (v1.20.1 `bpf/bpf_sock.c`, `__sock4_xlate_fwd`) handles exactly this case
|
|
when the hook does run: a host-netns connect to an L7-LB Service (the Gateway) is rewritten to
|
|
`127.0.0.1:<l7_lb_proxy_port>`, the local Envoy listener. The fix is to make the hook cover the
|
|
host.
|
|
|
|
## What I verified
|
|
|
|
- `cilium-dbg status --verbose` on k3s-prod-1: `Socket LB: Enabled`, `Coverage: Hostns-only`.
|
|
The live cilium-config shows `bpf-lb-sock=false`, but socket LB is implied by kube-proxy
|
|
replacement. The Gateway Service frontends `192.168.2.20:{80,443}` and
|
|
`10.43.225.79:{80,443}` map to `127.0.0.1:14337`.
|
|
- Host: `findmnt /run/cilium/cgroupv2` is empty on all three nodes, and `/sys/fs/cgroup` is
|
|
`cgroup2`.
|
|
- In the agent: `/run/cilium/cgroupv2` is a `cgroup2 none` mount, the agent's own cgroup reads
|
|
`0::/`, the mount has 0 child cgroups, and `bpftool cgroup show /run/cilium/cgroupv2` lists all
|
|
`cil_sock*` programs. k3s runs in `0::/system.slice/k3s.service`, and the agent container in
|
|
`/kubepods.slice/…/cri-containerd-3ad04cde….scope`.
|
|
- **The decisive test**, run on k3s-prod-1 in the same netns: raw TCP connect to
|
|
`192.168.2.20:443`, `10.43.225.79:443` and `10.43.59.141:8080` CONNECTED from inside the agent
|
|
container (covered cgroup) and FAILED from the host (`No route to host` / timeout).
|
|
- Blast radius: host-network pods are cilium, cilium-envoy, cilium-operator (API via
|
|
`192.168.2.9:6443`, not a Service), kube-vip (`127.0.0.1:6443`, `svc_enable=false`) and
|
|
node-exporter (no outbound). Nothing depends on host Service translation being absent; the
|
|
change only makes Service/LB addresses reachable from the host where they currently fail.
|
|
- `helm template cilium/cilium --version 1.20.1` with the new values, in the toolbox: the
|
|
DaemonSet bind-mounts hostPath `/sys/fs/cgroup` into `cilium-agent` and `clean-cilium-state`,
|
|
the cilium-config shows `cgroup-root: "/sys/fs/cgroup"`, and the init containers are unchanged
|
|
(no `mount-cgroup`).
|
|
- Toolbox gate: see the PR description.
|
|
|
|
## What I did NOT verify
|
|
|
|
- **The rollout itself.** After merge, the agent DaemonSet rolls one node at a time
|
|
(`maxUnavailable: 1`). On each node, afterwards: host TCP connects to the three targets above
|
|
must succeed. On k3s-prod-1, the apiserver's background OIDC init retry must stop logging
|
|
`authenticator not initialized` with no k3s restart.
|
|
- That the old programs, attached to the previous agent container's cgroup, disappear with it.
|
|
Expected, because cgroup BPF attachments die with the cgroup, but not observed.
|
|
- An end-to-end kubectl OIDC login (needs a browser).
|
|
- No regression test was added. A natural one is an L2 conformance check that a host-netns
|
|
connect to the Gateway VIP succeeds on the lease holder, but `conformance/**` is a sensitive
|
|
path. Left as a follow-up.
|