From 9a69f6906a0f2677f657d613d48a0bb35abf4385 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 3 Oct 2026 18:07:18 -0500 Subject: [PATCH] fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing mounts cgroup2 there. The agent mounted one from inside its own private cgroup namespace, so socket-LB attached to the agent's own cgroup and host processes got no Service/LB translation. k3s-prod-1 (Gateway L2 lease holder + kube-vip VIP) therefore could not reach the Gateway VIP, and its apiserver's OIDC authenticator has failed since 2026-09-12. Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-cilium-cgroup-hostroot.md | 76 +++++++++++++++++++ .../cilium/templates/cilium-values.yaml.j2 | 7 +- platform/cilium/helmrelease.yaml | 29 +++++-- platform/cilium/values.yaml | 6 +- 4 files changed, 109 insertions(+), 9 deletions(-) create mode 100644 decisions/2026-10-03-cilium-cgroup-hostroot.md diff --git a/decisions/2026-10-03-cilium-cgroup-hostroot.md b/decisions/2026-10-03-cilium-cgroup-hostroot.md new file mode 100644 index 0000000..02b083b --- /dev/null +++ b/decisions/2026-10-03-cilium-cgroup-hostroot.md @@ -0,0 +1,76 @@ +--- +date: 2026-10-03 +branch: fix/cilium-cgroup-hostroot +title: Cilium socket-LB never covered host processes — cgroup.hostRoot pointed at an unmounted path +--- + +## What changed + +`cgroup.hostRoot` changes from `/run/cilium/cgroupv2` to `/sys/fs/cgroup` in all three parity +copies: `platform/cilium/helmrelease.yaml` (authoritative), `platform/cilium/values.yaml` +(mirror) and `infra/ansible/roles/cilium/templates/cilium-values.yaml.j2` (bootstrap). +`autoMount` stays `false`. + +**The defect.** With `autoMount: false`, the Cilium chart only bind-mounts whatever exists at +`hostRoot`, and expects the operator to have mounted cgroup2 there (cilium 1.20.1 values.yaml). +The repo's rationale was "k3s provides the cgroupv2 mount". That is true only for +`/sys/fs/cgroup`. Nothing mounts cgroup2 at `/run/cilium/cgroupv2` on these nodes; it is an +empty `DirectoryOrCreate` hostPath. So the agent mounted cgroup2 there itself, from inside its +own private cgroup namespace, and that mount's root is the agent container's own leaf cgroup. +The socket-LB programs (`cil_sock4_connect` etc.) were attached to it and covered only the +agent. + +**The consequence.** Every other host-netns process got no Service/LB translation: +- Service ClusterIPs were unreachable from every node's host. +- The node holding the Gateway's L2 lease could not reach the Gateway VIP (`no route to host`). + Other nodes reached it only because their packets crossed the wire to the lease holder, where + the tc datapath handles them. +- k3s-prod-1 holds both the Gateway lease and the kube-vip API VIP. Its apiserver could not reach + the OIDC issuer, logging `oidc: authenticator not initialized` since at least 2026-09-12 + (899 occurrences by 2026-10-03). kubectl SSO through the VIP therefore failed, and in-cluster + OIDC clients failed on whichever calls landed on that apiserver. +- Webhooks and aggregated APIs were unaffected only because k3s runs + `--enable-aggregator-routing=true` (endpoint IPs, not ClusterIPs). + +Cilium's own source (v1.20.1 `bpf/bpf_sock.c`, `__sock4_xlate_fwd`) handles exactly this case +when the hook does run: a host-netns connect to an L7-LB Service (the Gateway) is rewritten to +`127.0.0.1:`, the local Envoy listener. The fix is to make the hook cover the +host. + +## What I verified + +- `cilium-dbg status --verbose` on k3s-prod-1: `Socket LB: Enabled`, `Coverage: Hostns-only`. + The live cilium-config shows `bpf-lb-sock=false`, but socket LB is implied by kube-proxy + replacement. The Gateway Service frontends `192.168.2.20:{80,443}` and + `10.43.225.79:{80,443}` map to `127.0.0.1:14337`. +- Host: `findmnt /run/cilium/cgroupv2` is empty on all three nodes, and `/sys/fs/cgroup` is + `cgroup2`. +- In the agent: `/run/cilium/cgroupv2` is a `cgroup2 none` mount, the agent's own cgroup reads + `0::/`, the mount has 0 child cgroups, and `bpftool cgroup show /run/cilium/cgroupv2` lists all + `cil_sock*` programs. k3s runs in `0::/system.slice/k3s.service`, and the agent container in + `/kubepods.slice/…/cri-containerd-3ad04cde….scope`. +- **The decisive test**, run on k3s-prod-1 in the same netns: raw TCP connect to + `192.168.2.20:443`, `10.43.225.79:443` and `10.43.59.141:8080` CONNECTED from inside the agent + container (covered cgroup) and FAILED from the host (`No route to host` / timeout). +- Blast radius: host-network pods are cilium, cilium-envoy, cilium-operator (API via + `192.168.2.9:6443`, not a Service), kube-vip (`127.0.0.1:6443`, `svc_enable=false`) and + node-exporter (no outbound). Nothing depends on host Service translation being absent; the + change only makes Service/LB addresses reachable from the host where they currently fail. +- `helm template cilium/cilium --version 1.20.1` with the new values, in the toolbox: the + DaemonSet bind-mounts hostPath `/sys/fs/cgroup` into `cilium-agent` and `clean-cilium-state`, + the cilium-config shows `cgroup-root: "/sys/fs/cgroup"`, and the init containers are unchanged + (no `mount-cgroup`). +- Toolbox gate: see the PR description. + +## What I did NOT verify + +- **The rollout itself.** After merge, the agent DaemonSet rolls one node at a time + (`maxUnavailable: 1`). On each node, afterwards: host TCP connects to the three targets above + must succeed. On k3s-prod-1, the apiserver's background OIDC init retry must stop logging + `authenticator not initialized` with no k3s restart. +- That the old programs, attached to the previous agent container's cgroup, disappear with it. + Expected, because cgroup BPF attachments die with the cgroup, but not observed. +- An end-to-end kubectl OIDC login (needs a browser). +- No regression test was added. A natural one is an L2 conformance check that a host-netns + connect to the Gateway VIP succeeds on the lease holder, but `conformance/**` is a sensitive + path. Left as a follow-up. diff --git a/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 b/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 index 0971e24..fd15eb6 100644 --- a/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 +++ b/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 @@ -51,11 +51,14 @@ hubble: l2announcements: enabled: true -# k3s ships its own cgroup mount; point Cilium at it instead of auto-mounting. +# Point Cilium at the host's real cgroup2 root (/sys/fs/cgroup, mounted by systemd) instead of +# auto-mounting. NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so the agent mounted its +# OWN namespaced cgroup and socket-LB covered only itself — host processes (the apiserver) lost +# Service/LB translation. [parity] platform/cilium/helmrelease.yaml explains (2026-10-03). cgroup: autoMount: enabled: false - hostRoot: /run/cilium/cgroupv2 + hostRoot: /sys/fs/cgroup # Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the # platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a diff --git a/platform/cilium/helmrelease.yaml b/platform/cilium/helmrelease.yaml index 46d177b..831c756 100644 --- a/platform/cilium/helmrelease.yaml +++ b/platform/cilium/helmrelease.yaml @@ -106,14 +106,33 @@ spec: # CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2. l2announcements: enabled: true - # --- cgroup: k3s ships its own cgroup mount; point Cilium at it, don't auto-mount --- - # [parity] MUST equal Ansible cgroup block. Not in the deliverable's headline list, but it IS - # part of the working CNI config on k3s: reconciling without it lets Cilium try to auto-mount - # cgroupv2 and break on adopt. Kept in parity for a safe Phase-2 adoption. + # --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup --- + # [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2). + # + # FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR + # EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists + # at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified + # cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at + # /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent + # mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the + # agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered + # only the agent. Every other host-netns process (the k3s apiserver in + # /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from + # every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all + # ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not + # initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright. + # Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443 + # and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and + # FAILED from the host, same node, same netns. Full write-up: + # decisions/2026-10-03-cilium-cgroup-hostroot.md. + # + # /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified + # with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount + # stays off because the mount it would create already exists. cgroup: autoMount: enabled: false - hostRoot: /run/cilium/cgroupv2 + hostRoot: /sys/fs/cgroup # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident --- # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1. # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate. diff --git a/platform/cilium/values.yaml b/platform/cilium/values.yaml index 44a10f5..4f0bf41 100644 --- a/platform/cilium/values.yaml +++ b/platform/cilium/values.yaml @@ -41,11 +41,13 @@ hubble: # [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1) l2announcements: enabled: true -# [parity] k3s provides the cgroupv2 mount; do not auto-mount +# [parity] the host's cgroup2 root is /sys/fs/cgroup (systemd mounts it); do not auto-mount. +# NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so socket-LB attached to the agent's own +# cgroup and host processes lost Service/LB translation (2026-10-03 — see helmrelease.yaml). cgroup: autoMount: enabled: false - hostRoot: /run/cilium/cgroupv2 + hostRoot: /sys/fs/cgroup # [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart # default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the # 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the -- 2.54.0