fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root
gate / gate (pull_request) Successful in 1m47s
gate / conformance (pull_request) Has been skipped

hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing
mounts cgroup2 there. The agent mounted one from inside its own private
cgroup namespace, so socket-LB attached to the agent's own cgroup and
host processes got no Service/LB translation. k3s-prod-1 (Gateway L2
lease holder + kube-vip VIP) therefore could not reach the Gateway VIP,
and its apiserver's OIDC authenticator has failed since 2026-09-12.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-10-03 18:07:18 -05:00
1 parent 7ef2af205d
commit 9a69f6906a
4 files changed
+109 -9

No files matched your search

+24 -5
View File
@@ -106,14 +106,33 @@ spec:
# CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
l2announcements:
enabled: true
# --- cgroup: k3s ships its own cgroup mount; point Cilium at it, don't auto-mount ---
# [parity] MUST equal Ansible cgroup block. Not in the deliverable's headline list, but it IS
# part of the working CNI config on k3s: reconciling without it lets Cilium try to auto-mount
# cgroupv2 and break on adopt. Kept in parity for a safe Phase-2 adoption.
# --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup ---
# [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2).
#
# FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR
# EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists
# at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified
# cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at
# /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent
# mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the
# agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered
# only the agent. Every other host-netns process (the k3s apiserver in
# /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from
# every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all
# ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not
# initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright.
# Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443
# and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and
# FAILED from the host, same node, same netns. Full write-up:
# decisions/2026-10-03-cilium-cgroup-hostroot.md.
#
# /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified
# with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount
# stays off because the mount it would create already exists.
cgroup:
autoMount:
enabled: false
hostRoot: /run/cilium/cgroupv2
hostRoot: /sys/fs/cgroup
# --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
# [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
# The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
+4 -2
View File
@@ -41,11 +41,13 @@ hubble:
# [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
l2announcements:
enabled: true
# [parity] k3s provides the cgroupv2 mount; do not auto-mount
# [parity] the host's cgroup2 root is /sys/fs/cgroup (systemd mounts it); do not auto-mount.
# NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so socket-LB attached to the agent's own
# cgroup and host processes lost Service/LB translation (2026-10-03 — see helmrelease.yaml).
cgroup:
autoMount:
enabled: false
hostRoot: /run/cilium/cgroupv2
hostRoot: /sys/fs/cgroup
# [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
# default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
# 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the