fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root
hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing mounts cgroup2 there. The agent mounted one from inside its own private cgroup namespace, so socket-LB attached to the agent's own cgroup and host processes got no Service/LB translation. k3s-prod-1 (Gateway L2 lease holder + kube-vip VIP) therefore could not reach the Gateway VIP, and its apiserver's OIDC authenticator has failed since 2026-09-12. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
7ef2af205d
commit
9a69f6906a
4 files changed
+109
-9
No files matched your search
@@ -106,14 +106,33 @@ spec:
|
||||
# CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
|
||||
l2announcements:
|
||||
enabled: true
|
||||
# --- cgroup: k3s ships its own cgroup mount; point Cilium at it, don't auto-mount ---
|
||||
# [parity] MUST equal Ansible cgroup block. Not in the deliverable's headline list, but it IS
|
||||
# part of the working CNI config on k3s: reconciling without it lets Cilium try to auto-mount
|
||||
# cgroupv2 and break on adopt. Kept in parity for a safe Phase-2 adoption.
|
||||
# --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup ---
|
||||
# [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2).
|
||||
#
|
||||
# FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR
|
||||
# EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists
|
||||
# at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified
|
||||
# cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at
|
||||
# /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent
|
||||
# mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the
|
||||
# agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered
|
||||
# only the agent. Every other host-netns process (the k3s apiserver in
|
||||
# /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from
|
||||
# every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all
|
||||
# ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not
|
||||
# initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright.
|
||||
# Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443
|
||||
# and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and
|
||||
# FAILED from the host, same node, same netns. Full write-up:
|
||||
# decisions/2026-10-03-cilium-cgroup-hostroot.md.
|
||||
#
|
||||
# /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified
|
||||
# with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount
|
||||
# stays off because the mount it would create already exists.
|
||||
cgroup:
|
||||
autoMount:
|
||||
enabled: false
|
||||
hostRoot: /run/cilium/cgroupv2
|
||||
hostRoot: /sys/fs/cgroup
|
||||
# --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
|
||||
# [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
|
||||
# The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
|
||||
|
||||
@@ -41,11 +41,13 @@ hubble:
|
||||
# [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
|
||||
l2announcements:
|
||||
enabled: true
|
||||
# [parity] k3s provides the cgroupv2 mount; do not auto-mount
|
||||
# [parity] the host's cgroup2 root is /sys/fs/cgroup (systemd mounts it); do not auto-mount.
|
||||
# NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so socket-LB attached to the agent's own
|
||||
# cgroup and host processes lost Service/LB translation (2026-10-03 — see helmrelease.yaml).
|
||||
cgroup:
|
||||
autoMount:
|
||||
enabled: false
|
||||
hostRoot: /run/cilium/cgroupv2
|
||||
hostRoot: /sys/fs/cgroup
|
||||
# [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
|
||||
# default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
|
||||
# 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the
|
||||
|
||||
Reference in new issue
Block a user