home-network-k3s-impl/decisions/2026-10-05-cilium-roll-out-on-config-change.md

date, branch, title

date	branch	title
2026-10-05	feat/cilium-roll-out-on-config-change	Cilium config changes were silently inert — roll the agents when cilium-config changes, which also finally activates the Hubble exporter

What changed

A merged Cilium config change did nothing, and everything reported success. The Hubble flow exporter (PR #50, merged 2026-10-04) wrote its settings into cilium-config and never took effect.

Measured 2026-10-05, over a day after the merge
HelmRelease cilium	Ready=True, UpgradeSucceeded, "Helm upgrade succeeded for release kube-system/cilium.v6"
cilium-config keys	hubble-export-file-path: stdout, hubble-export-allowlist: {"destination_port":["32400"]} {"source_port":["32400"]} — all present
cilium agent pods	all three still the PRE-MERGE pods, startTime 2026-10-04T07:11Z
exported flow lines in agent stdout	0, 0, 0
Flux Kustomizations	all Ready=True

Cause: the chart's rollOutCiliumPods defaults to false (cilium 1.20.1 values.yaml: "Roll out cilium agent pods automatically when configmap is updated"), so Helm rewrites the ConfigMap and leaves the DaemonSet's pod template untouched — and cilium-agent reads that ConfigMap only at startup. The feature was merged, believed delivered, green everywhere, and doing nothing at all.

That is the same class of failure this repo already guards against elsewhere: a green signal sitting over something that never ran.

• platform/cilium/helmrelease.yaml (authoritative): rollOutCiliumPods: true.
• platform/cilium/values.yaml (reference mirror) and infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 (the [parity] source of truth): the same, so a bootstrap render does not differ from what Flux reconciles.

The cost is real and is the point. Every future Cilium config change now triggers a cluster-wide agent rollout instead of waiting for someone to remember a manual restart. That is strictly more honest — the change applies when merged, or it fails visibly — but it makes config edits disruptive by default, so they want a quiet moment. Mitigation already in place: updateStrategy.rollingUpdate.maxUnavailable: 1 means the DaemonSet will not proceed past an unready agent, so a bad render costs one node rather than the cluster.

Merging this is itself one rollout, because adding the ConfigMap checksum annotation changes the pod template — and that rollout is what finally activates the Hubble exporter.

What I verified

• Every row of the table above, read from the live HelmRelease status, the live cilium-config ConfigMap, the agent pods' startTime, and a grep for "flow" in each agent's stdout.
• The chart value name, default and semantics, quoted from cilium v1.20.1's own values.yaml — the version actually running here (quay.io/cilium/cilium:v1.20.1), not a guess from another release: rollOutCiliumPods: false with the comment "Roll out cilium agent pods automatically when configmap is updated."
• All three value files carry the key and both YAML files still parse, with spec.values.rollOutCiliumPods resolving to true.
• The blast-radius mitigation is genuinely present, not assumed: updateStrategy.rollingUpdate.maxUnavailable: 1 is already set in the HelmRelease, with a long existing comment explaining that a CNI rollout has no healthy-peer fallback.
• THE MECHANISM IS CONFIRMED BY CONTRAST, not just reasoned. The cgroup fix (PR #42, 2026-10-03) DID take effect: both the DaemonSet and the running pod mount cilium-cgroup hostPath=/sys/fs/cgroup, matching what the HelmRelease declares. The difference is exactly the one claimed above — #42 changed the POD TEMPLATE (a volume hostPath), which rolls the DaemonSet on its own, while #50 changed ONLY the ConfigMap, which does not. It also explains the agents' startTime 2026-10-04T07:11Z: that was #42's rollout, and nothing has rolled them since. So the inertness is specific to ConfigMap-only changes, which is precisely what this flag fixes.
• Flux is healthy before this change — every Kustomization Ready=True, so this lands on a clean graph rather than into an existing cascade.
• Toolbox gate: recorded in the PR.

What I did NOT verify

• THE OUTCOME, ENTIRELY. Unreconciled. No agent has rolled, no flow has been exported. This change is an argument that the rollout will happen, not evidence that it did.
• Whether filePath: stdout actually works — still the open question from #50, and it cannot be answered until the agents restart. If the literal is not special-cased, the agent may create a FILE named stdout and nothing reaches Loki: a silent no-op, not a crash. Fallback documented in #50: keep the chart's default path and add a loki.source.file target to Alloy with a hostPath mount. Check for flow lines in the agent logs immediately after the rollout.
• THE ROLLOUT'S IMPACT ON LAN DNS IS UNMEASURED AND IS THE MAIN RISK. The node holding the L2 lease for the Gateway VIP, PiHole's DNS VIP and Plex's VIP loses them briefly as it cycles. maxUnavailable: 1 bounds it to one node at a time, and BPF maps persist across an agent restart, but I have not measured how long a VIP is unreachable during a lease move, nor whether the two PiHole replicas cover the gap. Merge it at a quiet moment.
• Whether enabling this is the operator's preferred trade. The alternative — leaving it false and restarting by hand — keeps config edits non-disruptive at the cost of them being silently inert. I judged silent inertness the worse failure, but it is a judgement, not a measurement.
• Whether anything ELSE in cilium-config is currently inert. I checked the one other recent Cilium change and it is fine (see below), but I did not audit every key against the running agents' actual state.
• hubble.relay.rollOutPods / hubble.ui.rollOutPods are left at their false defaults. Both have the same silently-inert property for their own ConfigMaps; not addressed here.