--- date: 2026-10-05 branch: feat/cilium-roll-out-on-config-change title: Cilium config changes were silently inert — roll the agents when cilium-config changes, which also finally activates the Hubble exporter --- ## What changed **A merged Cilium config change did nothing, and everything reported success.** The Hubble flow exporter (PR #50, merged 2026-10-04) wrote its settings into `cilium-config` and never took effect. | Measured 2026-10-05, over a day after the merge | | |---|---| | HelmRelease `cilium` | `Ready=True`, `UpgradeSucceeded`, *"Helm upgrade succeeded for release kube-system/cilium.v6"* | | `cilium-config` keys | `hubble-export-file-path: stdout`, `hubble-export-allowlist: {"destination_port":["32400"]} {"source_port":["32400"]}` — **all present** | | cilium agent pods | **all three still the PRE-MERGE pods**, `startTime 2026-10-04T07:11Z` | | exported flow lines in agent stdout | **0, 0, 0** | | Flux Kustomizations | all `Ready=True` | Cause: the chart's `rollOutCiliumPods` defaults to **`false`** (cilium 1.20.1 `values.yaml`: *"Roll out cilium agent pods automatically when configmap is updated"*), so Helm rewrites the ConfigMap and leaves the DaemonSet's pod template untouched — and **cilium-agent reads that ConfigMap only at startup**. The feature was merged, believed delivered, green everywhere, and doing nothing at all. That is the same class of failure this repo already guards against elsewhere: a green signal sitting over something that never ran. - **`platform/cilium/helmrelease.yaml`** (authoritative): `rollOutCiliumPods: true`. - **`platform/cilium/values.yaml`** (reference mirror) and **`infra/ansible/roles/cilium/templates/cilium-values.yaml.j2`** (the `[parity]` source of truth): the same, so a bootstrap render does not differ from what Flux reconciles. **The cost is real and is the point.** Every future Cilium config change now triggers a cluster-wide agent rollout instead of waiting for someone to remember a manual restart. That is strictly more honest — the change applies when merged, or it fails visibly — but it makes config edits disruptive by default, so they want a quiet moment. Mitigation already in place: `updateStrategy.rollingUpdate.maxUnavailable: 1` means the DaemonSet will not proceed past an unready agent, so a bad render costs one node rather than the cluster. **Merging this is itself one rollout**, because adding the ConfigMap checksum annotation changes the pod template — and that rollout is what finally activates the Hubble exporter. ## What I verified - **Every row of the table above**, read from the live HelmRelease status, the live `cilium-config` ConfigMap, the agent pods' `startTime`, and a grep for `"flow"` in each agent's stdout. - **The chart value name, default and semantics**, quoted from cilium **v1.20.1**'s own `values.yaml` — the version actually running here (`quay.io/cilium/cilium:v1.20.1`), not a guess from another release: `rollOutCiliumPods: false` with the comment *"Roll out cilium agent pods automatically when configmap is updated."* - **All three value files carry the key** and both YAML files still parse, with `spec.values.rollOutCiliumPods` resolving to `true`. - **The blast-radius mitigation is genuinely present**, not assumed: `updateStrategy.rollingUpdate.maxUnavailable: 1` is already set in the HelmRelease, with a long existing comment explaining that a CNI rollout has no healthy-peer fallback. - **THE MECHANISM IS CONFIRMED BY CONTRAST, not just reasoned.** The cgroup fix (PR #42, 2026-10-03) DID take effect: both the DaemonSet and the running pod mount `cilium-cgroup hostPath=/sys/fs/cgroup`, matching what the HelmRelease declares. The difference is exactly the one claimed above — #42 changed the POD TEMPLATE (a volume hostPath), which rolls the DaemonSet on its own, while #50 changed ONLY the ConfigMap, which does not. It also explains the agents' `startTime 2026-10-04T07:11Z`: that was #42's rollout, and nothing has rolled them since. So the inertness is specific to ConfigMap-only changes, which is precisely what this flag fixes. - **Flux is healthy before this change** — every Kustomization `Ready=True`, so this lands on a clean graph rather than into an existing cascade. - **Toolbox gate**: recorded in the PR. ## What I did NOT verify - **THE OUTCOME, ENTIRELY. Unreconciled.** No agent has rolled, no flow has been exported. This change is an argument that the rollout will happen, not evidence that it did. - **Whether `filePath: stdout` actually works** — still the open question from #50, and it cannot be answered until the agents restart. If the literal is not special-cased, the agent may create a FILE named `stdout` and nothing reaches Loki: a silent no-op, not a crash. Fallback documented in #50: keep the chart's default path and add a `loki.source.file` target to Alloy with a hostPath mount. **Check for flow lines in the agent logs immediately after the rollout.** - **THE ROLLOUT'S IMPACT ON LAN DNS IS UNMEASURED AND IS THE MAIN RISK.** The node holding the L2 lease for the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP loses them briefly as it cycles. `maxUnavailable: 1` bounds it to one node at a time, and BPF maps persist across an agent restart, but I have not measured how long a VIP is unreachable during a lease move, nor whether the two PiHole replicas cover the gap. Merge it at a quiet moment. - **Whether enabling this is the operator's preferred trade.** The alternative — leaving it `false` and restarting by hand — keeps config edits non-disruptive at the cost of them being silently inert. I judged silent inertness the worse failure, but it is a judgement, not a measurement. - **Whether anything ELSE in `cilium-config` is currently inert.** I checked the one other recent Cilium change and it is fine (see below), but I did not audit every key against the running agents' actual state. - **`hubble.relay.rollOutPods` / `hubble.ui.rollOutPods`** are left at their `false` defaults. Both have the same silently-inert property for their own ConfigMaps; not addressed here.