diff --git a/decisions/2026-10-05-cilium-roll-out-on-config-change.md b/decisions/2026-10-05-cilium-roll-out-on-config-change.md new file mode 100644 index 0000000..9b3b25e --- /dev/null +++ b/decisions/2026-10-05-cilium-roll-out-on-config-change.md @@ -0,0 +1,88 @@ +--- +date: 2026-10-05 +branch: feat/cilium-roll-out-on-config-change +title: Cilium config changes were silently inert — roll the agents when cilium-config changes, which also finally activates the Hubble exporter +--- + +## What changed + +**A merged Cilium config change did nothing, and everything reported success.** The Hubble flow +exporter (PR #50, merged 2026-10-04) wrote its settings into `cilium-config` and never took effect. + +| Measured 2026-10-05, over a day after the merge | | +|---|---| +| HelmRelease `cilium` | `Ready=True`, `UpgradeSucceeded`, *"Helm upgrade succeeded for release kube-system/cilium.v6"* | +| `cilium-config` keys | `hubble-export-file-path: stdout`, `hubble-export-allowlist: {"destination_port":["32400"]} {"source_port":["32400"]}` — **all present** | +| cilium agent pods | **all three still the PRE-MERGE pods**, `startTime 2026-10-04T07:11Z` | +| exported flow lines in agent stdout | **0, 0, 0** | +| Flux Kustomizations | all `Ready=True` | + +Cause: the chart's `rollOutCiliumPods` defaults to **`false`** (cilium 1.20.1 `values.yaml`: *"Roll +out cilium agent pods automatically when configmap is updated"*), so Helm rewrites the ConfigMap and +leaves the DaemonSet's pod template untouched — and **cilium-agent reads that ConfigMap only at +startup**. The feature was merged, believed delivered, green everywhere, and doing nothing at all. + +That is the same class of failure this repo already guards against elsewhere: a green signal sitting +over something that never ran. + +- **`platform/cilium/helmrelease.yaml`** (authoritative): `rollOutCiliumPods: true`. +- **`platform/cilium/values.yaml`** (reference mirror) and + **`infra/ansible/roles/cilium/templates/cilium-values.yaml.j2`** (the `[parity]` source of truth): + the same, so a bootstrap render does not differ from what Flux reconciles. + +**The cost is real and is the point.** Every future Cilium config change now triggers a cluster-wide +agent rollout instead of waiting for someone to remember a manual restart. That is strictly more +honest — the change applies when merged, or it fails visibly — but it makes config edits disruptive +by default, so they want a quiet moment. Mitigation already in place: +`updateStrategy.rollingUpdate.maxUnavailable: 1` means the DaemonSet will not proceed past an unready +agent, so a bad render costs one node rather than the cluster. + +**Merging this is itself one rollout**, because adding the ConfigMap checksum annotation changes the +pod template — and that rollout is what finally activates the Hubble exporter. + +## What I verified + +- **Every row of the table above**, read from the live HelmRelease status, the live `cilium-config` + ConfigMap, the agent pods' `startTime`, and a grep for `"flow"` in each agent's stdout. +- **The chart value name, default and semantics**, quoted from cilium **v1.20.1**'s own + `values.yaml` — the version actually running here (`quay.io/cilium/cilium:v1.20.1`), not a guess + from another release: `rollOutCiliumPods: false` with the comment *"Roll out cilium agent pods + automatically when configmap is updated."* +- **All three value files carry the key** and both YAML files still parse, with + `spec.values.rollOutCiliumPods` resolving to `true`. +- **The blast-radius mitigation is genuinely present**, not assumed: + `updateStrategy.rollingUpdate.maxUnavailable: 1` is already set in the HelmRelease, with a long + existing comment explaining that a CNI rollout has no healthy-peer fallback. +- **THE MECHANISM IS CONFIRMED BY CONTRAST, not just reasoned.** The cgroup fix (PR #42, + 2026-10-03) DID take effect: both the DaemonSet and the running pod mount + `cilium-cgroup hostPath=/sys/fs/cgroup`, matching what the HelmRelease declares. The difference is + exactly the one claimed above — #42 changed the POD TEMPLATE (a volume hostPath), which rolls the + DaemonSet on its own, while #50 changed ONLY the ConfigMap, which does not. It also explains the + agents' `startTime 2026-10-04T07:11Z`: that was #42's rollout, and nothing has rolled them since. + So the inertness is specific to ConfigMap-only changes, which is precisely what this flag fixes. +- **Flux is healthy before this change** — every Kustomization `Ready=True`, so this lands on a clean + graph rather than into an existing cascade. +- **Toolbox gate**: recorded in the PR. + +## What I did NOT verify + +- **THE OUTCOME, ENTIRELY. Unreconciled.** No agent has rolled, no flow has been exported. This + change is an argument that the rollout will happen, not evidence that it did. +- **Whether `filePath: stdout` actually works** — still the open question from #50, and it cannot be + answered until the agents restart. If the literal is not special-cased, the agent may create a + FILE named `stdout` and nothing reaches Loki: a silent no-op, not a crash. Fallback documented in + #50: keep the chart's default path and add a `loki.source.file` target to Alloy with a hostPath + mount. **Check for flow lines in the agent logs immediately after the rollout.** +- **THE ROLLOUT'S IMPACT ON LAN DNS IS UNMEASURED AND IS THE MAIN RISK.** The node holding the L2 + lease for the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP loses them briefly as it cycles. + `maxUnavailable: 1` bounds it to one node at a time, and BPF maps persist across an agent restart, + but I have not measured how long a VIP is unreachable during a lease move, nor whether the two + PiHole replicas cover the gap. Merge it at a quiet moment. +- **Whether enabling this is the operator's preferred trade.** The alternative — leaving it `false` + and restarting by hand — keeps config edits non-disruptive at the cost of them being silently + inert. I judged silent inertness the worse failure, but it is a judgement, not a measurement. +- **Whether anything ELSE in `cilium-config` is currently inert.** I checked the one other recent + Cilium change and it is fine (see below), but I did not audit every key against the running + agents' actual state. +- **`hubble.relay.rollOutPods` / `hubble.ui.rollOutPods`** are left at their `false` defaults. Both + have the same silently-inert property for their own ConfigMaps; not addressed here. diff --git a/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 b/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 index 01f7f17..24be12f 100644 --- a/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 +++ b/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 @@ -39,6 +39,12 @@ bpf: masquerade: true # Flow observability / isolation audit (ADR-0002). +# Roll the agents when cilium-config changes. Kept EQUAL to the Flux values +# (platform/cilium/helmrelease.yaml), which carry the full rationale: the chart default of false +# leaves the pod template untouched on a config change, and cilium-agent reads that ConfigMap only +# at startup, so the change is silently inert until someone restarts the DaemonSet by hand. +rollOutCiliumPods: true + hubble: enabled: true relay: diff --git a/platform/cilium/helmrelease.yaml b/platform/cilium/helmrelease.yaml index c1f2b65..c61df80 100644 --- a/platform/cilium/helmrelease.yaml +++ b/platform/cilium/helmrelease.yaml @@ -188,6 +188,37 @@ spec: enabled: false hostRoot: /sys/fs/cgroup # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident --- + # --- Roll the agents when their ConfigMap changes (2026-10-05) --- + # [parity] MUST equal Ansible: rollOutCiliumPods: true + # + # WHY: a merged Cilium config change DID NOTHING. The chart default is `false` + # (cilium 1.20.1 values.yaml: "Roll out cilium agent pods automatically when configmap is + # updated. rollOutCiliumPods: false"), so Helm rewrites `cilium-config` and leaves the + # DaemonSet's pod template untouched — and cilium-agent reads that ConfigMap only at STARTUP. + # The result is a silently inert change: Flux green, HelmRelease `UpgradeSucceeded`, the new + # keys visibly present in the ConfigMap, and the running agents still on the old settings + # until a human remembers to restart them. + # + # MEASURED, and it is why this exists: the Hubble flow exporter merged 2026-10-04 wrote + # `hubble-export-file-path: stdout` and its allowlist into `cilium-config`, the HelmRelease + # reported `Helm upgrade succeeded ... cilium.v6`, and over a DAY later all three agents were + # still the pre-merge pods (startTime 2026-10-04T07:11Z) with ZERO exported flow lines. The + # feature was merged, believed delivered, and doing nothing at all. That is the same class of + # failure this repo already documents elsewhere — a green signal over something that never ran. + # + # THE COST IS REAL AND IS THE POINT: every future Cilium config change now triggers a + # cluster-wide agent rollout instead of waiting for a manual restart. That is strictly more + # honest — the change applies when it is merged, or it fails visibly — but it means config + # edits are disruptive by default, so they want a quiet moment. One node at a time: + # `updateStrategy.rollingUpdate.maxUnavailable: 1` below means the DaemonSet will not proceed + # past an unready agent, so a bad render costs one node rather than the cluster. Note that the + # node holding the L2 lease for the Gateway, PiHole-DNS and Plex VIPs loses them briefly as it + # cycles, so a rollout can blip LAN DNS. + # + # ENABLING THIS IS ITSELF ONE ROLLOUT: adding the ConfigMap checksum annotation changes the pod + # template, so merging this rolls the agents once — which is what finally activates the Hubble + # exporter that has been inert since 2026-10-04. + rollOutCiliumPods: true # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1. # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate. # maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad diff --git a/platform/cilium/values.yaml b/platform/cilium/values.yaml index b37d11e..dba78a3 100644 --- a/platform/cilium/values.yaml +++ b/platform/cilium/values.yaml @@ -32,6 +32,15 @@ enableIPv4Masquerade: true bpf: masquerade: true # [parity] flow observability / isolation audit +# [parity] roll the agents when cilium-config changes. The chart default (false) rewrites the +# ConfigMap but leaves the pod template alone, and cilium-agent only reads it at STARTUP — so a +# merged config change is SILENTLY INERT. Measured 2026-10-04..05: the Hubble exporter keys were +# live in cilium-config and the HelmRelease said UpgradeSucceeded, while all three agents were +# still pre-merge pods exporting nothing for over a day. Cost: config edits now roll the agents +# (one node at a time via maxUnavailable: 1), which can briefly blip the VIPs on the lease-holding +# node. Full rationale in helmrelease.yaml. +rollOutCiliumPods: true + hubble: enabled: true relay: