feat(cilium): roll the agents when cilium-config changes (merged config was silently inert)
A merged Cilium config change did nothing, and everything reported success. The Hubble flow exporter (#50, merged 2026-10-04) never took effect: HelmRelease cilium Ready=True, "Helm upgrade succeeded ... cilium.v6" cilium-config hubble-export-file-path: stdout + the allowlist, PRESENT cilium agent pods all three still PRE-MERGE, startTime 2026-10-04T07:11Z exported flow lines 0, 0, 0 Flux every Kustomization Ready=True Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent reads that ConfigMap only at STARTUP. Merged, believed delivered, green everywhere, doing nothing. The same class of failure this repo already guards against elsewhere: a green signal over something that never ran. Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply -- both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has rolled them since. The inertness is specific to ConfigMap-only changes. Set in all three value files to hold the [parity] contract. Cost, and it is the point: every future Cilium config change now rolls the agents instead of waiting for a manual restart. Strictly more honest -- it applies when merged or fails visibly -- but config edits become disruptive by default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge at a quiet moment. Merging this IS one rollout (the ConfigMap checksum annotation changes the pod template) -- which is what finally activates the Hubble exporter. UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable after the agents restart (check for flow lines immediately); and the rollout's actual impact on LAN DNS is unmeasured. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
81f627167d
commit
d0db63911a
4 files changed
+134
No files matched your search
@@ -0,0 +1,88 @@
|
||||
---
|
||||
date: 2026-10-05
|
||||
branch: feat/cilium-roll-out-on-config-change
|
||||
title: Cilium config changes were silently inert — roll the agents when cilium-config changes, which also finally activates the Hubble exporter
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**A merged Cilium config change did nothing, and everything reported success.** The Hubble flow
|
||||
exporter (PR #50, merged 2026-10-04) wrote its settings into `cilium-config` and never took effect.
|
||||
|
||||
| Measured 2026-10-05, over a day after the merge | |
|
||||
|---|---|
|
||||
| HelmRelease `cilium` | `Ready=True`, `UpgradeSucceeded`, *"Helm upgrade succeeded for release kube-system/cilium.v6"* |
|
||||
| `cilium-config` keys | `hubble-export-file-path: stdout`, `hubble-export-allowlist: {"destination_port":["32400"]} {"source_port":["32400"]}` — **all present** |
|
||||
| cilium agent pods | **all three still the PRE-MERGE pods**, `startTime 2026-10-04T07:11Z` |
|
||||
| exported flow lines in agent stdout | **0, 0, 0** |
|
||||
| Flux Kustomizations | all `Ready=True` |
|
||||
|
||||
Cause: the chart's `rollOutCiliumPods` defaults to **`false`** (cilium 1.20.1 `values.yaml`: *"Roll
|
||||
out cilium agent pods automatically when configmap is updated"*), so Helm rewrites the ConfigMap and
|
||||
leaves the DaemonSet's pod template untouched — and **cilium-agent reads that ConfigMap only at
|
||||
startup**. The feature was merged, believed delivered, green everywhere, and doing nothing at all.
|
||||
|
||||
That is the same class of failure this repo already guards against elsewhere: a green signal sitting
|
||||
over something that never ran.
|
||||
|
||||
- **`platform/cilium/helmrelease.yaml`** (authoritative): `rollOutCiliumPods: true`.
|
||||
- **`platform/cilium/values.yaml`** (reference mirror) and
|
||||
**`infra/ansible/roles/cilium/templates/cilium-values.yaml.j2`** (the `[parity]` source of truth):
|
||||
the same, so a bootstrap render does not differ from what Flux reconciles.
|
||||
|
||||
**The cost is real and is the point.** Every future Cilium config change now triggers a cluster-wide
|
||||
agent rollout instead of waiting for someone to remember a manual restart. That is strictly more
|
||||
honest — the change applies when merged, or it fails visibly — but it makes config edits disruptive
|
||||
by default, so they want a quiet moment. Mitigation already in place:
|
||||
`updateStrategy.rollingUpdate.maxUnavailable: 1` means the DaemonSet will not proceed past an unready
|
||||
agent, so a bad render costs one node rather than the cluster.
|
||||
|
||||
**Merging this is itself one rollout**, because adding the ConfigMap checksum annotation changes the
|
||||
pod template — and that rollout is what finally activates the Hubble exporter.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **Every row of the table above**, read from the live HelmRelease status, the live `cilium-config`
|
||||
ConfigMap, the agent pods' `startTime`, and a grep for `"flow"` in each agent's stdout.
|
||||
- **The chart value name, default and semantics**, quoted from cilium **v1.20.1**'s own
|
||||
`values.yaml` — the version actually running here (`quay.io/cilium/cilium:v1.20.1`), not a guess
|
||||
from another release: `rollOutCiliumPods: false` with the comment *"Roll out cilium agent pods
|
||||
automatically when configmap is updated."*
|
||||
- **All three value files carry the key** and both YAML files still parse, with
|
||||
`spec.values.rollOutCiliumPods` resolving to `true`.
|
||||
- **The blast-radius mitigation is genuinely present**, not assumed:
|
||||
`updateStrategy.rollingUpdate.maxUnavailable: 1` is already set in the HelmRelease, with a long
|
||||
existing comment explaining that a CNI rollout has no healthy-peer fallback.
|
||||
- **THE MECHANISM IS CONFIRMED BY CONTRAST, not just reasoned.** The cgroup fix (PR #42,
|
||||
2026-10-03) DID take effect: both the DaemonSet and the running pod mount
|
||||
`cilium-cgroup hostPath=/sys/fs/cgroup`, matching what the HelmRelease declares. The difference is
|
||||
exactly the one claimed above — #42 changed the POD TEMPLATE (a volume hostPath), which rolls the
|
||||
DaemonSet on its own, while #50 changed ONLY the ConfigMap, which does not. It also explains the
|
||||
agents' `startTime 2026-10-04T07:11Z`: that was #42's rollout, and nothing has rolled them since.
|
||||
So the inertness is specific to ConfigMap-only changes, which is precisely what this flag fixes.
|
||||
- **Flux is healthy before this change** — every Kustomization `Ready=True`, so this lands on a clean
|
||||
graph rather than into an existing cascade.
|
||||
- **Toolbox gate**: recorded in the PR.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **THE OUTCOME, ENTIRELY. Unreconciled.** No agent has rolled, no flow has been exported. This
|
||||
change is an argument that the rollout will happen, not evidence that it did.
|
||||
- **Whether `filePath: stdout` actually works** — still the open question from #50, and it cannot be
|
||||
answered until the agents restart. If the literal is not special-cased, the agent may create a
|
||||
FILE named `stdout` and nothing reaches Loki: a silent no-op, not a crash. Fallback documented in
|
||||
#50: keep the chart's default path and add a `loki.source.file` target to Alloy with a hostPath
|
||||
mount. **Check for flow lines in the agent logs immediately after the rollout.**
|
||||
- **THE ROLLOUT'S IMPACT ON LAN DNS IS UNMEASURED AND IS THE MAIN RISK.** The node holding the L2
|
||||
lease for the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP loses them briefly as it cycles.
|
||||
`maxUnavailable: 1` bounds it to one node at a time, and BPF maps persist across an agent restart,
|
||||
but I have not measured how long a VIP is unreachable during a lease move, nor whether the two
|
||||
PiHole replicas cover the gap. Merge it at a quiet moment.
|
||||
- **Whether enabling this is the operator's preferred trade.** The alternative — leaving it `false`
|
||||
and restarting by hand — keeps config edits non-disruptive at the cost of them being silently
|
||||
inert. I judged silent inertness the worse failure, but it is a judgement, not a measurement.
|
||||
- **Whether anything ELSE in `cilium-config` is currently inert.** I checked the one other recent
|
||||
Cilium change and it is fine (see below), but I did not audit every key against the running
|
||||
agents' actual state.
|
||||
- **`hubble.relay.rollOutPods` / `hubble.ui.rollOutPods`** are left at their `false` defaults. Both
|
||||
have the same silently-inert property for their own ConfigMaps; not addressed here.
|
||||
@@ -39,6 +39,12 @@ bpf:
|
||||
masquerade: true
|
||||
|
||||
# Flow observability / isolation audit (ADR-0002).
|
||||
# Roll the agents when cilium-config changes. Kept EQUAL to the Flux values
|
||||
# (platform/cilium/helmrelease.yaml), which carry the full rationale: the chart default of false
|
||||
# leaves the pod template untouched on a config change, and cilium-agent reads that ConfigMap only
|
||||
# at startup, so the change is silently inert until someone restarts the DaemonSet by hand.
|
||||
rollOutCiliumPods: true
|
||||
|
||||
hubble:
|
||||
enabled: true
|
||||
relay:
|
||||
|
||||
@@ -188,6 +188,37 @@ spec:
|
||||
enabled: false
|
||||
hostRoot: /sys/fs/cgroup
|
||||
# --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
|
||||
# --- Roll the agents when their ConfigMap changes (2026-10-05) ---
|
||||
# [parity] MUST equal Ansible: rollOutCiliumPods: true
|
||||
#
|
||||
# WHY: a merged Cilium config change DID NOTHING. The chart default is `false`
|
||||
# (cilium 1.20.1 values.yaml: "Roll out cilium agent pods automatically when configmap is
|
||||
# updated. rollOutCiliumPods: false"), so Helm rewrites `cilium-config` and leaves the
|
||||
# DaemonSet's pod template untouched — and cilium-agent reads that ConfigMap only at STARTUP.
|
||||
# The result is a silently inert change: Flux green, HelmRelease `UpgradeSucceeded`, the new
|
||||
# keys visibly present in the ConfigMap, and the running agents still on the old settings
|
||||
# until a human remembers to restart them.
|
||||
#
|
||||
# MEASURED, and it is why this exists: the Hubble flow exporter merged 2026-10-04 wrote
|
||||
# `hubble-export-file-path: stdout` and its allowlist into `cilium-config`, the HelmRelease
|
||||
# reported `Helm upgrade succeeded ... cilium.v6`, and over a DAY later all three agents were
|
||||
# still the pre-merge pods (startTime 2026-10-04T07:11Z) with ZERO exported flow lines. The
|
||||
# feature was merged, believed delivered, and doing nothing at all. That is the same class of
|
||||
# failure this repo already documents elsewhere — a green signal over something that never ran.
|
||||
#
|
||||
# THE COST IS REAL AND IS THE POINT: every future Cilium config change now triggers a
|
||||
# cluster-wide agent rollout instead of waiting for a manual restart. That is strictly more
|
||||
# honest — the change applies when it is merged, or it fails visibly — but it means config
|
||||
# edits are disruptive by default, so they want a quiet moment. One node at a time:
|
||||
# `updateStrategy.rollingUpdate.maxUnavailable: 1` below means the DaemonSet will not proceed
|
||||
# past an unready agent, so a bad render costs one node rather than the cluster. Note that the
|
||||
# node holding the L2 lease for the Gateway, PiHole-DNS and Plex VIPs loses them briefly as it
|
||||
# cycles, so a rollout can blip LAN DNS.
|
||||
#
|
||||
# ENABLING THIS IS ITSELF ONE ROLLOUT: adding the ConfigMap checksum annotation changes the pod
|
||||
# template, so merging this rolls the agents once — which is what finally activates the Hubble
|
||||
# exporter that has been inert since 2026-10-04.
|
||||
rollOutCiliumPods: true
|
||||
# [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
|
||||
# The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
|
||||
# maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad
|
||||
|
||||
@@ -32,6 +32,15 @@ enableIPv4Masquerade: true
|
||||
bpf:
|
||||
masquerade: true
|
||||
# [parity] flow observability / isolation audit
|
||||
# [parity] roll the agents when cilium-config changes. The chart default (false) rewrites the
|
||||
# ConfigMap but leaves the pod template alone, and cilium-agent only reads it at STARTUP — so a
|
||||
# merged config change is SILENTLY INERT. Measured 2026-10-04..05: the Hubble exporter keys were
|
||||
# live in cilium-config and the HelmRelease said UpgradeSucceeded, while all three agents were
|
||||
# still pre-merge pods exporting nothing for over a day. Cost: config edits now roll the agents
|
||||
# (one node at a time via maxUnavailable: 1), which can briefly blip the VIPs on the lease-holding
|
||||
# node. Full rationale in helmrelease.yaml.
|
||||
rollOutCiliumPods: true
|
||||
|
||||
hubble:
|
||||
enabled: true
|
||||
relay:
|
||||
|
||||
Reference in new issue
Block a user