feat(cilium): roll the agents when cilium-config changes (merged config was silently inert)
gate / gate (pull_request) Successful in 1m46s
gate / conformance (pull_request) Has been skipped

A merged Cilium config change did nothing, and everything reported success.
The Hubble flow exporter (#50, merged 2026-10-04) never took effect:

  HelmRelease cilium   Ready=True, "Helm upgrade succeeded ... cilium.v6"
  cilium-config        hubble-export-file-path: stdout  + the allowlist, PRESENT
  cilium agent pods    all three still PRE-MERGE, startTime 2026-10-04T07:11Z
  exported flow lines  0, 0, 0
  Flux                 every Kustomization Ready=True

Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the
ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent
reads that ConfigMap only at STARTUP. Merged, believed delivered, green
everywhere, doing nothing. The same class of failure this repo already guards
against elsewhere: a green signal over something that never ran.

Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply --
both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed
the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why
the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has
rolled them since. The inertness is specific to ConfigMap-only changes.

Set in all three value files to hold the [parity] contract.

Cost, and it is the point: every future Cilium config change now rolls the
agents instead of waiting for a manual restart. Strictly more honest -- it
applies when merged or fails visibly -- but config edits become disruptive by
default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding
node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge
at a quiet moment.

Merging this IS one rollout (the ConfigMap checksum annotation changes the pod
template) -- which is what finally activates the Hubble exporter.

UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable
after the agents restart (check for flow lines immediately); and the rollout's
actual impact on LAN DNS is unmeasured.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-05 21:19:13 -05:00
1 parent 81f627167d
commit d0db63911a
4 files changed
+134

No files matched your search

+31
View File
@@ -188,6 +188,37 @@ spec:
enabled: false
hostRoot: /sys/fs/cgroup
# --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
# --- Roll the agents when their ConfigMap changes (2026-10-05) ---
# [parity] MUST equal Ansible: rollOutCiliumPods: true
#
# WHY: a merged Cilium config change DID NOTHING. The chart default is `false`
# (cilium 1.20.1 values.yaml: "Roll out cilium agent pods automatically when configmap is
# updated. rollOutCiliumPods: false"), so Helm rewrites `cilium-config` and leaves the
# DaemonSet's pod template untouched — and cilium-agent reads that ConfigMap only at STARTUP.
# The result is a silently inert change: Flux green, HelmRelease `UpgradeSucceeded`, the new
# keys visibly present in the ConfigMap, and the running agents still on the old settings
# until a human remembers to restart them.
#
# MEASURED, and it is why this exists: the Hubble flow exporter merged 2026-10-04 wrote
# `hubble-export-file-path: stdout` and its allowlist into `cilium-config`, the HelmRelease
# reported `Helm upgrade succeeded ... cilium.v6`, and over a DAY later all three agents were
# still the pre-merge pods (startTime 2026-10-04T07:11Z) with ZERO exported flow lines. The
# feature was merged, believed delivered, and doing nothing at all. That is the same class of
# failure this repo already documents elsewhere — a green signal over something that never ran.
#
# THE COST IS REAL AND IS THE POINT: every future Cilium config change now triggers a
# cluster-wide agent rollout instead of waiting for a manual restart. That is strictly more
# honest — the change applies when it is merged, or it fails visibly — but it means config
# edits are disruptive by default, so they want a quiet moment. One node at a time:
# `updateStrategy.rollingUpdate.maxUnavailable: 1` below means the DaemonSet will not proceed
# past an unready agent, so a bad render costs one node rather than the cluster. Note that the
# node holding the L2 lease for the Gateway, PiHole-DNS and Plex VIPs loses them briefly as it
# cycles, so a rollout can blip LAN DNS.
#
# ENABLING THIS IS ITSELF ONE ROLLOUT: adding the ConfigMap checksum annotation changes the pod
# template, so merging this rolls the agents once — which is what finally activates the Hubble
# exporter that has been inert since 2026-10-04.
rollOutCiliumPods: true
# [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
# The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
# maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad
+9
View File
@@ -32,6 +32,15 @@ enableIPv4Masquerade: true
bpf:
masquerade: true
# [parity] flow observability / isolation audit
# [parity] roll the agents when cilium-config changes. The chart default (false) rewrites the
# ConfigMap but leaves the pod template alone, and cilium-agent only reads it at STARTUP — so a
# merged config change is SILENTLY INERT. Measured 2026-10-04..05: the Hubble exporter keys were
# live in cilium-config and the HelmRelease said UpgradeSucceeded, while all three agents were
# still pre-merge pods exporting nothing for over a day. Cost: config edits now roll the agents
# (one node at a time via maxUnavailable: 1), which can briefly blip the VIPs on the lease-holding
# node. Full rationale in helmrelease.yaml.
rollOutCiliumPods: true
hubble:
enabled: true
relay: