feat(cilium): roll the agents when cilium-config changes (merged config was silently inert) #53

2026-10-06T02:19:41Z

gitea-admin commented

2026-10-06 02:19:41 +00:00

#50 merged a day ago and has been doing nothing. This makes it take effect, and closes the trap
that hid it.

A merged Cilium config change was silently inert

Measured 2026-10-05, over a day after #50
HelmRelease `cilium`	`Ready=True`, "Helm upgrade succeeded … cilium.v6"
`cilium-config`	`hubble-export-file-path: stdout` + the allowlist — present
cilium agent pods	all three still PRE-MERGE, `startTime 2026-10-04T07:11Z`
exported flow lines	0, 0, 0
Flux	every Kustomization `Ready=True`

The chart's rollOutCiliumPods defaults to false, so Helm rewrites the ConfigMap and leaves
the DaemonSet's pod template untouched — and cilium-agent reads that ConfigMap only at startup.
Merged, believed delivered, green everywhere, doing nothing. That's the same class of failure this
repo already guards against: a green signal over something that never ran.

Confirmed by contrast, not just reasoned: the cgroup fix (#42) did apply — the DaemonSet and
the running pod both mount /sys/fs/cgroup — because it changed the pod template, which rolls
the DaemonSet on its own. That also explains the agents' 2026-10-04T07:11Z start: that was #42's
rollout, and nothing has rolled them since. The inertness is specific to ConfigMap-only changes.

The change

rollOutCiliumPods: true in all three value files (HelmRelease authoritative, values.yaml mirror,
Ansible template) to hold the [parity] contract.

The cost is real and is the point. Every future Cilium config change now rolls the agents instead
of waiting for someone to remember a manual restart. Strictly more honest — it applies when merged,
or it fails visibly — but config edits become disruptive by default. maxUnavailable: 1 (already
set) bounds it to one node at a time and won't proceed past an unready agent.

Merging this is itself one rollout (the ConfigMap checksum annotation changes the pod template) —
and that rollout is what finally activates the Hubble exporter.

⚠️ Timing — merge at a quiet moment

The node holding the L2 lease for the Gateway VIP, PiHole's DNS VIP and Plex's VIP loses them
briefly as it cycles. maxUnavailable: 1 means one node at a time and BPF maps persist across an
agent restart, but I have not measured how long a VIP is unreachable during a lease move, nor
whether the two PiHole replicas cover the gap. This is the main risk.

Immediately after it rolls, check for flow lines in the agent logs:

kubectl -n kube-system logs <cilium-pod> -c cilium-agent | grep '"flow"'

If there are none, filePath: stdout isn't being special-cased — the fallback (chart default path +
an Alloy loki.source.file target with a hostPath mount) is documented in #50.

Gate

rc 3 — spec-immutability, decisions-log, lint, l0, render-golden, l1,
approval-policy all pass; rc 3 is the two known environmental skips. Not green — reported as
rc 3. No sensitive paths.

Unverified

Unreconciled — no agent has rolled, no flow exported. This is an argument that the rollout will
happen, not evidence that it did.
filePath: stdout remains unproven, and can't be answered until the restart.
The rollout's LAN-DNS impact is unmeasured (above).
Whether this is your preferred trade. The alternative is leaving it false and restarting by
hand — config edits stay non-disruptive, at the cost of being silently inert. I judged silent
inertness the worse failure, but that's a judgement, not a measurement.
hubble.relay.rollOutPods / hubble.ui.rollOutPods are left at their false defaults; they have
the same property for their own ConfigMaps.

🤖 Generated with Claude Code

**#50 merged a day ago and has been doing nothing.** This makes it take effect, and closes the trap that hid it. ## A merged Cilium config change was silently inert | Measured 2026-10-05, over a day after #50 | | |---|---| | HelmRelease `cilium` | `Ready=True`, *"Helm upgrade succeeded … cilium.v6"* | | `cilium-config` | `hubble-export-file-path: stdout` + the allowlist — **present** | | cilium agent pods | **all three still PRE-MERGE**, `startTime 2026-10-04T07:11Z` | | exported flow lines | **0, 0, 0** | | Flux | every Kustomization `Ready=True` | The chart's `rollOutCiliumPods` defaults to **`false`**, so Helm rewrites the ConfigMap and leaves the DaemonSet's pod template untouched — and **cilium-agent reads that ConfigMap only at startup**. Merged, believed delivered, green everywhere, doing nothing. That's the same class of failure this repo already guards against: a green signal over something that never ran. **Confirmed by contrast, not just reasoned:** the cgroup fix (#42) *did* apply — the DaemonSet and the running pod both mount `/sys/fs/cgroup` — because it changed the **pod template**, which rolls the DaemonSet on its own. That also explains the agents' `2026-10-04T07:11Z` start: that was #42's rollout, and nothing has rolled them since. The inertness is specific to **ConfigMap-only** changes. ## The change `rollOutCiliumPods: true` in all three value files (HelmRelease authoritative, `values.yaml` mirror, Ansible template) to hold the `[parity]` contract. **The cost is real and is the point.** Every future Cilium config change now rolls the agents instead of waiting for someone to remember a manual restart. Strictly more honest — it applies when merged, or it fails visibly — but config edits become disruptive by default. `maxUnavailable: 1` (already set) bounds it to one node at a time and won't proceed past an unready agent. **Merging this is itself one rollout** (the ConfigMap checksum annotation changes the pod template) — and that rollout is what finally activates the Hubble exporter. ## ⚠️ Timing — merge at a quiet moment The node holding the L2 lease for the **Gateway VIP, PiHole's DNS VIP and Plex's VIP** loses them briefly as it cycles. `maxUnavailable: 1` means one node at a time and BPF maps persist across an agent restart, but **I have not measured how long a VIP is unreachable during a lease move**, nor whether the two PiHole replicas cover the gap. This is the main risk. **Immediately after it rolls, check for flow lines** in the agent logs: ``` kubectl -n kube-system logs <cilium-pod> -c cilium-agent | grep '"flow"' ``` If there are none, `filePath: stdout` isn't being special-cased — the fallback (chart default path + an Alloy `loki.source.file` target with a hostPath mount) is documented in #50. ## Gate **rc 3** — `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` all pass; rc 3 is the two known environmental skips. **Not green — reported as rc 3.** No sensitive paths. ## Unverified - **Unreconciled** — no agent has rolled, no flow exported. This is an argument that the rollout will happen, not evidence that it did. - **`filePath: stdout` remains unproven**, and can't be answered until the restart. - **The rollout's LAN-DNS impact is unmeasured** (above). - **Whether this is your preferred trade.** The alternative is leaving it `false` and restarting by hand — config edits stay non-disruptive, at the cost of being silently inert. I judged silent inertness the worse failure, but that's a judgement, not a measurement. - `hubble.relay.rollOutPods` / `hubble.ui.rollOutPods` are left at their `false` defaults; they have the same property for their own ConfigMaps. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 2 commits 2026-10-06 02:19:41 +00:00

docs(obs): record that the gate-sidecar scrape works — four places still said it did not

gate / gate (pull_request) Successful in 1m48s

Details

gate / conformance (pull_request) Has been skipped

Details

81f627167d

PR #49 (merged 21c88b0) fixed the blocked scrape and it is verified working on prod
2026-10-05: all 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001-0.006s
against 10.0s and "context deadline exceeded" before. oauth2_proxy_* is in the TSDB for
these namespaces for the first time (5 metric names, 148 series).

Four places in the tree still asserted the pre-fix state as current, in the present tense,
in the files a reader consults first:

  platform/oauth2-proxy/podmonitor.yaml  "READ THIS BEFORE BELIEVING THE TARGETS ARE UP:
    THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ
    up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to
    disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers, with the
    diagnosis kept below it under an explicit HISTORY heading because it is why the fix is
    shaped the way it is.
  catalog/render.py:159  "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED".
  catalog/render.py:1365  the same claim on the sidecar arg.
  platform/oauth2-proxy/README.md  its subsections are dated and so are honest records, but
    a reader hits the stale body long before any correction. A SUPERSEDED pointer now sits
    under the heading and an Update 2026-10-05 section carries the before/after.

The "not scraped" claims are corrected; the "a flag alone is not enough" warning is KEPT
deliberately, because that part is still true and "flag without ingress allowance" is
exactly what caused the 63.7h outage.

Two substantive errors fixed while there, not just tense. The README's "three things are
needed" listed a Service port and a ServiceMonitor — wrong, a PodMonitor targets pod ports
directly, which is why no Service exposes 44180 and the renderer needed no Service change;
following it would have sent someone to edit Service rendering for nothing. And
podmonitor.yaml deferred the fix to "the catalog/renderer, in flight on another branch" —
no such branch existed; it now names r_monitoring_ingress_policy, with a new CRAFTY section
explaining the one namespace this file cannot help.

Comments only in render.py, so the rendered output is byte-identical: no apps/ file and no
golden changes, and therefore no sensitive path and no human approval needed — the opposite
of the PR this documents.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

feat(cilium): roll the agents when cilium-config changes (merged config was silently inert)

gate / gate (pull_request) Successful in 1m46s

Details

gate / conformance (pull_request) Has been skipped

Details

d0db63911a

A merged Cilium config change did nothing, and everything reported success.
The Hubble flow exporter (#50, merged 2026-10-04) never took effect:

  HelmRelease cilium   Ready=True, "Helm upgrade succeeded ... cilium.v6"
  cilium-config        hubble-export-file-path: stdout  + the allowlist, PRESENT
  cilium agent pods    all three still PRE-MERGE, startTime 2026-10-04T07:11Z
  exported flow lines  0, 0, 0
  Flux                 every Kustomization Ready=True

Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the
ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent
reads that ConfigMap only at STARTUP. Merged, believed delivered, green
everywhere, doing nothing. The same class of failure this repo already guards
against elsewhere: a green signal over something that never ran.

Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply --
both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed
the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why
the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has
rolled them since. The inertness is specific to ConfigMap-only changes.

Set in all three value files to hold the [parity] contract.

Cost, and it is the point: every future Cilium config change now rolls the
agents instead of waiting for a manual restart. Strictly more honest -- it
applies when merged or fails visibly -- but config edits become disruptive by
default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding
node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge
at a quiet moment.

Merging this IS one rollout (the ConfigMap checksum annotation changes the pod
template) -- which is what finally activates the Hubble exporter.

UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable
after the agents restart (check for flow lines immediately); and the rollout's
actual impact on LAN DNS is unmeasured.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 merged commit 22c94effff into main

2026-10-06 02:30:31 +00:00

jakman85 referenced this issue from a commit

2026-10-06 02:30:33 +00:00

Merge pull request 'feat(cilium): roll the agents when cilium-config changes (merged config was silently inert)' (#53) from feat/cilium-roll-out-on-config-change into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#53