A merged Cilium config change did nothing, and everything reported success.
The Hubble flow exporter (#50, merged 2026-10-04) never took effect:
HelmRelease cilium Ready=True, "Helm upgrade succeeded ... cilium.v6"
cilium-config hubble-export-file-path: stdout + the allowlist, PRESENT
cilium agent pods all three still PRE-MERGE, startTime 2026-10-04T07:11Z
exported flow lines 0, 0, 0
Flux every Kustomization Ready=True
Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the
ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent
reads that ConfigMap only at STARTUP. Merged, believed delivered, green
everywhere, doing nothing. The same class of failure this repo already guards
against elsewhere: a green signal over something that never ran.
Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply --
both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed
the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why
the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has
rolled them since. The inertness is specific to ConfigMap-only changes.
Set in all three value files to hold the [parity] contract.
Cost, and it is the point: every future Cilium config change now rolls the
agents instead of waiting for a manual restart. Strictly more honest -- it
applies when merged or fails visibly -- but config edits become disruptive by
default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding
node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge
at a quiet moment.
Merging this IS one rollout (the ConfigMap checksum annotation changes the pod
template) -- which is what finally activates the Hubble exporter.
UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable
after the agents restart (check for flow lines immediately); and the rollout's
actual impact on LAN DNS is unmeasured.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PR #49 (merged 21c88b0) fixed the blocked scrape and it is verified working on prod
2026-10-05: all 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001-0.006s
against 10.0s and "context deadline exceeded" before. oauth2_proxy_* is in the TSDB for
these namespaces for the first time (5 metric names, 148 series).
Four places in the tree still asserted the pre-fix state as current, in the present tense,
in the files a reader consults first:
platform/oauth2-proxy/podmonitor.yaml "READ THIS BEFORE BELIEVING THE TARGETS ARE UP:
THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ
up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to
disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers, with the
diagnosis kept below it under an explicit HISTORY heading because it is why the fix is
shaped the way it is.
catalog/render.py:159 "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED".
catalog/render.py:1365 the same claim on the sidecar arg.
platform/oauth2-proxy/README.md its subsections are dated and so are honest records, but
a reader hits the stale body long before any correction. A SUPERSEDED pointer now sits
under the heading and an Update 2026-10-05 section carries the before/after.
The "not scraped" claims are corrected; the "a flag alone is not enough" warning is KEPT
deliberately, because that part is still true and "flag without ingress allowance" is
exactly what caused the 63.7h outage.
Two substantive errors fixed while there, not just tense. The README's "three things are
needed" listed a Service port and a ServiceMonitor — wrong, a PodMonitor targets pod ports
directly, which is why no Service exposes 44180 and the renderer needed no Service change;
following it would have sent someone to edit Service rendering for nothing. And
podmonitor.yaml deferred the fix to "the catalog/renderer, in flight on another branch" —
no such branch existed; it now names r_monitoring_ingress_policy, with a new CRAFTY section
explaining the one namespace this file cannot help.
Comments only in render.py, so the rendered output is byte-identical: no apps/ file and no
golden changes, and therefore no sensitive path and no human approval needed — the opposite
of the PR this documents.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85andClaude Opus 5