Compare commits

...
Author SHA1 Message Date
jakman85andClaude Opus 5 2cb891c05d docs(agents): one workspace per agent run — the shared checkout is off-limits
gate / gate (pull_request) Successful in 1m48s
gate / conformance (pull_request) Has been skipped
Agents have been working in the operator's checkout and clobbering each other.
Nothing in the tree told them not to: worktrees appeared in CLAUDE.md only as
§3's gate workarounds, and decisions/README.md described worktree isolation as
something agents already do rather than as a rule.

New CLAUDE.md §0 states the rule, with the commands and an explicit do-not list
for the shared checkout. Placed before §1 so every existing §-reference in the
repo keeps its meaning.

Measured in the shared checkout on 2026-10-05, each row a distinct failure mode:
107 `checkout: moving` reflog entries; sitting on feat/pm-agent-issue-to-pr, not
main; two other runs' untracked trees left in it; local main two commits behind
origin/main; 42 registered worktrees of which 31 are on already-merged branches;
9 gate scripts carrying a `.claude/` exclusion because an in-tree worktree made
find-based discovery count the same manifest up to 18 times.

Corrects a claim carried in earlier briefs: `git worktree remove` is NOT blocked
by the agent sandbox. Provoked on a throwaway worktree — rc 0.

§3 reframed as the normal path and grown from two breakages to three:
spec-immutability's skip is avoidable, not merely tolerated, with the spec repo
mounted read-only in the same ENGINE shim plus -e SPEC_REPO=/home-network-k3s.

CLAUDE.md is a sensitive path, so this PR needs a non-author review or an
administrator /approve-override posted as a PR review. Not posted by me: §1
forbids an agent clearing that on its own PR.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-05 22:31:40 -05:00
jakman85 22c94effff Merge pull request 'feat(cilium): roll the agents when cilium-config changes (merged config was silently inert)' (#53) from feat/cilium-roll-out-on-config-change into main
gate / gate (push) Failing after 1m45s
gate / conformance (push) Has been skipped
Reviewed-on: #53
2026-10-06 02:30:31 +00:00
jakman85andClaude Opus 5 d0db63911a feat(cilium): roll the agents when cilium-config changes (merged config was silently inert)
gate / gate (pull_request) Successful in 1m46s
gate / conformance (pull_request) Has been skipped
A merged Cilium config change did nothing, and everything reported success.
The Hubble flow exporter (#50, merged 2026-10-04) never took effect:

  HelmRelease cilium   Ready=True, "Helm upgrade succeeded ... cilium.v6"
  cilium-config        hubble-export-file-path: stdout  + the allowlist, PRESENT
  cilium agent pods    all three still PRE-MERGE, startTime 2026-10-04T07:11Z
  exported flow lines  0, 0, 0
  Flux                 every Kustomization Ready=True

Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the
ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent
reads that ConfigMap only at STARTUP. Merged, believed delivered, green
everywhere, doing nothing. The same class of failure this repo already guards
against elsewhere: a green signal over something that never ran.

Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply --
both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed
the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why
the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has
rolled them since. The inertness is specific to ConfigMap-only changes.

Set in all three value files to hold the [parity] contract.

Cost, and it is the point: every future Cilium config change now rolls the
agents instead of waiting for a manual restart. Strictly more honest -- it
applies when merged or fails visibly -- but config edits become disruptive by
default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding
node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge
at a quiet moment.

Merging this IS one rollout (the ConfigMap checksum annotation changes the pod
template) -- which is what finally activates the Hubble exporter.

UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable
after the agents restart (check for flow lines immediately); and the rollout's
actual impact on LAN DNS is unmeasured.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-05 21:19:13 -05:00
8 changed files with 437 additions and 6 deletions

No files matched your search

+137 -5
View File
@@ -4,6 +4,125 @@ Read this before doing anything durable. It is the delivery contract for AI codi
sessions and agents. The newest entry in `decisions/` is the current PM handoff and
takes precedence on *what* to work on; this file governs *how* it lands.
**Start with §0.** It is first because it is the rule you can break before you have read
anything else: the very first `git checkout -b` in the wrong directory has already done the
damage.
## 0. Work in your OWN workspace. Never in the shared checkout.
Before you create a branch, edit a file, or run `git` anything: **make yourself a git
worktree outside the repository root, and do all of your work there.** One workspace per
agent run, no exceptions.
# from the shared checkout, ONCE, at the start of your run
git fetch origin main
git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
cd /path/outside/the/repo/wt-<slug> # everything after this happens HERE
`/path/outside/the/repo` means outside this repository's tree — your session scratchpad,
or a sibling like `/var/home/keith/source/wt-<slug>`. Branch from **`origin/main`**, never
from `main` and never from `HEAD`; see "stale base" below.
**The shared checkout is `/var/home/keith/source/home-network-k3s-impl`** — the operator's
working copy, the one you are probably started in. In it, do **not**:
- **`git checkout` / `git switch` / `git checkout -b`.** Its HEAD is shared state; moving
it moves the tree out from under whoever else is using it.
- **edit, create or delete a file.** Including "just one doc", including a file you intend
to move later.
- **`git add` / `git commit` / `git stash` / `git restore`.** `git add -A` there stages
whatever another agent left lying around; `git stash` *removes* their uncommitted work
into a place they will not look.
- **`git worktree remove` anything you did not create.**
Read-only commands there are fine, and the one write you do need is creating your own
worktree.
### Why — measured in the shared checkout, 2026-10-05
| What was measured | Value |
|---|---|
| `checkout: moving` entries in its reflog | **107** — the tree has been yanked between branches 107 times |
| the branch it was sitting on | `feat/pm-agent-issue-to-pr` — **not `main`** |
| uncommitted trees left in it by other runs | **2** (`infra/agent/`, `platform/pm-agent/`), mtimes five minutes apart the same evening |
| its local `main` vs `origin/main` | **two commits behind** (`4386d14` vs `22c94ef`) |
| registered worktrees | **42**, of which **31** are on branches already merged to `main` |
| gate scripts carrying an explicit `.claude/` exclusion | **9** |
Each row is a distinct way the shared checkout damages work:
- **Dirty tree, foreign files.** Two other runs' uncommitted directories were sitting in
it. An agent that edits there and runs `git add -A` ships someone else's half-finished
work inside its own PR, and the PR's author has no idea what those files are. This is not
hypothetical and it is not historical: **eleven minutes after those two directories were
measured, while this very section was being written, another run committed them** —
`5334abe feat(pm-agent): ...`, 1,000+ lines across `infra/agent/`, `clusters/`, `catalog/`
and `apps/` — onto the shared checkout's checked-out branch, moving its HEAD under anyone
reading it. Had this rule's author been working in that tree rather than in a worktree,
that commit and this one would have collided.
- **Stale base.** Its `main` was two commits behind `origin/main`, so
`git checkout -b x main` there branches off a tree that is already not what CI will merge
into — the gate you run locally proves a base nobody else has. Worse, it was not even *on*
`main`: a bare `git checkout -b x` would have branched off `feat/pm-agent-issue-to-pr` and
silently inherited that work into your PR. `git fetch` first, and name `origin/main`
explicitly as the start point.
- **Shared HEAD.** 107 branch switches in one tree is 107 chances to switch it while
another agent is mid-edit. The loser's changes are either carried onto the wrong branch
or lost to a checkout conflict — and the symptom surfaces in *their* session, not yours.
### Put the workspace OUTSIDE the repo root
A worktree *inside* the tree (`.claude/worktrees/agent-*/` — which is where Claude Code's
own default puts it) is a full second copy of the repo that every tree-walking check then
finds. Measured 2026-09-12: `find`-based discovery reported **the same manifest under test
once per worktree, up to 18 times**, and the L0 check comparing that listing against the
tree failed on paths that are not authored surface at all
(`platform/monitoring/tests/run.sh:115-123`). Nine gate scripts now carry an explicit
`.claude/` prune for this reason — which means the hazard is *worked around*, not fixed,
and the next check that walks the tree will hit it again. Keep the copy out of the tree and
none of that applies.
### Clean up — and expect not to be able to
31 of the 42 registered worktrees are on branches already merged to `main`. A registered
worktree **pins its branch**: while it exists, that branch cannot be deleted or checked out
anywhere else. So a left-behind workspace is a lock on a branch name, and the cost lands on
whoever next needs it.
Remove yours when you are done:
git worktree remove /path/outside/the/repo/wt-<slug>
Measured 2026-10-05: on a worktree **you created yourself, in the session scratchpad**, that
command **succeeds** — rc 0, no prompt, with or without a branch checked out. The "worktree
removal is sandbox-blocked" note in circulation is about the `.claude/worktrees/agent-*` ones
belonging to *other* sessions, some of which are additionally `locked` on git's own terms. So:
remove yours, and only yours. If removal is refused, say so in your handoff and leave the
worktree registered — do **not** `rm -rf` the directory, which leaves a registered worktree
pointing at nothing and still holding the branch.
If someone *else's* stale worktree is holding the branch you need, do not try to reclaim the
branch name. Push a `rework/<pr-number>` branch to the PR's refspec instead.
### If you are already standing in the shared checkout with edits
Stop. Do not `git stash` (that moves other agents' work too). Create your worktree as
above, redo the edits there, and leave the shared checkout's HEAD and working tree exactly
as you found them.
### Gating from a worktree
Three known breakages, all with workarounds, all required every run: see §3, "Running the
gate from a worktree".
### If you are writing a brief for another agent
The brief must name **its own explicit branch** and tell it to create **its own worktree**
from `origin/main`. "Work on the current branch" and "continue in the checkout" are how
branch sprawl and cross-contamination got here: an agent given no workspace uses the one it
is started in, which is the shared one. Also carry §3's three gate workarounds into the
brief — an agent that cannot gate will report rc 3 as green.
## 1. Never push to `main`. Ever.
Every durable change reaches `main` exactly one way:
@@ -147,15 +266,21 @@ Exit codes are a contract — do not flatten them:
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
beside this one; that single skip is expected and tolerated. Any *other* skip is a
real gap in your run.
beside this one; that single skip is expected and tolerated — and from a worktree it is
avoidable outright (§3's worktree list, item 3). `gitea-settings` is the other expected
skip: it needs `SETTINGS_API_TOKEN`, which exists only in CI, so **a local run cannot
return 0** — the honest best local result is rc 3 with `gitea-settings` as the only stage
that did not run. Any *further* skip is a real gap in your run. Report the rc and name the
skipped stages; never round rc 3 up.
**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
### Running the gate from a git worktree
### Running the gate from a worktree
Two unrelated failures, both with known workarounds — put both in any agent brief:
This is the **normal** path, not an edge case — §0 requires you to be in a worktree, so
every gate run you do hits all of these. Three known breakages, all with
workarounds, all of them yours to apply every run — put them in any agent brief:
1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
and exists only in the main checkout. Do **not** copy it (it is a credential).
@@ -166,10 +291,17 @@ Two unrelated failures, both with known workarounds — put both in any agent br
the `ENGINE` hook — a shim that intercepts `run` and re-execs
`podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
`tracked_files()` is this bug, not a broken tree.
3. **`spec-immutability` skips** — the skip called tolerated above is also *avoidable*. `dev.sh`
derives the sibling spec path from its own `REPO_ROOT`, which in a worktree is your
workspace, so `../home-network-k3s` does not resolve and it adds no mount. Mount the spec
repo read-only in the same `ENGINE` shim and pass `-e SPEC_REPO=/home-network-k3s`: with
both mounts the stage reports `pass`, not SKIPPED, and R-TEST-06 is actually covered
(measured — `decisions/2026-10-01-render-selftest-runs-in-gate.md`).
## 4. Branch naming
Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
Use `feat/`, `fix/`, `docs/`, `chore/`. **One branch, one workspace, one PR** — if you
need a second branch, make a second worktree (§0); do not re-point an existing one. Note that `.gitea/workflows/gate.yml`'s **push**
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
+17
View File
@@ -11,6 +11,9 @@ specification** at [`../home-network-k3s`](../home-network-k3s).
- **Done = a green, reproducible, requirement-tagged conformance test** — never "merged," never a
manual demo (spec `docs/14`, `agents/definition-of-done.md`).
- **No work on `main`.** Every change is a feature branch merged into `main` (spec `CONTRIBUTING.md`).
- **No work in a shared checkout.** Every agent run gets its own git worktree, outside the repo
root, branched from `origin/main`. This checkout is shared mutable state — see
[`CLAUDE.md` §0](CLAUDE.md).
- **No plaintext secrets** — SOPS-encrypted only (spec `docs/07`).
- **Phase discipline** — only build what the active roadmap phase authorizes (spec `roadmap/phases.md`).
@@ -94,6 +97,20 @@ The running narrative — decisions, root causes, incidents, follow-ups — is
## Working on this repo
**First, get out of this checkout.** It is the operator's working copy and several agents
share it; concurrent runs here have overwritten each other's branches and left each other's
uncommitted files behind. One workspace per change:
```bash
git fetch origin main
git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
cd /path/outside/the/repo/wt-<slug>
```
Outside the repo root specifically — a worktree under `.claude/worktrees/` is a second full
copy that tree-walking gate checks then count twice. The rule, the measurements behind it, and
the three gate workarounds a worktree needs are in [`CLAUDE.md` §0](CLAUDE.md) and §3.
Everything the gate needs is baked into the pinned dev container — **do not install the
toolchain on the host**:
@@ -0,0 +1,147 @@
---
date: 2026-10-05
branch: docs/agents-work-in-their-own-workspace
title: One workspace per agent run — the shared checkout is now documented as off-limits
---
## What changed
Agents have been doing their work in the operator's checkout at
`/var/home/keith/source/home-network-k3s-impl` and clobbering each other. Nothing in the tree
told them not to: worktrees appeared in CLAUDE.md only as §3's *gate workarounds*, i.e. as an
awkward edge case, and `decisions/README.md` asserted worktree isolation as a fact about how
agents already behave rather than as a rule they must follow. Nowhere was the rule stated.
Documentation only — no code, no manifests, no pipeline logic.
- **`CLAUDE.md`: new `## 0. Work in your OWN workspace. Never in the shared checkout.`**
Placed before §1 so the section numbers every other file and handoff entry cites (§1, §2b,
§3, §7) all keep their meaning. It gives the three commands (`git fetch origin main`;
`git worktree add -b <type>/<slug> <path outside the repo> origin/main`; `cd`), an explicit
do-not list for the shared checkout (no `checkout`/`switch`, no edits, no
`add`/`commit`/`stash`/`restore`, no removing others' worktrees), a measurement table, the
"put it outside the repo root" rule, cleanup, what to do if you are already standing in the
shared checkout with edits, and what a PM must put in a brief it writes for another agent.
- **`CLAUDE.md` intro** now says "Start with §0", with the reason: it is the rule you can
break before you have read anything else.
- **`CLAUDE.md` §3** retitled "Running the gate from a worktree" and reframed as the normal
path rather than an edge case, since §0 now makes every gate run a worktree run. Its list
grew from two breakages to three: `spec-immutability`'s skip is *avoidable*, not merely
tolerated, if the `ENGINE` shim also mounts the sibling spec repo read-only and passes
`-e SPEC_REPO=/home-network-k3s`.
- **`CLAUDE.md` §4** gained "One branch, one workspace, one PR".
- **`README.md`**: a fourth non-negotiable rule ("No work in a shared checkout"), and
"Working on this repo" now opens with how to get out of the checkout before describing the
dev container that runs in it.
- **`decisions/README.md`**: the worktree-isolation sentence now cites §0 as the requirement
instead of describing it as something agents happen to do.
I did this work the way the new §0 requires — a worktree at
`<session scratchpad>/wt-workspace-rule`, outside the repo root, branched from `origin/main`,
with the shared checkout left on the branch and with the untracked files I found in it.
## What I verified
Every number in the new §0 table was measured in the shared checkout on 2026-10-05, not
recalled:
| Claim | Command | Value |
|---|---|---|
| shared HEAD is yanked between branches | `git reflog \| grep -c 'checkout: moving'` | **107** |
| it is not sitting on `main` | `git worktree list` | `feat/pm-agent-issue-to-pr` |
| other runs' uncommitted work is in it | `git status --porcelain` | **2** trees: `infra/agent/` (22:04), `platform/pm-agent/` (22:09) — `stat -c %y` |
| its `main` is stale | `git rev-parse main` vs `origin/main` | `4386d14` vs `22c94ef`, **2 behind** |
| worktrees registered / already merged | `git worktree list --porcelain` + `git merge-base --is-ancestor <b> main` | **42** branches, **31** merged |
| nested worktrees are a gate hazard | `grep -rl '\.claude' conformance/tests platform/monitoring/tests` | **9** scripts carry an explicit exclusion |
| the 18× double-count | read `platform/monitoring/tests/run.sh:115-123` | in-tree comment, observed 2026-09-12 |
**The failure mode was observed live, mid-session, not just reconstructed.** The two untracked
trees I measured at 22:04 and 22:09 were committed at 22:20 by another run — `5334abe`
`feat(pm-agent): an in-cluster AI Program Manager...`, 1,000+ lines across `infra/agent/`,
`clusters/`, `catalog/`, `apps/`, `docs/plans/` and its own `decisions/` entry — onto the shared
checkout's checked-out branch. Measured after the fact: the shared checkout's HEAD moved
`22c94ef` → `5334abe` and its working tree went clean between my first `git status` and my last,
with `git log -1` naming a commit that is not mine. That is §0's "shared HEAD" row happening
during the session that wrote §0, and it is now cited in the doc.
**A claim of mine was wrong and is corrected in the doc.** I had it recorded that
`git worktree remove` is blocked by the agent sandbox. Provoked rather than assumed: created
a throwaway detached worktree in the scratchpad and removed it — **rc 0**, and
`git worktree list` no longer shows it. §0 therefore tells agents to remove their own, and
scopes the "blocked" note to what it actually covers: *other* sessions'
`.claude/worktrees/agent-*`, some of which are additionally `locked` on git's own terms. The
`rework/<pr-number>` escape hatch is now scoped to someone else's stale worktree holding a
branch, which is a real and separate problem (31 of them). Both of my own worktrees in this
session were removed cleanly at rc 0 after pushing.
`spec-immutability` is avoidable-not-tolerated: taken from
`decisions/2026-10-01-render-selftest-runs-in-gate.md`, which records
`stage spec-immutability: pass` under the two-mount shim plus `-e SPEC_REPO=/home-network-k3s`.
**Gate — `rc 3`, UNDETERMINED, and I am not calling it green.** Pinned toolbox, from this
worktree, `DEV_KUBECONFIG=/dev/null` and an `ENGINE` shim bind-mounting both the main
checkout's `.git` (at its own host path) and the sibling spec repo read-only with
`-e SPEC_REPO=/home-network-k3s`:
ENGINE=<two-mount shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' # rc 3
| stage | result |
|---|---|
| spec-immutability | **pass** — it genuinely RAN; the second mount is what makes it run from a worktree |
| approval-policy | pass, but **N/A**: `event='none'`. It cannot see a PR locally, so this green says nothing. See below |
| decisions-log | pass — the three required headings and the frozen-archive pin |
| lint · l0 · render-golden · l1 | pass (L1: 3 offline checks, 0 failures, 0 skips, 0 unwired) |
| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset; it exists only in CI |
| conformance (L2/L3) | SKIPPED — `CLUSTER_AVAILABLE!=true`; the decision notes no cluster-needing requirement was touched |
`gitea-settings` is the *only* stage that did not run, and it cannot be made to run from a
workstation, so rc 3 is the ceiling for any local run — not a defect in this branch. The
first gate attempt returned rc 3 for a different and real reason (`HEAD is not ahead of
origin/main`: I had not committed yet, so the touched-requirement set was empty by default
rather than by verification); it was re-run after committing, and the table above is the
post-commit run. §3 now records both expected skips, because it previously named only
`spec-immutability`.
## What I did NOT verify
- **GATE: UNVERIFIED at the time of writing.** The pinned-toolbox result for this branch and
its exit code are reported in the PR, not predicted here. **A local rc 3 is not green** and
will not be described as one.
- **CI gate: RED, measured — `MERGE GATE: BLOCKED`, one failing stage.** Run 245 on PR #56,
head `b988da8b`, 1m47s. The only failure is `approval-policy`: *"a sensitive change lacks a
non-author approval"*, `sensitive paths touched: CLAUDE.md`, author `gitea-admin`. Every other
stage passed — `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, and `gitea-settings`
(which can only run in CI, and reported `required_approvals = 0`,
`dismiss_stale_approvals = true`). `spec-immutability` SKIPPED in CI (no sibling spec repo in
the CI workspace — the inverse of the local run, where the two-mount shim made it pass) and
`conformance` SKIPPED. **Note `gate / conformance (pull_request)` reports status `success`
with description "Has been skipped"** — CLAUDE.md §2b's trap, live on this very PR. Nothing
in this branch is broken; the red is the structural human gate, below. *Measured at
`b988da8b`; this entry was then amended to record it, so the head SHA in the PR is newer than
the run quoted here and CI re-ran on the amended commit.*
- **This PR is a sensitive-path PR and is CI-RED on `approval-policy` until a human acts.**
`CLAUDE.md` is in `infra/gitea/approval-policy.json`'s `sensitive_paths` ("the delivery
contract agents follow"), so the stage requires an approving review from someone who is not
the author, or an `/approve-override <reason>` posted by an administrator **as a PR review
of type "Comment"**, not in the plain comment box. `ci/approval-policy.sh` run locally
reports N/A (it cannot see a PR), so the local gate cannot catch this and a green local run
says nothing about it. **I have not posted an override and must not** — CLAUDE.md §1 forbids
an agent clearing this on its own PR even though `gitea-admin` holds `owner` and could.
- **Nothing here is enforced.** §0 is prose. There is no gate stage that detects work done in
the shared checkout, no check that a worktree is outside the repo root, and no check that a
branch was cut from `origin/main`. Every other rule in CLAUDE.md that agents kept breaking
(`DECISIONS-LOG.md` appends, the three handoff headings) ended up as a gate stage, and that
is the honest follow-up here. I did not write one: the ask was documentation, and an
enforcement stage lands in `ci/**`, which is itself a sensitive path and deserves its own PR.
- **I did not clean up the 31 stale merged worktrees or the 35 nested under
`.claude/worktrees/`.** They are other sessions' workspaces; removing them is an operator
action, and §0 explicitly tells agents not to remove worktrees they did not create.
- **I did not touch the untracked `infra/agent/` and `platform/pm-agent/` trees** sitting in the
shared checkout. They are another run's in-flight work. Whose, and whether they are wanted,
is unknown to me.
- **`docs/plans/ssd-etcd-execution-runbook.md:18` still points at a `.claude/worktrees/` path**
inside the repo. Left as-is deliberately: that runbook is marked COMPLETED/historical and
describes the now-decommissioned dev cluster, so the path is a record of what was done, not
an instruction. It does mean a grep for in-repo worktree paths is not yet clean.
- **No runtime, cluster or Flux behaviour was touched or checked.** This branch changes three
Markdown files; `kubectl` was not run.
@@ -0,0 +1,88 @@
---
date: 2026-10-05
branch: feat/cilium-roll-out-on-config-change
title: Cilium config changes were silently inert — roll the agents when cilium-config changes, which also finally activates the Hubble exporter
---
## What changed
**A merged Cilium config change did nothing, and everything reported success.** The Hubble flow
exporter (PR #50, merged 2026-10-04) wrote its settings into `cilium-config` and never took effect.
| Measured 2026-10-05, over a day after the merge | |
|---|---|
| HelmRelease `cilium` | `Ready=True`, `UpgradeSucceeded`, *"Helm upgrade succeeded for release kube-system/cilium.v6"* |
| `cilium-config` keys | `hubble-export-file-path: stdout`, `hubble-export-allowlist: {"destination_port":["32400"]} {"source_port":["32400"]}` — **all present** |
| cilium agent pods | **all three still the PRE-MERGE pods**, `startTime 2026-10-04T07:11Z` |
| exported flow lines in agent stdout | **0, 0, 0** |
| Flux Kustomizations | all `Ready=True` |
Cause: the chart's `rollOutCiliumPods` defaults to **`false`** (cilium 1.20.1 `values.yaml`: *"Roll
out cilium agent pods automatically when configmap is updated"*), so Helm rewrites the ConfigMap and
leaves the DaemonSet's pod template untouched — and **cilium-agent reads that ConfigMap only at
startup**. The feature was merged, believed delivered, green everywhere, and doing nothing at all.
That is the same class of failure this repo already guards against elsewhere: a green signal sitting
over something that never ran.
- **`platform/cilium/helmrelease.yaml`** (authoritative): `rollOutCiliumPods: true`.
- **`platform/cilium/values.yaml`** (reference mirror) and
**`infra/ansible/roles/cilium/templates/cilium-values.yaml.j2`** (the `[parity]` source of truth):
the same, so a bootstrap render does not differ from what Flux reconciles.
**The cost is real and is the point.** Every future Cilium config change now triggers a cluster-wide
agent rollout instead of waiting for someone to remember a manual restart. That is strictly more
honest — the change applies when merged, or it fails visibly — but it makes config edits disruptive
by default, so they want a quiet moment. Mitigation already in place:
`updateStrategy.rollingUpdate.maxUnavailable: 1` means the DaemonSet will not proceed past an unready
agent, so a bad render costs one node rather than the cluster.
**Merging this is itself one rollout**, because adding the ConfigMap checksum annotation changes the
pod template — and that rollout is what finally activates the Hubble exporter.
## What I verified
- **Every row of the table above**, read from the live HelmRelease status, the live `cilium-config`
ConfigMap, the agent pods' `startTime`, and a grep for `"flow"` in each agent's stdout.
- **The chart value name, default and semantics**, quoted from cilium **v1.20.1**'s own
`values.yaml` — the version actually running here (`quay.io/cilium/cilium:v1.20.1`), not a guess
from another release: `rollOutCiliumPods: false` with the comment *"Roll out cilium agent pods
automatically when configmap is updated."*
- **All three value files carry the key** and both YAML files still parse, with
`spec.values.rollOutCiliumPods` resolving to `true`.
- **The blast-radius mitigation is genuinely present**, not assumed:
`updateStrategy.rollingUpdate.maxUnavailable: 1` is already set in the HelmRelease, with a long
existing comment explaining that a CNI rollout has no healthy-peer fallback.
- **THE MECHANISM IS CONFIRMED BY CONTRAST, not just reasoned.** The cgroup fix (PR #42,
2026-10-03) DID take effect: both the DaemonSet and the running pod mount
`cilium-cgroup hostPath=/sys/fs/cgroup`, matching what the HelmRelease declares. The difference is
exactly the one claimed above — #42 changed the POD TEMPLATE (a volume hostPath), which rolls the
DaemonSet on its own, while #50 changed ONLY the ConfigMap, which does not. It also explains the
agents' `startTime 2026-10-04T07:11Z`: that was #42's rollout, and nothing has rolled them since.
So the inertness is specific to ConfigMap-only changes, which is precisely what this flag fixes.
- **Flux is healthy before this change** — every Kustomization `Ready=True`, so this lands on a clean
graph rather than into an existing cascade.
- **Toolbox gate**: recorded in the PR.
## What I did NOT verify
- **THE OUTCOME, ENTIRELY. Unreconciled.** No agent has rolled, no flow has been exported. This
change is an argument that the rollout will happen, not evidence that it did.
- **Whether `filePath: stdout` actually works** — still the open question from #50, and it cannot be
answered until the agents restart. If the literal is not special-cased, the agent may create a
FILE named `stdout` and nothing reaches Loki: a silent no-op, not a crash. Fallback documented in
#50: keep the chart's default path and add a `loki.source.file` target to Alloy with a hostPath
mount. **Check for flow lines in the agent logs immediately after the rollout.**
- **THE ROLLOUT'S IMPACT ON LAN DNS IS UNMEASURED AND IS THE MAIN RISK.** The node holding the L2
lease for the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP loses them briefly as it cycles.
`maxUnavailable: 1` bounds it to one node at a time, and BPF maps persist across an agent restart,
but I have not measured how long a VIP is unreachable during a lease move, nor whether the two
PiHole replicas cover the gap. Merge it at a quiet moment.
- **Whether enabling this is the operator's preferred trade.** The alternative — leaving it `false`
and restarting by hand — keeps config edits non-disruptive at the cost of them being silently
inert. I judged silent inertness the worse failure, but it is a judgement, not a measurement.
- **Whether anything ELSE in `cilium-config` is currently inert.** I checked the one other recent
Cilium change and it is fine (see below), but I did not audit every key against the running
agents' actual state.
- **`hubble.relay.rollOutPods` / `hubble.ui.rollOutPods`** are left at their `false` defaults. Both
have the same silently-inert property for their own ConfigMaps; not addressed here.
+2 -1
View File
@@ -26,7 +26,8 @@ The fix is the obvious one for concurrent appenders: **stop sharing a file**. Ea
handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
nothing for git to merge and nothing to conflict. This scales to any number of parallel
branches with zero coordination — which is the point, since the agents are deliberately
isolated from each other in separate worktrees and cannot coordinate.
isolated from each other in separate worktrees (required by `CLAUDE.md` §0) and cannot
coordinate.
There is deliberately **no index file** and no counter. An index would be a shared
mutable file — exactly the thing that just failed — and a sequence number would have
@@ -39,6 +39,12 @@ bpf:
masquerade: true
# Flow observability / isolation audit (ADR-0002).
# Roll the agents when cilium-config changes. Kept EQUAL to the Flux values
# (platform/cilium/helmrelease.yaml), which carry the full rationale: the chart default of false
# leaves the pod template untouched on a config change, and cilium-agent reads that ConfigMap only
# at startup, so the change is silently inert until someone restarts the DaemonSet by hand.
rollOutCiliumPods: true
hubble:
enabled: true
relay:
+31
View File
@@ -188,6 +188,37 @@ spec:
enabled: false
hostRoot: /sys/fs/cgroup
# --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
# --- Roll the agents when their ConfigMap changes (2026-10-05) ---
# [parity] MUST equal Ansible: rollOutCiliumPods: true
#
# WHY: a merged Cilium config change DID NOTHING. The chart default is `false`
# (cilium 1.20.1 values.yaml: "Roll out cilium agent pods automatically when configmap is
# updated. rollOutCiliumPods: false"), so Helm rewrites `cilium-config` and leaves the
# DaemonSet's pod template untouched — and cilium-agent reads that ConfigMap only at STARTUP.
# The result is a silently inert change: Flux green, HelmRelease `UpgradeSucceeded`, the new
# keys visibly present in the ConfigMap, and the running agents still on the old settings
# until a human remembers to restart them.
#
# MEASURED, and it is why this exists: the Hubble flow exporter merged 2026-10-04 wrote
# `hubble-export-file-path: stdout` and its allowlist into `cilium-config`, the HelmRelease
# reported `Helm upgrade succeeded ... cilium.v6`, and over a DAY later all three agents were
# still the pre-merge pods (startTime 2026-10-04T07:11Z) with ZERO exported flow lines. The
# feature was merged, believed delivered, and doing nothing at all. That is the same class of
# failure this repo already documents elsewhere — a green signal over something that never ran.
#
# THE COST IS REAL AND IS THE POINT: every future Cilium config change now triggers a
# cluster-wide agent rollout instead of waiting for a manual restart. That is strictly more
# honest — the change applies when it is merged, or it fails visibly — but it means config
# edits are disruptive by default, so they want a quiet moment. One node at a time:
# `updateStrategy.rollingUpdate.maxUnavailable: 1` below means the DaemonSet will not proceed
# past an unready agent, so a bad render costs one node rather than the cluster. Note that the
# node holding the L2 lease for the Gateway, PiHole-DNS and Plex VIPs loses them briefly as it
# cycles, so a rollout can blip LAN DNS.
#
# ENABLING THIS IS ITSELF ONE ROLLOUT: adding the ConfigMap checksum annotation changes the pod
# template, so merging this rolls the agents once — which is what finally activates the Hubble
# exporter that has been inert since 2026-10-04.
rollOutCiliumPods: true
# [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
# The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
# maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad
+9
View File
@@ -32,6 +32,15 @@ enableIPv4Masquerade: true
bpf:
masquerade: true
# [parity] flow observability / isolation audit
# [parity] roll the agents when cilium-config changes. The chart default (false) rewrites the
# ConfigMap but leaves the pod template alone, and cilium-agent only reads it at STARTUP — so a
# merged config change is SILENTLY INERT. Measured 2026-10-04..05: the Hubble exporter keys were
# live in cilium-config and the HelmRelease said UpgradeSucceeded, while all three agents were
# still pre-merge pods exporting nothing for over a day. Cost: config edits now roll the agents
# (one node at a time via maxUnavailable: 1), which can briefly blip the VIPs on the lease-holding
# node. Full rationale in helmrelease.yaml.
rollOutCiliumPods: true
hubble:
enabled: true
relay: