jakman85andClaude Opus 5
55b232e2ce
The operator got no mail when Flux began applying a git change and none when it finished. Measured cause: notification-controller v1.4.0 is running and its CRDs serve v1beta3, but there are ZERO Provider and ZERO Alert objects in the cluster — nothing in git ever created any. The obvious fix does not work. Neither Flux event can be filtered into a deploy signal: ReconciliationSucceeded is a heartbeat (101 events in one hour across 16 Kustomizations, message byte-identical whether the apply changed anything), and Progressing is worse (five Kustomizations re-apply the same object on every tick — event counts 2505/2504/2501/2495/2490). gotk_* carries no revision label at all (gotk_reconcile_condition 120 series, gotk_suspend_status 40, none with one). The applied revision is the only signal that moves when and only when git moves, and it lives in object status alone. So export it via the kube-state-metrics custom-resource-state path that already serves the Velero Schedule metrics — kube_flux_kustomization_info/_suspended and kube_flux_gitrepository_info — and ask the question in PromQL: FluxRolloutInProgress (severity: info, no `for:`) fires the moment a non-suspended Kustomization's applied revision is not the revision the GitRepository fetched from main, and CLEARS when all have caught up. Fired mail = a deploy started; RESOLVED mail = it completed. The completion half is send_resolved, not a second rule. FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the escalation for a rollout that started and never converged. Two guards, each with a test whose only job is to fail if the clause is removed: `and on () (count(gitrepository_info) > 0)` — without it a missing exporter makes `unless on (revision) <empty>` remove nothing and all 16 Kustomizations alert forever, the permanently-firing mode this repo shipped once already — and the suspended-Kustomization exclusion, without which a suspended tier holds the rollout open and the "applied" mail never arrives. A dedicated Alertmanager severity=info route + email-deploy receiver keeps deploy notices off the [WARNING] catch-all. podAnnotations carries a roll-counter because the chart sets no checksum over the custom-resource-state ConfigMap, so a config-only edit would leave KSM on the old collector set with the new alerts silently inert. Verified: promtool SUCCESS on 6 scenarios (both guards, plus the convergence test that proves the alert empties itself); lint/l0/render-golden/l1/decisions-log/ spec-immutability green in the pinned toolbox. NOT verified: no mail has been sent — the metrics do not exist until this merges. Full gap list, and the deliberately-cut notification-controller half, in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
decisions/ — the handoff log, one file per entry
This directory is the decision log. ../DECISIONS-LOG.md is the frozen archive of
everything written before 2026-09-29 and is never edited again (see "The archive is
frozen" below).
Why this exists
DECISIONS-LOG.md was append-only in a single file, and every agent appended to its
tail. Two branches that both append hit the same last hunk, so git cannot
auto-merge them: the second one to be reviewed conflicts, and so does the third, and
the fourth. Measured on 2026-09-29: four of the five open PRs were unmergeable, and
DECISIONS-LOG.md was the only conflicting file in all four:
#22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
#20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
#19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
#18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
#17 feat/ci-is-the-approver clean (it touched no log entry)
Nothing was wrong with any of those changes. The log's storage format serialised work that was otherwise parallel, and it got worse with every concurrent branch: N branches produce N-1 rebases, each of which invalidates any gate run that preceded it.
The fix is the obvious one for concurrent appenders: stop sharing a file. Each handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is nothing for git to merge and nothing to conflict. This scales to any number of parallel branches with zero coordination — which is the point, since the agents are deliberately isolated from each other in separate worktrees and cannot coordinate.
There is deliberately no index file and no counter. An index would be a shared
mutable file — exactly the thing that just failed — and a sequence number would have
two agents racing for 0043. The directory listing is the index.
Writing an entry
One file per handoff, named:
decisions/YYYY-MM-DD-<slug>.md
<slug> is lowercase [a-z0-9-]. Use your branch name's last segment — it is
already unique per unit of work, which is precisely the property the filename needs.
A branch fix/crafty-oauth2-proxy-signin handed off on 2026-09-29 writes:
decisions/2026-09-29-crafty-oauth2-proxy-signin.md
The file is YAML frontmatter followed by three required sections:
---
date: 2026-09-29
branch: fix/crafty-oauth2-proxy-signin
title: The Keycloak gate works — Crafty's own login was the wall
---
## What changed
Prose. What you actually did, and why.
## What I verified
The measurements, with their numbers. Quote the field you read, not the prose you
believed. Name the gate result and its exit code.
## What I did NOT verify
Everything else. An honest "UNVERIFIED" is worth more than an optimistic "green" —
this repo has been burned by the difference more than once.
All three headings are required and enforced (ci/decisions-log.sh). They are
CLAUDE.md §7's contract; making them a checked structure is what stops an entry
degrading into "done ✅".
Reading the log
The whole log, oldest first:
ls decisions/[0-9]*.md | sort
The current PM handoff — the newest entry. Ties within a day are broken by what landed
on main last, which is the honest answer, so ask git rather than the filename:
git log --diff-filter=A --format='%H' --name-only -- 'decisions/[0-9]*.md' \
| grep '^decisions/' | head -1
For a quick skim of who said what:
grep -H '^title:' decisions/[0-9]*.md | sort
The archive is frozen
DECISIONS-LOG.md is read-only history. Its SHA-256 is pinned in
decisions/ARCHIVE.sha256 and ci/decisions-log.sh fails the merge gate if the file
changes. That is not ceremony: without it, the next agent to read runbooks/ or an old
CLAUDE.md habit appends to the tail again and re-creates the conflict for everybody
else, silently, one PR at a time.
If the archive genuinely must change (a leaked credential to redact, say), that is a
deliberate, reviewed act: change it, run sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256,
and say in your handoff entry why. Re-pinning is visible in the diff. Appending is not.