chore(log): the handoff log was serialising every parallel branch #24

Merged

jakman85 merged 1 commits from chore/decisions-log-conflict-free into main 2026-09-30 03:40:19 +00:00

Conversation 1 Commits 1 Files Changed 12

+600 -12

gitea-admin commented 2026-09-30 03:38:12 +00:00

Owner

Copy link

Copy source

The problem, measured

Four of the five open PRs were unmergeable, and DECISIONS-LOG.md was the only conflicting path in all four. Measured against the live Gitea, not inferred from the tree (GET /api/v1/repos/Home-Network/home-network-k3s-impl/pulls, 2026-09-29):

PR	branch	mergeable	git merge-tree vs main
#22	fix/crafty-oauth2-proxy-signin	false	CONFLICT (content): DECISIONS-LOG.md
#20	feat/conformance-job-template	false	CONFLICT (content): DECISIONS-LOG.md
#19	feat/conformance-trigger-kubeconfig	false	CONFLICT (content): DECISIONS-LOG.md
#18	feat/conformance-runner-image	false	CONFLICT (content): DECISIONS-LOG.md
#17	feat/ci-is-the-approver	true	clean — it wrote no handoff entry

Nothing is wrong with any of those changes; they have nothing to do with each other. CLAUDE.md §7 told every agent to append a handoff, and appending always landed on the same last hunk of one 5,228-line file. Two branches that both did it cannot auto-merge. N concurrent branches cost N−1 rebases, each invalidating the gate run before it. The agents work in isolated worktrees precisely so they do not block each other — this one file undid that.

The fix

Stop sharing a file. Handoffs are now one-per-file under decisions/, named YYYY-MM-DD-<slug>.md (slug = the branch's last segment, already unique per unit of work). Disjoint paths have nothing to merge, and this scales to any number of parallel branches with zero coordination — which matters, because the agents cannot coordinate.

There is deliberately no index file (a shared mutable file is exactly the thing that just failed) and no sequence counter (two agents would race for 0043). The directory listing is the index.

DECISIONS-LOG.md is frozen as the pre-2026-09-29 archive, SHA-256 pinned in decisions/ARCHIVE.sha256.

Why a gate stage and not just a paragraph

The old habit is written into runbooks/, into older log entries, and into every agent that has read the log. An agent that appends anyway does not fail — it produces a clean-looking commit that quietly re-creates the conflict for every other open branch, and the damage surfaces later, to somebody else, in a PR that is not theirs. Silent, delayed, misattributed: the exact shape this repo keeps getting burned by. So it gets a guard.

ci/decisions-log.sh asserts:

1. DECISIONS-LOG.md matches its pin. Re-pinning (to redact a leaked credential, say) is a deliberate line in a diff; appending is not.
2. Every entry is YYYY-MM-DD-<slug>.md with a real calendar date.
3. Frontmatter date/branch/title, with date agreeing with the filename — a disagreement means a copied template nobody edited.
4. All three of §7's sections, each with a real body: ## What changed, ## What I verified, ## What I did NOT verify. The third is the load-bearing one — this repo's recurring failure is a confident report of something nobody measured, and an entry with nowhere to write "UNVERIFIED" does not get one written.

Exit 0 or 1 only. The stage has no skip path by construction: it reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run". Adding an rc 3 would put a fail-open surface on a guard whose whole job is to close one. ci/gate.sh blocks on both fail and (impossible) skip, and there is deliberately no REQUIRE_DECISIONS_LOG knob — a guard with an off switch is not a guard.

A pre-existing bug found on the way

ci/tests/gate-decision.test.sh's stub list never included gitea-settings. That stage's script was therefore absent from the stub CI dir, run_stage got exit 127 from the missing file, and the gate recorded a FAILED stage in every single case. 15 of 47 assertions failed on main, including every one expecting a PASS / UNDETERMINED / NOT-APPLICABLE headline. The suite was measuring its own missing stub, not the decision logic. Fixed in the same change that adds decisions-log to that list, with a comment saying why the list must stay complete.

Verification

Toolbox gate (./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'), rc 3 — UNDETERMINED, not green:

spec-immutability: green
gitea-settings:    SKIPPED (no GITEA_SETTINGS_TOKEN)   <- the only reason for rc 3
decisions-log:     green
lint:              green
l0:                green
render-golden:     green
l1:                green
conformance:       L2/L3 UNVERIFIED (CLUSTER_AVAILABLE unset — Phase 0, declared)
touched requirements: R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06
MERGE GATE: UNDETERMINED

The rc 3 is the standing baseline for every PR in this repo (the Gitea settings token is unwired), not something this change introduces. It is stated rather than rounded to green.

make -C ci test in the toolbox — all five suites passed, 144 assertions:

suite	result
decisions-log.test.sh	PASS (22) — new
gate-decision.test.sh	PASS (50) — was FAIL (15 failed, 32 passed) on main
touched-requirements-base.test.sh	PASS (31)
touched-requirements-test-attribution.test.sh	PASS (15)
touched-requirements.test.sh	PASS (26)

The assertion that matters: an append to DECISIONS-LOG.md exits non-zero, and the message names both the frozen archive and the file to write instead.

NOT verified

• Nothing applied, nothing merged. No kubectl, no flux reconcile. CI and documentation only; no cluster-visible effect.
• L2/L3 did not run (CLUSTER_AVAILABLE unset). Irrelevant to this change's content, stated so the gate result is not over-read.
• I have not watched a Gitea check run this stage. Proven by its unit test and the local gate; that the incluster-dind runner executes it is unobserved. A skipped Gitea job reports success — read the job log, not the tick.
• The four hung PRs are not fixed by this commit. They still carry their DECISIONS-LOG.md appends and still conflict until each is rebased onto this change with its append converted to a decisions/ entry. That work follows immediately, one agent per PR.
• runbooks/ still says "append to DECISIONS-LOG.md" in places; not swept. The gate now rejects the action, so the stale wording misleads but cannot cause the damage. Joins the tracked work on stale git push origin main wording.

Merge order

Merge this first. The four follow-up PRs rebase onto it.

🤖 Generated with Claude Code

## The problem, measured Four of the five open PRs were unmergeable, and **`DECISIONS-LOG.md` was the only conflicting path in all four**. Measured against the live Gitea, not inferred from the tree (`GET /api/v1/repos/Home-Network/home-network-k3s-impl/pulls`, 2026-09-29): | PR | branch | `mergeable` | `git merge-tree` vs `main` | |---|---|---|---| | #22 | `fix/crafty-oauth2-proxy-signin` | `false` | `CONFLICT (content): DECISIONS-LOG.md` | | #20 | `feat/conformance-job-template` | `false` | `CONFLICT (content): DECISIONS-LOG.md` | | #19 | `feat/conformance-trigger-kubeconfig` | `false` | `CONFLICT (content): DECISIONS-LOG.md` | | #18 | `feat/conformance-runner-image` | `false` | `CONFLICT (content): DECISIONS-LOG.md` | | #17 | `feat/ci-is-the-approver` | `true` | clean — it wrote no handoff entry | Nothing is wrong with any of those changes; they have nothing to do with each other. CLAUDE.md §7 told every agent to append a handoff, and appending always landed on the **same last hunk** of one 5,228-line file. Two branches that both did it cannot auto-merge. N concurrent branches cost N−1 rebases, each invalidating the gate run before it. The agents work in isolated worktrees precisely so they do not block each other — this one file undid that. ## The fix **Stop sharing a file.** Handoffs are now one-per-file under `decisions/`, named `YYYY-MM-DD-<slug>.md` (slug = the branch's last segment, already unique per unit of work). Disjoint paths have nothing to merge, and this scales to any number of parallel branches with zero coordination — which matters, because the agents cannot coordinate. There is deliberately **no index file** (a shared mutable file is exactly the thing that just failed) and **no sequence counter** (two agents would race for `0043`). The directory listing is the index. `DECISIONS-LOG.md` is frozen as the pre-2026-09-29 archive, SHA-256 pinned in `decisions/ARCHIVE.sha256`. ## Why a gate stage and not just a paragraph The old habit is written into `runbooks/`, into older log entries, and into every agent that has read the log. An agent that appends anyway **does not fail** — it produces a clean-looking commit that quietly re-creates the conflict for every *other* open branch, and the damage surfaces later, to somebody else, in a PR that is not theirs. Silent, delayed, misattributed: the exact shape this repo keeps getting burned by. So it gets a guard. `ci/decisions-log.sh` asserts: 1. `DECISIONS-LOG.md` matches its pin. Re-pinning (to redact a leaked credential, say) is a deliberate line in a diff; appending is not. 2. Every entry is `YYYY-MM-DD-<slug>.md` with a **real calendar date**. 3. Frontmatter `date`/`branch`/`title`, with `date` agreeing with the filename — a disagreement means a copied template nobody edited. 4. All three of §7's sections, each with a real body: `## What changed`, `## What I verified`, `## What I did NOT verify`. The third is the load-bearing one — this repo's recurring failure is a confident report of something nobody measured, and an entry with nowhere to write "UNVERIFIED" does not get one written. Exit **0 or 1 only**. The stage has **no skip path by construction**: it reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run". Adding an rc 3 would put a fail-open surface on a guard whose whole job is to close one. `ci/gate.sh` blocks on both `fail` and (impossible) `skip`, and there is deliberately **no `REQUIRE_DECISIONS_LOG` knob** — a guard with an off switch is not a guard. ## A pre-existing bug found on the way `ci/tests/gate-decision.test.sh`'s stub list never included `gitea-settings`. That stage's script was therefore absent from the stub CI dir, `run_stage` got exit **127** from the missing file, and the gate recorded a FAILED stage in **every single case**. **15 of 47 assertions failed on `main`**, including every one expecting a PASS / UNDETERMINED / NOT-APPLICABLE headline. The suite was measuring its own missing stub, not the decision logic. Fixed in the same change that adds `decisions-log` to that list, with a comment saying why the list must stay complete. ## Verification Toolbox gate (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`), **rc 3 — UNDETERMINED, not green**: ``` spec-immutability: green gitea-settings: SKIPPED (no GITEA_SETTINGS_TOKEN) <- the only reason for rc 3 decisions-log: green lint: green l0: green render-golden: green l1: green conformance: L2/L3 UNVERIFIED (CLUSTER_AVAILABLE unset — Phase 0, declared) touched requirements: R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06 MERGE GATE: UNDETERMINED ``` The rc 3 is the **standing baseline** for every PR in this repo (the Gitea settings token is unwired), not something this change introduces. It is stated rather than rounded to green. `make -C ci test` in the toolbox — **all five suites passed, 144 assertions**: | suite | result | |---|---| | `decisions-log.test.sh` | **PASS (22)** — new | | `gate-decision.test.sh` | **PASS (50)** — was **FAIL (15 failed, 32 passed)** on `main` | | `touched-requirements-base.test.sh` | PASS (31) | | `touched-requirements-test-attribution.test.sh` | PASS (15) | | `touched-requirements.test.sh` | PASS (26) | The assertion that matters: an append to `DECISIONS-LOG.md` exits non-zero, and the message names both the frozen archive and the file to write instead. ## NOT verified - **Nothing applied, nothing merged.** No `kubectl`, no `flux reconcile`. CI and documentation only; no cluster-visible effect. - **L2/L3 did not run** (`CLUSTER_AVAILABLE` unset). Irrelevant to this change's content, stated so the gate result is not over-read. - **I have not watched a Gitea check run this stage.** Proven by its unit test and the local gate; that the `incluster-dind` runner executes it is unobserved. A skipped Gitea job reports `success` — read the job log, not the tick. - **The four hung PRs are not fixed by this commit.** They still carry their `DECISIONS-LOG.md` appends and still conflict until each is rebased onto this change with its append converted to a `decisions/` entry. That work follows immediately, one agent per PR. - **`runbooks/` still says "append to `DECISIONS-LOG.md`"** in places; not swept. The gate now rejects the action, so the stale wording misleads but cannot cause the damage. Joins the tracked work on stale `git push origin main` wording. ## Merge order **Merge this first.** The four follow-up PRs rebase onto it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-09-30 03:38:13 +00:00

chore(log): the handoff log was serialising every parallel branch ... 72a46ad82d

DECISIONS-LOG.md was append-only in ONE file and every agent appended to its
tail, so any two concurrent branches collided on the same last hunk. Measured
against the live Gitea today: four of the five open PRs were mergeable=false,
and `git merge-tree` named DECISIONS-LOG.md as the ONLY conflicting path in
all four.

    #22 fix/crafty-oauth2-proxy-signin       CONFLICT: DECISIONS-LOG.md
    #20 feat/conformance-job-template        CONFLICT: DECISIONS-LOG.md
    #19 feat/conformance-trigger-kubeconfig  CONFLICT: DECISIONS-LOG.md
    #18 feat/conformance-runner-image        CONFLICT: DECISIONS-LOG.md
    #17 feat/ci-is-the-approver              clean — it wrote no handoff

Nothing was wrong with any of those changes. The log's storage format
serialised work that was otherwise parallel, and the cost grew with every
concurrent branch (N branches -> N-1 rebases, each invalidating the gate run
before it). The agents work in isolated worktrees precisely so they do not
block each other; this one file undid that.

So: stop sharing a file. Handoffs are now one-per-file under decisions/,
named YYYY-MM-DD-<slug>.md. Disjoint paths have nothing to merge. There is
deliberately no index file (a shared mutable file is the thing that just
failed) and no counter (two agents would race for the same number) — the
directory listing is the index.

DECISIONS-LOG.md is frozen as the pre-2026-09-29 archive, its SHA-256 pinned
in decisions/ARCHIVE.sha256, and a new gate stage rejects any change to it.
That is a guard and not a paragraph because the old habit is written into the
runbooks and into every agent that has read the log, and getting it wrong does
not fail: it produces a clean-looking commit that re-creates the conflict on
OTHER branches, later, misattributed.

  ci/decisions-log.sh        new stage: archive pinned; every entry named with
                             a real date, carrying date/branch/title frontmatter
                             that agrees with its filename, and all three of
                             CLAUDE.md §7's sections with a real body under each.
                             Exit 0 or 1 only — no skip path by construction, so
                             no fail-open surface on a guard that closes one.
  ci/gate.sh                 runs it; fail and (impossible) skip both BLOCK.
                             No REQUIRE_DECISIONS_LOG knob: a guard with an off
                             switch is not a guard.
  ci/tests/decisions-log.test.sh   22 assertions, incl. the one that matters —
                             an append to the archive is rejected, and the
                             message names the file to write instead.
  ci/tests/gate-decision.test.sh   +3 assertions for the new wiring, AND a
                             pre-existing bug: the stub list never included
                             gitea-settings, so run_stage saw exit 127 and
                             recorded a FAILED stage in every case. 15 of 47
                             assertions failed on main for that reason alone —
                             the suite was measuring its own missing stub.
                             Now 50/50.
  CLAUDE.md                  §7 rewritten; §2b widened to the new path.
  ci/Makefile, ci/README.md, .gitea/workflows/README.md — stage named where
                             the other stages are named.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 approved these changes 2026-09-30 03:40:13 +00:00

jakman85 merged commit c59fe9f923 into main 2026-09-30 03:40:19 +00:00

jakman85 referenced this issue from a commit 2026-09-30 03:40:20 +00:00

Merge pull request 'chore(log): the handoff log was serialising every parallel branch' (#24) from chore/decisions-log-conflict-free into main

gitea-admin referenced this issue from a commit 2026-09-30 03:46:53 +00:00

docs(decisions): move PR #19's handoff out of the frozen DECISIONS-LOG.md

gitea-admin referenced this issue from a commit 2026-09-30 03:47:20 +00:00

docs(decisions): Crafty sign-in handoff — the gate was fine, the local login was the wall

gitea-admin referenced this pull request 2026-09-30 03:47:26 +00:00

feat(sec): provision the narrow conformance CI trigger credential -- and refuse to arm a fake green #19

gitea-admin referenced this pull request 2026-09-30 03:47:47 +00:00

fix(crafty): the Keycloak gate works — Crafty's own login was the wall #22

gitea-admin referenced this issue from a commit 2026-09-30 03:49:45 +00:00

docs(decisions): Crafty sign-in handoff — the gate was fine, the local login was the wall

gitea-admin referenced this issue from a commit 2026-09-30 03:52:37 +00:00

docs(decisions): move PR #19's handoff out of the frozen DECISIONS-LOG.md

gitea-admin referenced this issue from a commit 2026-09-30 03:52:44 +00:00

docs(decisions): handoff for the conformance-runner image, as its own file

gitea-admin referenced this pull request 2026-09-30 03:53:18 +00:00

feat(conformance): build the runner image the in-cluster L2/L3 Job could never have run #18

gitea-admin referenced this issue from a commit 2026-09-30 03:53:19 +00:00

docs(decisions): the PR #20 handoff moves out of DECISIONS-LOG.md into its own file

gitea-admin referenced this pull request 2026-09-30 03:53:59 +00:00

feat(conformance): the in-cluster Job can now run a named commit and get its results out #20

gitea-admin referenced this issue from a commit 2026-09-30 04:07:20 +00:00

docs(decisions): the PR #20 handoff moves out of DECISIONS-LOG.md into its own file

Sign in to join this conversation.

Reviewers

No Reviewers

jakman85

Labels

Clear labels

agent

automated-update

blast-radius

breaking

ci-tooling

dependencies

needs-review

own-image

range-widen

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

gitea-admin jakman85 (Keith Jackson) pm-agent renovate-bot

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#24