diff --git a/CLAUDE.md b/CLAUDE.md index 8489d19..acc7603 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,6 +4,125 @@ Read this before doing anything durable. It is the delivery contract for AI codi sessions and agents. The newest entry in `decisions/` is the current PM handoff and takes precedence on *what* to work on; this file governs *how* it lands. +**Start with §0.** It is first because it is the rule you can break before you have read +anything else: the very first `git checkout -b` in the wrong directory has already done the +damage. + +## 0. Work in your OWN workspace. Never in the shared checkout. + +Before you create a branch, edit a file, or run `git` anything: **make yourself a git +worktree outside the repository root, and do all of your work there.** One workspace per +agent run, no exceptions. + + # from the shared checkout, ONCE, at the start of your run + git fetch origin main + git worktree add -b / /path/outside/the/repo/wt- origin/main + cd /path/outside/the/repo/wt- # everything after this happens HERE + +`/path/outside/the/repo` means outside this repository's tree — your session scratchpad, +or a sibling like `/var/home/keith/source/wt-`. Branch from **`origin/main`**, never +from `main` and never from `HEAD`; see "stale base" below. + +**The shared checkout is `/var/home/keith/source/home-network-k3s-impl`** — the operator's +working copy, the one you are probably started in. In it, do **not**: + +- **`git checkout` / `git switch` / `git checkout -b`.** Its HEAD is shared state; moving + it moves the tree out from under whoever else is using it. +- **edit, create or delete a file.** Including "just one doc", including a file you intend + to move later. +- **`git add` / `git commit` / `git stash` / `git restore`.** `git add -A` there stages + whatever another agent left lying around; `git stash` *removes* their uncommitted work + into a place they will not look. +- **`git worktree remove` anything you did not create.** + +Read-only commands there are fine, and the one write you do need is creating your own +worktree. + +### Why — measured in the shared checkout, 2026-10-05 + +| What was measured | Value | +|---|---| +| `checkout: moving` entries in its reflog | **107** — the tree has been yanked between branches 107 times | +| the branch it was sitting on | `feat/pm-agent-issue-to-pr` — **not `main`** | +| uncommitted trees left in it by other runs | **2** (`infra/agent/`, `platform/pm-agent/`), mtimes five minutes apart the same evening | +| its local `main` vs `origin/main` | **two commits behind** (`4386d14` vs `22c94ef`) | +| registered worktrees | **42**, of which **31** are on branches already merged to `main` | +| gate scripts carrying an explicit `.claude/` exclusion | **9** | + +Each row is a distinct way the shared checkout damages work: + +- **Dirty tree, foreign files.** Two other runs' uncommitted directories were sitting in + it. An agent that edits there and runs `git add -A` ships someone else's half-finished + work inside its own PR, and the PR's author has no idea what those files are. This is not + hypothetical and it is not historical: **eleven minutes after those two directories were + measured, while this very section was being written, another run committed them** — + `5334abe feat(pm-agent): ...`, 1,000+ lines across `infra/agent/`, `clusters/`, `catalog/` + and `apps/` — onto the shared checkout's checked-out branch, moving its HEAD under anyone + reading it. Had this rule's author been working in that tree rather than in a worktree, + that commit and this one would have collided. +- **Stale base.** Its `main` was two commits behind `origin/main`, so + `git checkout -b x main` there branches off a tree that is already not what CI will merge + into — the gate you run locally proves a base nobody else has. Worse, it was not even *on* + `main`: a bare `git checkout -b x` would have branched off `feat/pm-agent-issue-to-pr` and + silently inherited that work into your PR. `git fetch` first, and name `origin/main` + explicitly as the start point. +- **Shared HEAD.** 107 branch switches in one tree is 107 chances to switch it while + another agent is mid-edit. The loser's changes are either carried onto the wrong branch + or lost to a checkout conflict — and the symptom surfaces in *their* session, not yours. + +### Put the workspace OUTSIDE the repo root + +A worktree *inside* the tree (`.claude/worktrees/agent-*/` — which is where Claude Code's +own default puts it) is a full second copy of the repo that every tree-walking check then +finds. Measured 2026-09-12: `find`-based discovery reported **the same manifest under test +once per worktree, up to 18 times**, and the L0 check comparing that listing against the +tree failed on paths that are not authored surface at all +(`platform/monitoring/tests/run.sh:115-123`). Nine gate scripts now carry an explicit +`.claude/` prune for this reason — which means the hazard is *worked around*, not fixed, +and the next check that walks the tree will hit it again. Keep the copy out of the tree and +none of that applies. + +### Clean up — and expect not to be able to + +31 of the 42 registered worktrees are on branches already merged to `main`. A registered +worktree **pins its branch**: while it exists, that branch cannot be deleted or checked out +anywhere else. So a left-behind workspace is a lock on a branch name, and the cost lands on +whoever next needs it. + +Remove yours when you are done: + + git worktree remove /path/outside/the/repo/wt- + +Measured 2026-10-05: on a worktree **you created yourself, in the session scratchpad**, that +command **succeeds** — rc 0, no prompt, with or without a branch checked out. The "worktree +removal is sandbox-blocked" note in circulation is about the `.claude/worktrees/agent-*` ones +belonging to *other* sessions, some of which are additionally `locked` on git's own terms. So: +remove yours, and only yours. If removal is refused, say so in your handoff and leave the +worktree registered — do **not** `rm -rf` the directory, which leaves a registered worktree +pointing at nothing and still holding the branch. + +If someone *else's* stale worktree is holding the branch you need, do not try to reclaim the +branch name. Push a `rework/` branch to the PR's refspec instead. + +### If you are already standing in the shared checkout with edits + +Stop. Do not `git stash` (that moves other agents' work too). Create your worktree as +above, redo the edits there, and leave the shared checkout's HEAD and working tree exactly +as you found them. + +### Gating from a worktree + +Three known breakages, all with workarounds, all required every run: see §3, "Running the +gate from a worktree". + +### If you are writing a brief for another agent + +The brief must name **its own explicit branch** and tell it to create **its own worktree** +from `origin/main`. "Work on the current branch" and "continue in the checkout" are how +branch sprawl and cross-contamination got here: an agent given no workspace uses the one it +is started in, which is the shared one. Also carry §3's three gate workarounds into the +brief — an agent that cannot gate will report rc 3 as green. + ## 1. Never push to `main`. Ever. Every durable change reaches `main` exactly one way: @@ -147,15 +266,21 @@ Exit codes are a contract — do not flatten them: | 3 | UNDETERMINED — a stage did not run. **Never report this as green.** | `spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out -beside this one; that single skip is expected and tolerated. Any *other* skip is a -real gap in your run. +beside this one; that single skip is expected and tolerated — and from a worktree it is +avoidable outright (§3's worktree list, item 3). `gitea-settings` is the other expected +skip: it needs `SETTINGS_API_TOKEN`, which exists only in CI, so **a local run cannot +return 0** — the honest best local result is rc 3 with `gitea-settings` as the only stage +that did not run. Any *further* skip is a real gap in your run. Report the rc and name the +skipped stages; never round rc 3 up. **Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`, so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts. -### Running the gate from a git worktree +### Running the gate from a worktree -Two unrelated failures, both with known workarounds — put both in any agent brief: +This is the **normal** path, not an edge case — §0 requires you to be in a worktree, so +every gate run you do hits all of these. Three known breakages, all with +workarounds, all of them yours to apply every run — put them in any agent brief: 1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked and exists only in the main checkout. Do **not** copy it (it is a credential). @@ -166,10 +291,17 @@ Two unrelated failures, both with known workarounds — put both in any agent br the `ENGINE` hook — a shim that intercepts `run` and re-execs `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in `tracked_files()` is this bug, not a broken tree. +3. **`spec-immutability` skips** — the skip called tolerated above is also *avoidable*. `dev.sh` + derives the sibling spec path from its own `REPO_ROOT`, which in a worktree is your + workspace, so `../home-network-k3s` does not resolve and it adds no mount. Mount the spec + repo read-only in the same `ENGINE` shim and pass `-e SPEC_REPO=/home-network-k3s`: with + both mounts the stage reports `pass`, not SKIPPED, and R-TEST-06 is actually covered + (measured — `decisions/2026-10-01-render-selftest-runs-in-gate.md`). ## 4. Branch naming -Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push** +Use `feat/`, `fix/`, `docs/`, `chore/`. **One branch, one workspace, one PR** — if you +need a second branch, make a second worktree (§0); do not re-point an existing one. Note that `.gitea/workflows/gate.yml`'s **push** trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run. The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected. diff --git a/README.md b/README.md index 39c2afc..c3e5bea 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,9 @@ specification** at [`../home-network-k3s`](../home-network-k3s). - **Done = a green, reproducible, requirement-tagged conformance test** — never "merged," never a manual demo (spec `docs/14`, `agents/definition-of-done.md`). - **No work on `main`.** Every change is a feature branch merged into `main` (spec `CONTRIBUTING.md`). +- **No work in a shared checkout.** Every agent run gets its own git worktree, outside the repo + root, branched from `origin/main`. This checkout is shared mutable state — see + [`CLAUDE.md` §0](CLAUDE.md). - **No plaintext secrets** — SOPS-encrypted only (spec `docs/07`). - **Phase discipline** — only build what the active roadmap phase authorizes (spec `roadmap/phases.md`). @@ -95,6 +98,20 @@ The running narrative — decisions, root causes, incidents, follow-ups — is ## Working on this repo +**First, get out of this checkout.** It is the operator's working copy and several agents +share it; concurrent runs here have overwritten each other's branches and left each other's +uncommitted files behind. One workspace per change: + +```bash +git fetch origin main +git worktree add -b / /path/outside/the/repo/wt- origin/main +cd /path/outside/the/repo/wt- +``` + +Outside the repo root specifically — a worktree under `.claude/worktrees/` is a second full +copy that tree-walking gate checks then count twice. The rule, the measurements behind it, and +the three gate workarounds a worktree needs are in [`CLAUDE.md` §0](CLAUDE.md) and §3. + Everything the gate needs is baked into the pinned dev container — **do not install the toolchain on the host**: diff --git a/decisions/2026-10-05-agents-work-in-their-own-workspace.md b/decisions/2026-10-05-agents-work-in-their-own-workspace.md new file mode 100644 index 0000000..91e268b --- /dev/null +++ b/decisions/2026-10-05-agents-work-in-their-own-workspace.md @@ -0,0 +1,147 @@ +--- +date: 2026-10-05 +branch: docs/agents-work-in-their-own-workspace +title: One workspace per agent run — the shared checkout is now documented as off-limits +--- + +## What changed + +Agents have been doing their work in the operator's checkout at +`/var/home/keith/source/home-network-k3s-impl` and clobbering each other. Nothing in the tree +told them not to: worktrees appeared in CLAUDE.md only as §3's *gate workarounds*, i.e. as an +awkward edge case, and `decisions/README.md` asserted worktree isolation as a fact about how +agents already behave rather than as a rule they must follow. Nowhere was the rule stated. + +Documentation only — no code, no manifests, no pipeline logic. + +- **`CLAUDE.md`: new `## 0. Work in your OWN workspace. Never in the shared checkout.`** + Placed before §1 so the section numbers every other file and handoff entry cites (§1, §2b, + §3, §7) all keep their meaning. It gives the three commands (`git fetch origin main`; + `git worktree add -b / origin/main`; `cd`), an explicit + do-not list for the shared checkout (no `checkout`/`switch`, no edits, no + `add`/`commit`/`stash`/`restore`, no removing others' worktrees), a measurement table, the + "put it outside the repo root" rule, cleanup, what to do if you are already standing in the + shared checkout with edits, and what a PM must put in a brief it writes for another agent. +- **`CLAUDE.md` intro** now says "Start with §0", with the reason: it is the rule you can + break before you have read anything else. +- **`CLAUDE.md` §3** retitled "Running the gate from a worktree" and reframed as the normal + path rather than an edge case, since §0 now makes every gate run a worktree run. Its list + grew from two breakages to three: `spec-immutability`'s skip is *avoidable*, not merely + tolerated, if the `ENGINE` shim also mounts the sibling spec repo read-only and passes + `-e SPEC_REPO=/home-network-k3s`. +- **`CLAUDE.md` §4** gained "One branch, one workspace, one PR". +- **`README.md`**: a fourth non-negotiable rule ("No work in a shared checkout"), and + "Working on this repo" now opens with how to get out of the checkout before describing the + dev container that runs in it. +- **`decisions/README.md`**: the worktree-isolation sentence now cites §0 as the requirement + instead of describing it as something agents happen to do. + +I did this work the way the new §0 requires — a worktree at +`/wt-workspace-rule`, outside the repo root, branched from `origin/main`, +with the shared checkout left on the branch and with the untracked files I found in it. + +## What I verified + +Every number in the new §0 table was measured in the shared checkout on 2026-10-05, not +recalled: + +| Claim | Command | Value | +|---|---|---| +| shared HEAD is yanked between branches | `git reflog \| grep -c 'checkout: moving'` | **107** | +| it is not sitting on `main` | `git worktree list` | `feat/pm-agent-issue-to-pr` | +| other runs' uncommitted work is in it | `git status --porcelain` | **2** trees: `infra/agent/` (22:04), `platform/pm-agent/` (22:09) — `stat -c %y` | +| its `main` is stale | `git rev-parse main` vs `origin/main` | `4386d14` vs `22c94ef`, **2 behind** | +| worktrees registered / already merged | `git worktree list --porcelain` + `git merge-base --is-ancestor main` | **42** branches, **31** merged | +| nested worktrees are a gate hazard | `grep -rl '\.claude' conformance/tests platform/monitoring/tests` | **9** scripts carry an explicit exclusion | +| the 18× double-count | read `platform/monitoring/tests/run.sh:115-123` | in-tree comment, observed 2026-09-12 | + +**The failure mode was observed live, mid-session, not just reconstructed.** The two untracked +trees I measured at 22:04 and 22:09 were committed at 22:20 by another run — `5334abe` +`feat(pm-agent): an in-cluster AI Program Manager...`, 1,000+ lines across `infra/agent/`, +`clusters/`, `catalog/`, `apps/`, `docs/plans/` and its own `decisions/` entry — onto the shared +checkout's checked-out branch. Measured after the fact: the shared checkout's HEAD moved +`22c94ef` → `5334abe` and its working tree went clean between my first `git status` and my last, +with `git log -1` naming a commit that is not mine. That is §0's "shared HEAD" row happening +during the session that wrote §0, and it is now cited in the doc. + +**A claim of mine was wrong and is corrected in the doc.** I had it recorded that +`git worktree remove` is blocked by the agent sandbox. Provoked rather than assumed: created +a throwaway detached worktree in the scratchpad and removed it — **rc 0**, and +`git worktree list` no longer shows it. §0 therefore tells agents to remove their own, and +scopes the "blocked" note to what it actually covers: *other* sessions' +`.claude/worktrees/agent-*`, some of which are additionally `locked` on git's own terms. The +`rework/` escape hatch is now scoped to someone else's stale worktree holding a +branch, which is a real and separate problem (31 of them). Both of my own worktrees in this +session were removed cleanly at rc 0 after pushing. + +`spec-immutability` is avoidable-not-tolerated: taken from +`decisions/2026-10-01-render-selftest-runs-in-gate.md`, which records +`stage spec-immutability: pass` under the two-mount shim plus `-e SPEC_REPO=/home-network-k3s`. + +**Gate — `rc 3`, UNDETERMINED, and I am not calling it green.** Pinned toolbox, from this +worktree, `DEV_KUBECONFIG=/dev/null` and an `ENGINE` shim bind-mounting both the main +checkout's `.git` (at its own host path) and the sibling spec repo read-only with +`-e SPEC_REPO=/home-network-k3s`: + + ENGINE= DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' # rc 3 + +| stage | result | +|---|---| +| spec-immutability | **pass** — it genuinely RAN; the second mount is what makes it run from a worktree | +| approval-policy | pass, but **N/A**: `event='none'`. It cannot see a PR locally, so this green says nothing. See below | +| decisions-log | pass — the three required headings and the frozen-archive pin | +| lint · l0 · render-golden · l1 | pass (L1: 3 offline checks, 0 failures, 0 skips, 0 unwired) | +| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset; it exists only in CI | +| conformance (L2/L3) | SKIPPED — `CLUSTER_AVAILABLE!=true`; the decision notes no cluster-needing requirement was touched | + +`gitea-settings` is the *only* stage that did not run, and it cannot be made to run from a +workstation, so rc 3 is the ceiling for any local run — not a defect in this branch. The +first gate attempt returned rc 3 for a different and real reason (`HEAD is not ahead of +origin/main`: I had not committed yet, so the touched-requirement set was empty by default +rather than by verification); it was re-run after committing, and the table above is the +post-commit run. §3 now records both expected skips, because it previously named only +`spec-immutability`. + +## What I did NOT verify + +- **GATE: UNVERIFIED at the time of writing.** The pinned-toolbox result for this branch and + its exit code are reported in the PR, not predicted here. **A local rc 3 is not green** and + will not be described as one. +- **CI gate: RED, measured — `MERGE GATE: BLOCKED`, one failing stage.** Run 245 on PR #56, + head `b988da8b`, 1m47s. The only failure is `approval-policy`: *"a sensitive change lacks a + non-author approval"*, `sensitive paths touched: CLAUDE.md`, author `gitea-admin`. Every other + stage passed — `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, and `gitea-settings` + (which can only run in CI, and reported `required_approvals = 0`, + `dismiss_stale_approvals = true`). `spec-immutability` SKIPPED in CI (no sibling spec repo in + the CI workspace — the inverse of the local run, where the two-mount shim made it pass) and + `conformance` SKIPPED. **Note `gate / conformance (pull_request)` reports status `success` + with description "Has been skipped"** — CLAUDE.md §2b's trap, live on this very PR. Nothing + in this branch is broken; the red is the structural human gate, below. *Measured at + `b988da8b`; this entry was then amended to record it, so the head SHA in the PR is newer than + the run quoted here and CI re-ran on the amended commit.* +- **This PR is a sensitive-path PR and is CI-RED on `approval-policy` until a human acts.** + `CLAUDE.md` is in `infra/gitea/approval-policy.json`'s `sensitive_paths` ("the delivery + contract agents follow"), so the stage requires an approving review from someone who is not + the author, or an `/approve-override ` posted by an administrator **as a PR review + of type "Comment"**, not in the plain comment box. `ci/approval-policy.sh` run locally + reports N/A (it cannot see a PR), so the local gate cannot catch this and a green local run + says nothing about it. **I have not posted an override and must not** — CLAUDE.md §1 forbids + an agent clearing this on its own PR even though `gitea-admin` holds `owner` and could. +- **Nothing here is enforced.** §0 is prose. There is no gate stage that detects work done in + the shared checkout, no check that a worktree is outside the repo root, and no check that a + branch was cut from `origin/main`. Every other rule in CLAUDE.md that agents kept breaking + (`DECISIONS-LOG.md` appends, the three handoff headings) ended up as a gate stage, and that + is the honest follow-up here. I did not write one: the ask was documentation, and an + enforcement stage lands in `ci/**`, which is itself a sensitive path and deserves its own PR. +- **I did not clean up the 31 stale merged worktrees or the 35 nested under + `.claude/worktrees/`.** They are other sessions' workspaces; removing them is an operator + action, and §0 explicitly tells agents not to remove worktrees they did not create. +- **I did not touch the untracked `infra/agent/` and `platform/pm-agent/` trees** sitting in the + shared checkout. They are another run's in-flight work. Whose, and whether they are wanted, + is unknown to me. +- **`docs/plans/ssd-etcd-execution-runbook.md:18` still points at a `.claude/worktrees/` path** + inside the repo. Left as-is deliberately: that runbook is marked COMPLETED/historical and + describes the now-decommissioned dev cluster, so the path is a record of what was done, not + an instruction. It does mean a grep for in-repo worktree paths is not yet clean. +- **No runtime, cluster or Flux behaviour was touched or checked.** This branch changes three + Markdown files; `kubectl` was not run. diff --git a/decisions/README.md b/decisions/README.md index b3cadd2..32458fe 100644 --- a/decisions/README.md +++ b/decisions/README.md @@ -26,7 +26,8 @@ The fix is the obvious one for concurrent appenders: **stop sharing a file**. Ea handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is nothing for git to merge and nothing to conflict. This scales to any number of parallel branches with zero coordination — which is the point, since the agents are deliberately -isolated from each other in separate worktrees and cannot coordinate. +isolated from each other in separate worktrees (required by `CLAUDE.md` §0) and cannot +coordinate. There is deliberately **no index file** and no counter. An index would be a shared mutable file — exactly the thing that just failed — and a sequence number would have