Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
18f741d744 | ||
|
|
3ee663dbcc | ||
|
|
ad3f56eee8 | ||
|
|
2cb891c05d |
No files matched your search
@@ -4,6 +4,125 @@ Read this before doing anything durable. It is the delivery contract for AI codi
|
||||
sessions and agents. The newest entry in `decisions/` is the current PM handoff and
|
||||
takes precedence on *what* to work on; this file governs *how* it lands.
|
||||
|
||||
**Start with §0.** It is first because it is the rule you can break before you have read
|
||||
anything else: the very first `git checkout -b` in the wrong directory has already done the
|
||||
damage.
|
||||
|
||||
## 0. Work in your OWN workspace. Never in the shared checkout.
|
||||
|
||||
Before you create a branch, edit a file, or run `git` anything: **make yourself a git
|
||||
worktree outside the repository root, and do all of your work there.** One workspace per
|
||||
agent run, no exceptions.
|
||||
|
||||
# from the shared checkout, ONCE, at the start of your run
|
||||
git fetch origin main
|
||||
git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
|
||||
cd /path/outside/the/repo/wt-<slug> # everything after this happens HERE
|
||||
|
||||
`/path/outside/the/repo` means outside this repository's tree — your session scratchpad,
|
||||
or a sibling like `/var/home/keith/source/wt-<slug>`. Branch from **`origin/main`**, never
|
||||
from `main` and never from `HEAD`; see "stale base" below.
|
||||
|
||||
**The shared checkout is `/var/home/keith/source/home-network-k3s-impl`** — the operator's
|
||||
working copy, the one you are probably started in. In it, do **not**:
|
||||
|
||||
- **`git checkout` / `git switch` / `git checkout -b`.** Its HEAD is shared state; moving
|
||||
it moves the tree out from under whoever else is using it.
|
||||
- **edit, create or delete a file.** Including "just one doc", including a file you intend
|
||||
to move later.
|
||||
- **`git add` / `git commit` / `git stash` / `git restore`.** `git add -A` there stages
|
||||
whatever another agent left lying around; `git stash` *removes* their uncommitted work
|
||||
into a place they will not look.
|
||||
- **`git worktree remove` anything you did not create.**
|
||||
|
||||
Read-only commands there are fine, and the one write you do need is creating your own
|
||||
worktree.
|
||||
|
||||
### Why — measured in the shared checkout, 2026-10-05
|
||||
|
||||
| What was measured | Value |
|
||||
|---|---|
|
||||
| `checkout: moving` entries in its reflog | **107** — the tree has been yanked between branches 107 times |
|
||||
| the branch it was sitting on | `feat/pm-agent-issue-to-pr` — **not `main`** |
|
||||
| uncommitted trees left in it by other runs | **2** (`infra/agent/`, `platform/pm-agent/`), mtimes five minutes apart the same evening |
|
||||
| its local `main` vs `origin/main` | **two commits behind** (`4386d14` vs `22c94ef`) |
|
||||
| registered worktrees | **42**, of which **31** are on branches already merged to `main` |
|
||||
| gate scripts carrying an explicit `.claude/` exclusion | **9** |
|
||||
|
||||
Each row is a distinct way the shared checkout damages work:
|
||||
|
||||
- **Dirty tree, foreign files.** Two other runs' uncommitted directories were sitting in
|
||||
it. An agent that edits there and runs `git add -A` ships someone else's half-finished
|
||||
work inside its own PR, and the PR's author has no idea what those files are. This is not
|
||||
hypothetical and it is not historical: **eleven minutes after those two directories were
|
||||
measured, while this very section was being written, another run committed them** —
|
||||
`5334abe feat(pm-agent): ...`, 1,000+ lines across `infra/agent/`, `clusters/`, `catalog/`
|
||||
and `apps/` — onto the shared checkout's checked-out branch, moving its HEAD under anyone
|
||||
reading it. Had this rule's author been working in that tree rather than in a worktree,
|
||||
that commit and this one would have collided.
|
||||
- **Stale base.** Its `main` was two commits behind `origin/main`, so
|
||||
`git checkout -b x main` there branches off a tree that is already not what CI will merge
|
||||
into — the gate you run locally proves a base nobody else has. Worse, it was not even *on*
|
||||
`main`: a bare `git checkout -b x` would have branched off `feat/pm-agent-issue-to-pr` and
|
||||
silently inherited that work into your PR. `git fetch` first, and name `origin/main`
|
||||
explicitly as the start point.
|
||||
- **Shared HEAD.** 107 branch switches in one tree is 107 chances to switch it while
|
||||
another agent is mid-edit. The loser's changes are either carried onto the wrong branch
|
||||
or lost to a checkout conflict — and the symptom surfaces in *their* session, not yours.
|
||||
|
||||
### Put the workspace OUTSIDE the repo root
|
||||
|
||||
A worktree *inside* the tree (`.claude/worktrees/agent-*/` — which is where Claude Code's
|
||||
own default puts it) is a full second copy of the repo that every tree-walking check then
|
||||
finds. Measured 2026-09-12: `find`-based discovery reported **the same manifest under test
|
||||
once per worktree, up to 18 times**, and the L0 check comparing that listing against the
|
||||
tree failed on paths that are not authored surface at all
|
||||
(`platform/monitoring/tests/run.sh:115-123`). Nine gate scripts now carry an explicit
|
||||
`.claude/` prune for this reason — which means the hazard is *worked around*, not fixed,
|
||||
and the next check that walks the tree will hit it again. Keep the copy out of the tree and
|
||||
none of that applies.
|
||||
|
||||
### Clean up — and expect not to be able to
|
||||
|
||||
31 of the 42 registered worktrees are on branches already merged to `main`. A registered
|
||||
worktree **pins its branch**: while it exists, that branch cannot be deleted or checked out
|
||||
anywhere else. So a left-behind workspace is a lock on a branch name, and the cost lands on
|
||||
whoever next needs it.
|
||||
|
||||
Remove yours when you are done:
|
||||
|
||||
git worktree remove /path/outside/the/repo/wt-<slug>
|
||||
|
||||
Measured 2026-10-05: on a worktree **you created yourself, in the session scratchpad**, that
|
||||
command **succeeds** — rc 0, no prompt, with or without a branch checked out. The "worktree
|
||||
removal is sandbox-blocked" note in circulation is about the `.claude/worktrees/agent-*` ones
|
||||
belonging to *other* sessions, some of which are additionally `locked` on git's own terms. So:
|
||||
remove yours, and only yours. If removal is refused, say so in your handoff and leave the
|
||||
worktree registered — do **not** `rm -rf` the directory, which leaves a registered worktree
|
||||
pointing at nothing and still holding the branch.
|
||||
|
||||
If someone *else's* stale worktree is holding the branch you need, do not try to reclaim the
|
||||
branch name. Push a `rework/<pr-number>` branch to the PR's refspec instead.
|
||||
|
||||
### If you are already standing in the shared checkout with edits
|
||||
|
||||
Stop. Do not `git stash` (that moves other agents' work too). Create your worktree as
|
||||
above, redo the edits there, and leave the shared checkout's HEAD and working tree exactly
|
||||
as you found them.
|
||||
|
||||
### Gating from a worktree
|
||||
|
||||
Three known breakages, all with workarounds, all required every run: see §3, "Running the
|
||||
gate from a worktree".
|
||||
|
||||
### If you are writing a brief for another agent
|
||||
|
||||
The brief must name **its own explicit branch** and tell it to create **its own worktree**
|
||||
from `origin/main`. "Work on the current branch" and "continue in the checkout" are how
|
||||
branch sprawl and cross-contamination got here: an agent given no workspace uses the one it
|
||||
is started in, which is the shared one. Also carry §3's three gate workarounds into the
|
||||
brief — an agent that cannot gate will report rc 3 as green.
|
||||
|
||||
## 1. Never push to `main`. Ever.
|
||||
|
||||
Every durable change reaches `main` exactly one way:
|
||||
@@ -147,15 +266,21 @@ Exit codes are a contract — do not flatten them:
|
||||
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
|
||||
|
||||
`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
|
||||
beside this one; that single skip is expected and tolerated. Any *other* skip is a
|
||||
real gap in your run.
|
||||
beside this one; that single skip is expected and tolerated — and from a worktree it is
|
||||
avoidable outright (§3's worktree list, item 3). `gitea-settings` is the other expected
|
||||
skip: it needs `SETTINGS_API_TOKEN`, which exists only in CI, so **a local run cannot
|
||||
return 0** — the honest best local result is rc 3 with `gitea-settings` as the only stage
|
||||
that did not run. Any *further* skip is a real gap in your run. Report the rc and name the
|
||||
skipped stages; never round rc 3 up.
|
||||
|
||||
**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
|
||||
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
|
||||
|
||||
### Running the gate from a git worktree
|
||||
### Running the gate from a worktree
|
||||
|
||||
Two unrelated failures, both with known workarounds — put both in any agent brief:
|
||||
This is the **normal** path, not an edge case — §0 requires you to be in a worktree, so
|
||||
every gate run you do hits all of these. Three known breakages, all with
|
||||
workarounds, all of them yours to apply every run — put them in any agent brief:
|
||||
|
||||
1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
|
||||
and exists only in the main checkout. Do **not** copy it (it is a credential).
|
||||
@@ -166,10 +291,17 @@ Two unrelated failures, both with known workarounds — put both in any agent br
|
||||
the `ENGINE` hook — a shim that intercepts `run` and re-execs
|
||||
`podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
|
||||
`tracked_files()` is this bug, not a broken tree.
|
||||
3. **`spec-immutability` skips** — the skip called tolerated above is also *avoidable*. `dev.sh`
|
||||
derives the sibling spec path from its own `REPO_ROOT`, which in a worktree is your
|
||||
workspace, so `../home-network-k3s` does not resolve and it adds no mount. Mount the spec
|
||||
repo read-only in the same `ENGINE` shim and pass `-e SPEC_REPO=/home-network-k3s`: with
|
||||
both mounts the stage reports `pass`, not SKIPPED, and R-TEST-06 is actually covered
|
||||
(measured — `decisions/2026-10-01-render-selftest-runs-in-gate.md`).
|
||||
|
||||
## 4. Branch naming
|
||||
|
||||
Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
|
||||
Use `feat/`, `fix/`, `docs/`, `chore/`. **One branch, one workspace, one PR** — if you
|
||||
need a second branch, make a second worktree (§0); do not re-point an existing one. Note that `.gitea/workflows/gate.yml`'s **push**
|
||||
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
|
||||
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
|
||||
|
||||
|
||||
@@ -11,6 +11,9 @@ specification** at [`../home-network-k3s`](../home-network-k3s).
|
||||
- **Done = a green, reproducible, requirement-tagged conformance test** — never "merged," never a
|
||||
manual demo (spec `docs/14`, `agents/definition-of-done.md`).
|
||||
- **No work on `main`.** Every change is a feature branch merged into `main` (spec `CONTRIBUTING.md`).
|
||||
- **No work in a shared checkout.** Every agent run gets its own git worktree, outside the repo
|
||||
root, branched from `origin/main`. This checkout is shared mutable state — see
|
||||
[`CLAUDE.md` §0](CLAUDE.md).
|
||||
- **No plaintext secrets** — SOPS-encrypted only (spec `docs/07`).
|
||||
- **Phase discipline** — only build what the active roadmap phase authorizes (spec `roadmap/phases.md`).
|
||||
|
||||
@@ -95,6 +98,20 @@ The running narrative — decisions, root causes, incidents, follow-ups — is
|
||||
|
||||
## Working on this repo
|
||||
|
||||
**First, get out of this checkout.** It is the operator's working copy and several agents
|
||||
share it; concurrent runs here have overwritten each other's branches and left each other's
|
||||
uncommitted files behind. One workspace per change:
|
||||
|
||||
```bash
|
||||
git fetch origin main
|
||||
git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
|
||||
cd /path/outside/the/repo/wt-<slug>
|
||||
```
|
||||
|
||||
Outside the repo root specifically — a worktree under `.claude/worktrees/` is a second full
|
||||
copy that tree-walking gate checks then count twice. The rule, the measurements behind it, and
|
||||
the three gate workarounds a worktree needs are in [`CLAUDE.md` §0](CLAUDE.md) and §3.
|
||||
|
||||
Everything the gate needs is baked into the pinned dev container — **do not install the
|
||||
toolchain on the host**:
|
||||
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
date: 2026-10-05
|
||||
branch: docs/agents-work-in-their-own-workspace
|
||||
title: One workspace per agent run — the shared checkout is now documented as off-limits
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Agents have been doing their work in the operator's checkout at
|
||||
`/var/home/keith/source/home-network-k3s-impl` and clobbering each other. Nothing in the tree
|
||||
told them not to: worktrees appeared in CLAUDE.md only as §3's *gate workarounds*, i.e. as an
|
||||
awkward edge case, and `decisions/README.md` asserted worktree isolation as a fact about how
|
||||
agents already behave rather than as a rule they must follow. Nowhere was the rule stated.
|
||||
|
||||
Documentation only — no code, no manifests, no pipeline logic.
|
||||
|
||||
- **`CLAUDE.md`: new `## 0. Work in your OWN workspace. Never in the shared checkout.`**
|
||||
Placed before §1 so the section numbers every other file and handoff entry cites (§1, §2b,
|
||||
§3, §7) all keep their meaning. It gives the three commands (`git fetch origin main`;
|
||||
`git worktree add -b <type>/<slug> <path outside the repo> origin/main`; `cd`), an explicit
|
||||
do-not list for the shared checkout (no `checkout`/`switch`, no edits, no
|
||||
`add`/`commit`/`stash`/`restore`, no removing others' worktrees), a measurement table, the
|
||||
"put it outside the repo root" rule, cleanup, what to do if you are already standing in the
|
||||
shared checkout with edits, and what a PM must put in a brief it writes for another agent.
|
||||
- **`CLAUDE.md` intro** now says "Start with §0", with the reason: it is the rule you can
|
||||
break before you have read anything else.
|
||||
- **`CLAUDE.md` §3** retitled "Running the gate from a worktree" and reframed as the normal
|
||||
path rather than an edge case, since §0 now makes every gate run a worktree run. Its list
|
||||
grew from two breakages to three: `spec-immutability`'s skip is *avoidable*, not merely
|
||||
tolerated, if the `ENGINE` shim also mounts the sibling spec repo read-only and passes
|
||||
`-e SPEC_REPO=/home-network-k3s`.
|
||||
- **`CLAUDE.md` §4** gained "One branch, one workspace, one PR".
|
||||
- **`README.md`**: a fourth non-negotiable rule ("No work in a shared checkout"), and
|
||||
"Working on this repo" now opens with how to get out of the checkout before describing the
|
||||
dev container that runs in it.
|
||||
- **`decisions/README.md`**: the worktree-isolation sentence now cites §0 as the requirement
|
||||
instead of describing it as something agents happen to do.
|
||||
|
||||
I did this work the way the new §0 requires — a worktree at
|
||||
`<session scratchpad>/wt-workspace-rule`, outside the repo root, branched from `origin/main`,
|
||||
with the shared checkout left on the branch and with the untracked files I found in it.
|
||||
|
||||
## What I verified
|
||||
|
||||
Every number in the new §0 table was measured in the shared checkout on 2026-10-05, not
|
||||
recalled:
|
||||
|
||||
| Claim | Command | Value |
|
||||
|---|---|---|
|
||||
| shared HEAD is yanked between branches | `git reflog \| grep -c 'checkout: moving'` | **107** |
|
||||
| it is not sitting on `main` | `git worktree list` | `feat/pm-agent-issue-to-pr` |
|
||||
| other runs' uncommitted work is in it | `git status --porcelain` | **2** trees: `infra/agent/` (22:04), `platform/pm-agent/` (22:09) — `stat -c %y` |
|
||||
| its `main` is stale | `git rev-parse main` vs `origin/main` | `4386d14` vs `22c94ef`, **2 behind** |
|
||||
| worktrees registered / already merged | `git worktree list --porcelain` + `git merge-base --is-ancestor <b> main` | **42** branches, **31** merged |
|
||||
| nested worktrees are a gate hazard | `grep -rl '\.claude' conformance/tests platform/monitoring/tests` | **9** scripts carry an explicit exclusion |
|
||||
| the 18× double-count | read `platform/monitoring/tests/run.sh:115-123` | in-tree comment, observed 2026-09-12 |
|
||||
|
||||
**The failure mode was observed live, mid-session, not just reconstructed.** The two untracked
|
||||
trees I measured at 22:04 and 22:09 were committed at 22:20 by another run — `5334abe`
|
||||
`feat(pm-agent): an in-cluster AI Program Manager...`, 1,000+ lines across `infra/agent/`,
|
||||
`clusters/`, `catalog/`, `apps/`, `docs/plans/` and its own `decisions/` entry — onto the shared
|
||||
checkout's checked-out branch. Measured after the fact: the shared checkout's HEAD moved
|
||||
`22c94ef` → `5334abe` and its working tree went clean between my first `git status` and my last,
|
||||
with `git log -1` naming a commit that is not mine. That is §0's "shared HEAD" row happening
|
||||
during the session that wrote §0, and it is now cited in the doc.
|
||||
|
||||
**A claim of mine was wrong and is corrected in the doc.** I had it recorded that
|
||||
`git worktree remove` is blocked by the agent sandbox. Provoked rather than assumed: created
|
||||
a throwaway detached worktree in the scratchpad and removed it — **rc 0**, and
|
||||
`git worktree list` no longer shows it. §0 therefore tells agents to remove their own, and
|
||||
scopes the "blocked" note to what it actually covers: *other* sessions'
|
||||
`.claude/worktrees/agent-*`, some of which are additionally `locked` on git's own terms. The
|
||||
`rework/<pr-number>` escape hatch is now scoped to someone else's stale worktree holding a
|
||||
branch, which is a real and separate problem (31 of them). Both of my own worktrees in this
|
||||
session were removed cleanly at rc 0 after pushing.
|
||||
|
||||
`spec-immutability` is avoidable-not-tolerated: taken from
|
||||
`decisions/2026-10-01-render-selftest-runs-in-gate.md`, which records
|
||||
`stage spec-immutability: pass` under the two-mount shim plus `-e SPEC_REPO=/home-network-k3s`.
|
||||
|
||||
**Gate — `rc 3`, UNDETERMINED, and I am not calling it green.** Pinned toolbox, from this
|
||||
worktree, `DEV_KUBECONFIG=/dev/null` and an `ENGINE` shim bind-mounting both the main
|
||||
checkout's `.git` (at its own host path) and the sibling spec repo read-only with
|
||||
`-e SPEC_REPO=/home-network-k3s`:
|
||||
|
||||
ENGINE=<two-mount shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' # rc 3
|
||||
|
||||
| stage | result |
|
||||
|---|---|
|
||||
| spec-immutability | **pass** — it genuinely RAN; the second mount is what makes it run from a worktree |
|
||||
| approval-policy | pass, but **N/A**: `event='none'`. It cannot see a PR locally, so this green says nothing. See below |
|
||||
| decisions-log | pass — the three required headings and the frozen-archive pin |
|
||||
| lint · l0 · render-golden · l1 | pass (L1: 3 offline checks, 0 failures, 0 skips, 0 unwired) |
|
||||
| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset; it exists only in CI |
|
||||
| conformance (L2/L3) | SKIPPED — `CLUSTER_AVAILABLE!=true`; the decision notes no cluster-needing requirement was touched |
|
||||
|
||||
`gitea-settings` is the *only* stage that did not run, and it cannot be made to run from a
|
||||
workstation, so rc 3 is the ceiling for any local run — not a defect in this branch. The
|
||||
first gate attempt returned rc 3 for a different and real reason (`HEAD is not ahead of
|
||||
origin/main`: I had not committed yet, so the touched-requirement set was empty by default
|
||||
rather than by verification); it was re-run after committing, and the table above is the
|
||||
post-commit run. §3 now records both expected skips, because it previously named only
|
||||
`spec-immutability`.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **GATE: UNVERIFIED at the time of writing.** The pinned-toolbox result for this branch and
|
||||
its exit code are reported in the PR, not predicted here. **A local rc 3 is not green** and
|
||||
will not be described as one.
|
||||
- **CI gate: RED, measured — `MERGE GATE: BLOCKED`, one failing stage.** Run 245 on PR #56,
|
||||
head `b988da8b`, 1m47s. The only failure is `approval-policy`: *"a sensitive change lacks a
|
||||
non-author approval"*, `sensitive paths touched: CLAUDE.md`, author `gitea-admin`. Every other
|
||||
stage passed — `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, and `gitea-settings`
|
||||
(which can only run in CI, and reported `required_approvals = 0`,
|
||||
`dismiss_stale_approvals = true`). `spec-immutability` SKIPPED in CI (no sibling spec repo in
|
||||
the CI workspace — the inverse of the local run, where the two-mount shim made it pass) and
|
||||
`conformance` SKIPPED. **Note `gate / conformance (pull_request)` reports status `success`
|
||||
with description "Has been skipped"** — CLAUDE.md §2b's trap, live on this very PR. Nothing
|
||||
in this branch is broken; the red is the structural human gate, below. *Measured at
|
||||
`b988da8b`; this entry was then amended to record it, so the head SHA in the PR is newer than
|
||||
the run quoted here and CI re-ran on the amended commit.*
|
||||
- **This PR is a sensitive-path PR and is CI-RED on `approval-policy` until a human acts.**
|
||||
`CLAUDE.md` is in `infra/gitea/approval-policy.json`'s `sensitive_paths` ("the delivery
|
||||
contract agents follow"), so the stage requires an approving review from someone who is not
|
||||
the author, or an `/approve-override <reason>` posted by an administrator **as a PR review
|
||||
of type "Comment"**, not in the plain comment box. `ci/approval-policy.sh` run locally
|
||||
reports N/A (it cannot see a PR), so the local gate cannot catch this and a green local run
|
||||
says nothing about it. **I have not posted an override and must not** — CLAUDE.md §1 forbids
|
||||
an agent clearing this on its own PR even though `gitea-admin` holds `owner` and could.
|
||||
- **Nothing here is enforced.** §0 is prose. There is no gate stage that detects work done in
|
||||
the shared checkout, no check that a worktree is outside the repo root, and no check that a
|
||||
branch was cut from `origin/main`. Every other rule in CLAUDE.md that agents kept breaking
|
||||
(`DECISIONS-LOG.md` appends, the three handoff headings) ended up as a gate stage, and that
|
||||
is the honest follow-up here. I did not write one: the ask was documentation, and an
|
||||
enforcement stage lands in `ci/**`, which is itself a sensitive path and deserves its own PR.
|
||||
- **I did not clean up the 31 stale merged worktrees or the 35 nested under
|
||||
`.claude/worktrees/`.** They are other sessions' workspaces; removing them is an operator
|
||||
action, and §0 explicitly tells agents not to remove worktrees they did not create.
|
||||
- **I did not touch the untracked `infra/agent/` and `platform/pm-agent/` trees** sitting in the
|
||||
shared checkout. They are another run's in-flight work. Whose, and whether they are wanted,
|
||||
is unknown to me.
|
||||
- **`docs/plans/ssd-etcd-execution-runbook.md:18` still points at a `.claude/worktrees/` path**
|
||||
inside the repo. Left as-is deliberately: that runbook is marked COMPLETED/historical and
|
||||
describes the now-decommissioned dev cluster, so the path is a record of what was done, not
|
||||
an instruction. It does mean a grep for in-repo worktree paths is not yet clean.
|
||||
- **No runtime, cluster or Flux behaviour was touched or checked.** This branch changes three
|
||||
Markdown files; `kubectl` was not run.
|
||||
@@ -0,0 +1,112 @@
|
||||
---
|
||||
date: 2026-10-05
|
||||
branch: fix/vpn-runbook-ansible-command
|
||||
title: vpn-remote-access runbook — step 1 named a script that does not exist, and step 2 cited the wrong reason the key is gitignored
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Two defects in `runbooks/vpn-remote-access.md` (merged in #55), both authored by me, both reported
|
||||
by the operator the first time they tried to follow it.
|
||||
|
||||
**1. Step 1 named a command that does not exist.** It said:
|
||||
|
||||
cd infra/ansible && ./site.sh --tags system --limit k3s_prod
|
||||
|
||||
There is **no `site.sh` anywhere in this repo** (`find . -name site.sh` → nothing). The invocation
|
||||
was wrong in four separate ways at once, which is what makes it worth recording rather than just
|
||||
patching: wrong kind of thing (this repo drives Ansible with `ansible-playbook <playbook>`, never a
|
||||
wrapper script — `playbooks/site.yml` exists and is the full bring-up chain, not this); wrong
|
||||
playbook (the module work lives in `playbooks/node-baseline.yml`); wrong targeting flag (`--limit`
|
||||
instead of this repo's `-e k3s_hosts=...` convention, which `infra/ansible/README.md` "Targeting
|
||||
prod" documents for every playbook); and no toolbox wrapper, though Ansible exists only inside the
|
||||
pinned container and `ansible.cfg` is only read from `infra/ansible`.
|
||||
|
||||
Replaced with the real command, plus the three things about it that are load-bearing:
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
|
||||
|
||||
- `-e k3s_hosts=k3s_prod` — the playbook defaults to `k3s_servers`, which is the **dev** group;
|
||||
dev is decommissioned, so an unqualified run does nothing useful.
|
||||
- `--tags system`, never untagged — the two kernel-module tasks carry that tag, and an untagged run
|
||||
would also hit the `disks` tasks, which assert on block-device variables prod deliberately leaves
|
||||
as placeholders (the playbook's own header warns about this). `--tags system` is broader than the
|
||||
modules alone (sysctls, iscsid, qemu-guest-agent, swap masking) but each is idempotent and
|
||||
already in its desired state on a live node.
|
||||
|
||||
**2. Step 2 gave the wrong reason the server key is safe from git — and the right reason is
|
||||
fragile.** It told the operator to stage the key at `/work/.cache/server.key` "inside the repo's
|
||||
gitignored cache". `.cache/` is **not** gitignored: `git check-ignore -v .cache/serverkey.txt`
|
||||
matches nothing. The file was ignored only incidentally, by `.gitignore` line 11 `*.key` — so the
|
||||
path happened to be safe for a reason I had not identified, and **any operator who renamed it
|
||||
(`server.pem`, `wg-server`, `privatekey`) would have had a committable WireGuard private key sitting
|
||||
in the checkout.** The gate would not have caught it either; `no-plaintext-secrets` does not
|
||||
recognise a bare WireGuard key.
|
||||
|
||||
The step now states the real rule, marks the `.key` suffix as the thing doing the work, and gives
|
||||
the operator a verification they can run before trusting it (`git check-ignore -v` must print a
|
||||
rule, and `git status --short` must not list the file).
|
||||
|
||||
Nothing outside `runbooks/vpn-remote-access.md` changed; no manifest, no behaviour.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **`site.sh` does not exist**: `find . -name 'site.sh' -not -path './.git/*'` → no output.
|
||||
- **The real surface**: `infra/ansible/playbooks/` contains `node-baseline.yml` and `site.yml` (and
|
||||
11 others); `node-baseline.yml`'s own header documents both
|
||||
`-e k3s_hosts=k3s_prod` and the tag-scoping hazard. The `-e <playbook>_hosts=` convention is
|
||||
confirmed by `infra/ansible/README.md` §"Targeting prod" and by the existing
|
||||
`runbooks/node-memory-swap.md`, which invokes `ansible-playbook playbooks/node-health.yml -e
|
||||
node_health_hosts=...`.
|
||||
- **`k3s_prod` is a real inventory group**: `[k3s_prod:children]` in `inventory/hosts.ini`, with
|
||||
`[k3s_prod_primary]`/`[k3s_prod_secondary]`.
|
||||
- **`--tags system` covers the kernel-module tasks**: both "Persist kernel modules…" and "Load
|
||||
kernel modules now" carry `tags: [system]` in `roles/node_baseline/tasks/main.yml`.
|
||||
- **The gitignore claim, measured both directions**:
|
||||
`git check-ignore -v .cache/server.key` → `.gitignore:11:*.key`;
|
||||
`git check-ignore -v .cache/serverkey.txt` → no match. So the suffix is the only thing ignoring it.
|
||||
- **Every path the component's docs reference resolves**: all six relative markdown links in
|
||||
`platform/vpn-ingress/README.md` (checked from that file's own directory), and both Ansible
|
||||
playbook paths.
|
||||
- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
|
||||
`lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; the rc 3 is the two
|
||||
environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no `CLUSTER_AVAILABLE`).
|
||||
**Not green. Reported as rc 3, not as pass.**
|
||||
- **This branch touches NO sensitive path** (`runbooks/**` and `decisions/**` are not in
|
||||
`infra/gitea/approval-policy.json`), so unlike #55 it needs no `/approve-override`.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The corrected command has not been RUN.** Ansible is an operator action, and this is the
|
||||
decisive gap: I have verified the playbook, the group, the tag and the flag all exist and are
|
||||
the documented convention, but not that the run succeeds end to end or that
|
||||
`lsmod | grep '^wireguard'` then reports the module. It is a better-evidenced command, not a
|
||||
proven one.
|
||||
- **Whether `--tags system` is entirely side-effect-free on a live prod node.** Each task it runs is
|
||||
idempotent by construction, but I did not dry-run it (`--check`) to confirm it reports zero
|
||||
changes beyond the module load.
|
||||
- **The rest of the runbook's commands past step 2** are unchanged and still unexercised — in
|
||||
particular `kubectl exec ds/wireguard -- wg show`, which cannot be reached while the pods sit in
|
||||
`Init` waiting on the server key.
|
||||
- **Whether any OTHER document I wrote in #55 contains an invented command.** I re-checked every
|
||||
file path and markdown link in the component (all resolve) and every script the docs tell an
|
||||
operator to run, but "the referenced file exists" is weaker than "the command is correct" — which
|
||||
is exactly the gap that produced `site.sh`.
|
||||
|
||||
## Post-merge state of #55, recorded here since its own entry predates the merge
|
||||
|
||||
Measured on live prod after #55 merged as `ad3f56e`:
|
||||
|
||||
- Flux applied it by the real path: Kustomization `vpn-ingress` `lastAppliedRevision:
|
||||
main@sha1:ad3f56e`, equal to the GitRepository's artifact revision, `Ready=True`.
|
||||
- All **18** Flux Kustomizations `Ready=True` — the new tier disturbed nothing, and `wait: false`
|
||||
behaved as intended (`vpn-ingress` is Ready while its own pods are not, so nothing downstream
|
||||
stalled on it).
|
||||
- **`hostNetwork` is confirmed in effect, not merely configured**: the three pods report pod IPs
|
||||
`192.168.2.10/.11/.12` — the node addresses, one per node.
|
||||
- **It fails closed exactly as designed**, with a legible cause: all three pods `Init:0/1`,
|
||||
`MountVolume.SetUp failed for volume "wg-server-key" : secret "wg-server-key" not found`, and no
|
||||
Secret in the namespace. The #55 entry's "the DaemonSet has never been reconciled" is superseded
|
||||
by this paragraph; its remaining unverified claims (the tunnel carrying a packet, the
|
||||
iptables/nftables backend question) still stand.
|
||||
+2
-1
@@ -26,7 +26,8 @@ The fix is the obvious one for concurrent appenders: **stop sharing a file**. Ea
|
||||
handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
|
||||
nothing for git to merge and nothing to conflict. This scales to any number of parallel
|
||||
branches with zero coordination — which is the point, since the agents are deliberately
|
||||
isolated from each other in separate worktrees and cannot coordinate.
|
||||
isolated from each other in separate worktrees (required by `CLAUDE.md` §0) and cannot
|
||||
coordinate.
|
||||
|
||||
There is deliberately **no index file** and no counter. An index would be a shared
|
||||
mutable file — exactly the thing that just failed — and a sequence number would have
|
||||
|
||||
@@ -34,10 +34,26 @@ deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module.
|
||||
`wg-quick` fails outright and the pod CrashLoops.
|
||||
|
||||
```bash
|
||||
# operator action (Ansible is not an agent action)
|
||||
cd infra/ansible && ./site.sh --tags system --limit k3s_prod
|
||||
# operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox,
|
||||
# and ansible.cfg is only picked up from infra/ansible, so cd there inside the container.
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
|
||||
```
|
||||
|
||||
Three details in that command are load-bearing, not style:
|
||||
|
||||
- **`-e k3s_hosts=k3s_prod`.** `node-baseline.yml` targets `k3s_servers` by default, which is the
|
||||
**dev** group — and dev was decommissioned, so an unqualified run does nothing useful. This is
|
||||
the targeting pattern `infra/ansible/README.md` "Targeting prod" documents for every playbook.
|
||||
- **`--tags system`, and never an untagged run.** The two kernel-module tasks carry the `system`
|
||||
tag. An untagged run would also execute the `disks` tasks, which assert on block-device variables
|
||||
prod has deliberately left as placeholders (`infra/PROD-PROVISIONING.md` §2) — the playbook's own
|
||||
header warns about this. `--tags system` is broader than the modules alone (it also re-asserts
|
||||
sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and
|
||||
already in its desired state on a live node.
|
||||
- **It is a playbook, not a wrapper script.** There is no `site.sh` in this repo; `playbooks/site.yml`
|
||||
is the full bring-up chain and is the wrong tool here.
|
||||
|
||||
Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:
|
||||
|
||||
```bash
|
||||
@@ -66,9 +82,16 @@ lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE
|
||||
--from-file=privatekey=/work/.cache/server.key'
|
||||
```
|
||||
|
||||
…where `/work/.cache/server.key` is a path you copied the key to **inside the repo's gitignored
|
||||
cache** (or mount it however you prefer — the requirement is only that the bytes never pass through
|
||||
a log). Delete that copy afterwards: `shred -u .cache/server.key`.
|
||||
…where `/work/.cache/server.key` is a path you copied the key to (`.cache/` is not tracked; create
|
||||
it if absent). Delete that copy as soon as the Secret exists: `shred -u .cache/server.key`.
|
||||
|
||||
> **KEEP THE `.key` SUFFIX.** That filename is what keeps the key out of git: `.gitignore` line 11
|
||||
> is `*.key`, and it is the **only** rule covering this path — `.cache/` itself is NOT ignored
|
||||
> (`git check-ignore -v .cache/serverkey.txt` matches nothing). So a copy named `server.pem`,
|
||||
> `wg-server` or `privatekey` sitting in that directory is **committable**, and the gate's
|
||||
> `no-plaintext-secrets` stage does not know a WireGuard private key when it sees one. Verify before
|
||||
> you trust it: `git check-ignore -v .cache/<your-filename>` must print a rule, and
|
||||
> `git status --short` must not list the file.
|
||||
|
||||
Confirm it exists **by key name only**:
|
||||
|
||||
|
||||
Reference in new issue
Block a user