Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5 18f741d744 fix(runbooks): vpn-remote-access step 1 named a nonexistent script; step 2 cited the wrong gitignore rule
gate / gate (pull_request) Successful in 1m47s
gate / conformance (pull_request) Has been skipped
Both reported by the operator on first use of the runbook merged in #55.

Step 1 said `cd infra/ansible && ./site.sh --tags system --limit k3s_prod`.
There is no site.sh anywhere in this repo. It was wrong four ways at once:
this repo drives Ansible with `ansible-playbook <playbook>` and never a
wrapper (playbooks/site.yml is the full bring-up chain, not this); the module
work is in playbooks/node-baseline.yml; targeting here is `-e k3s_hosts=...`,
not `--limit`; and Ansible exists only in the pinned toolbox, with ansible.cfg
read only from infra/ansible. Replaced with the real command plus why
`-e k3s_hosts=k3s_prod` matters (the default group is decommissioned dev) and
why `--tags system` must never be dropped (an untagged run also executes the
`disks` tasks, which assert on prod's deliberate placeholder devices).

Step 2 told the operator to stage the server key at /work/.cache/server.key
"inside the repo's gitignored cache". .cache/ is NOT gitignored -- the file was
ignored only incidentally by `*.key` (.gitignore:11). So the path was safe for
a reason I had not identified, and renaming it (server.pem, wg-server) would
have left a committable WireGuard private key in the checkout, which
no-plaintext-secrets does not recognise. The step now names the real rule,
marks the suffix as the load-bearing part, and gives a check-ignore
verification to run before trusting it.

Docs only -- no manifest and no behaviour change. Touches no sensitive path.
Also records the post-merge measurements for #55, whose own entry predates the
merge: Flux applied ad3f56e by the real path, all 18 Kustomizations Ready,
hostNetwork confirmed (pod IPs are the node IPs), and the designed fail-closed
state (Init:0/1 on the missing wg-server-key Secret).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-05 23:08:19 -05:00
jakman85 3ee663dbcc Merge pull request 'docs(agents): one workspace per agent run — the shared checkout is off-limits' (#56) from docs/agents-work-in-their-own-workspace into main
gate / gate (push) Successful in 1m46s
gate / conformance (push) Has been skipped
Reviewed-on: #56
Reviewed-by: Keith Jackson <[email protected]>
2026-10-06 03:49:28 +00:00
jakman85 ad3f56eee8 Merge pull request 'feat(vpn-ingress): WireGuard remote access on udp/51820 — hostNetwork, because a pod identity would 403 every Gateway hostname' (#55) from feat/vpn-remote-access-wireguard into main
gate / gate (push) Successful in 1m47s
gate / conformance (push) Has been skipped
Reviewed-on: #55
Reviewed-by: Keith Jackson <[email protected]>
2026-10-06 03:41:48 +00:00
jakman85andClaude Opus 5 2cb891c05d docs(agents): one workspace per agent run — the shared checkout is off-limits
gate / gate (pull_request) Successful in 1m48s
gate / conformance (pull_request) Has been skipped
Agents have been working in the operator's checkout and clobbering each other.
Nothing in the tree told them not to: worktrees appeared in CLAUDE.md only as
§3's gate workarounds, and decisions/README.md described worktree isolation as
something agents already do rather than as a rule.

New CLAUDE.md §0 states the rule, with the commands and an explicit do-not list
for the shared checkout. Placed before §1 so every existing §-reference in the
repo keeps its meaning.

Measured in the shared checkout on 2026-10-05, each row a distinct failure mode:
107 `checkout: moving` reflog entries; sitting on feat/pm-agent-issue-to-pr, not
main; two other runs' untracked trees left in it; local main two commits behind
origin/main; 42 registered worktrees of which 31 are on already-merged branches;
9 gate scripts carrying a `.claude/` exclusion because an in-tree worktree made
find-based discovery count the same manifest up to 18 times.

Corrects a claim carried in earlier briefs: `git worktree remove` is NOT blocked
by the agent sandbox. Provoked on a throwaway worktree — rc 0.

§3 reframed as the normal path and grown from two breakages to three:
spec-immutability's skip is avoidable, not merely tolerated, with the spec repo
mounted read-only in the same ENGINE shim plus -e SPEC_REPO=/home-network-k3s.

CLAUDE.md is a sensitive path, so this PR needs a non-author review or an
administrator /approve-override posted as a PR review. Not posted by me: §1
forbids an agent clearing that on its own PR.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-05 22:31:40 -05:00
6 changed files with 443 additions and 11 deletions

No files matched your search

+137 -5
View File
@@ -4,6 +4,125 @@ Read this before doing anything durable. It is the delivery contract for AI codi
sessions and agents. The newest entry in `decisions/` is the current PM handoff and
takes precedence on *what* to work on; this file governs *how* it lands.
**Start with §0.** It is first because it is the rule you can break before you have read
anything else: the very first `git checkout -b` in the wrong directory has already done the
damage.
## 0. Work in your OWN workspace. Never in the shared checkout.
Before you create a branch, edit a file, or run `git` anything: **make yourself a git
worktree outside the repository root, and do all of your work there.** One workspace per
agent run, no exceptions.
# from the shared checkout, ONCE, at the start of your run
git fetch origin main
git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
cd /path/outside/the/repo/wt-<slug> # everything after this happens HERE
`/path/outside/the/repo` means outside this repository's tree — your session scratchpad,
or a sibling like `/var/home/keith/source/wt-<slug>`. Branch from **`origin/main`**, never
from `main` and never from `HEAD`; see "stale base" below.
**The shared checkout is `/var/home/keith/source/home-network-k3s-impl`** — the operator's
working copy, the one you are probably started in. In it, do **not**:
- **`git checkout` / `git switch` / `git checkout -b`.** Its HEAD is shared state; moving
it moves the tree out from under whoever else is using it.
- **edit, create or delete a file.** Including "just one doc", including a file you intend
to move later.
- **`git add` / `git commit` / `git stash` / `git restore`.** `git add -A` there stages
whatever another agent left lying around; `git stash` *removes* their uncommitted work
into a place they will not look.
- **`git worktree remove` anything you did not create.**
Read-only commands there are fine, and the one write you do need is creating your own
worktree.
### Why — measured in the shared checkout, 2026-10-05
| What was measured | Value |
|---|---|
| `checkout: moving` entries in its reflog | **107** — the tree has been yanked between branches 107 times |
| the branch it was sitting on | `feat/pm-agent-issue-to-pr` — **not `main`** |
| uncommitted trees left in it by other runs | **2** (`infra/agent/`, `platform/pm-agent/`), mtimes five minutes apart the same evening |
| its local `main` vs `origin/main` | **two commits behind** (`4386d14` vs `22c94ef`) |
| registered worktrees | **42**, of which **31** are on branches already merged to `main` |
| gate scripts carrying an explicit `.claude/` exclusion | **9** |
Each row is a distinct way the shared checkout damages work:
- **Dirty tree, foreign files.** Two other runs' uncommitted directories were sitting in
it. An agent that edits there and runs `git add -A` ships someone else's half-finished
work inside its own PR, and the PR's author has no idea what those files are. This is not
hypothetical and it is not historical: **eleven minutes after those two directories were
measured, while this very section was being written, another run committed them** —
`5334abe feat(pm-agent): ...`, 1,000+ lines across `infra/agent/`, `clusters/`, `catalog/`
and `apps/` — onto the shared checkout's checked-out branch, moving its HEAD under anyone
reading it. Had this rule's author been working in that tree rather than in a worktree,
that commit and this one would have collided.
- **Stale base.** Its `main` was two commits behind `origin/main`, so
`git checkout -b x main` there branches off a tree that is already not what CI will merge
into — the gate you run locally proves a base nobody else has. Worse, it was not even *on*
`main`: a bare `git checkout -b x` would have branched off `feat/pm-agent-issue-to-pr` and
silently inherited that work into your PR. `git fetch` first, and name `origin/main`
explicitly as the start point.
- **Shared HEAD.** 107 branch switches in one tree is 107 chances to switch it while
another agent is mid-edit. The loser's changes are either carried onto the wrong branch
or lost to a checkout conflict — and the symptom surfaces in *their* session, not yours.
### Put the workspace OUTSIDE the repo root
A worktree *inside* the tree (`.claude/worktrees/agent-*/` — which is where Claude Code's
own default puts it) is a full second copy of the repo that every tree-walking check then
finds. Measured 2026-09-12: `find`-based discovery reported **the same manifest under test
once per worktree, up to 18 times**, and the L0 check comparing that listing against the
tree failed on paths that are not authored surface at all
(`platform/monitoring/tests/run.sh:115-123`). Nine gate scripts now carry an explicit
`.claude/` prune for this reason — which means the hazard is *worked around*, not fixed,
and the next check that walks the tree will hit it again. Keep the copy out of the tree and
none of that applies.
### Clean up — and expect not to be able to
31 of the 42 registered worktrees are on branches already merged to `main`. A registered
worktree **pins its branch**: while it exists, that branch cannot be deleted or checked out
anywhere else. So a left-behind workspace is a lock on a branch name, and the cost lands on
whoever next needs it.
Remove yours when you are done:
git worktree remove /path/outside/the/repo/wt-<slug>
Measured 2026-10-05: on a worktree **you created yourself, in the session scratchpad**, that
command **succeeds** — rc 0, no prompt, with or without a branch checked out. The "worktree
removal is sandbox-blocked" note in circulation is about the `.claude/worktrees/agent-*` ones
belonging to *other* sessions, some of which are additionally `locked` on git's own terms. So:
remove yours, and only yours. If removal is refused, say so in your handoff and leave the
worktree registered — do **not** `rm -rf` the directory, which leaves a registered worktree
pointing at nothing and still holding the branch.
If someone *else's* stale worktree is holding the branch you need, do not try to reclaim the
branch name. Push a `rework/<pr-number>` branch to the PR's refspec instead.
### If you are already standing in the shared checkout with edits
Stop. Do not `git stash` (that moves other agents' work too). Create your worktree as
above, redo the edits there, and leave the shared checkout's HEAD and working tree exactly
as you found them.
### Gating from a worktree
Three known breakages, all with workarounds, all required every run: see §3, "Running the
gate from a worktree".
### If you are writing a brief for another agent
The brief must name **its own explicit branch** and tell it to create **its own worktree**
from `origin/main`. "Work on the current branch" and "continue in the checkout" are how
branch sprawl and cross-contamination got here: an agent given no workspace uses the one it
is started in, which is the shared one. Also carry §3's three gate workarounds into the
brief — an agent that cannot gate will report rc 3 as green.
## 1. Never push to `main`. Ever.
Every durable change reaches `main` exactly one way:
@@ -147,15 +266,21 @@ Exit codes are a contract — do not flatten them:
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
beside this one; that single skip is expected and tolerated. Any *other* skip is a
real gap in your run.
beside this one; that single skip is expected and tolerated — and from a worktree it is
avoidable outright (§3's worktree list, item 3). `gitea-settings` is the other expected
skip: it needs `SETTINGS_API_TOKEN`, which exists only in CI, so **a local run cannot
return 0** — the honest best local result is rc 3 with `gitea-settings` as the only stage
that did not run. Any *further* skip is a real gap in your run. Report the rc and name the
skipped stages; never round rc 3 up.
**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
### Running the gate from a git worktree
### Running the gate from a worktree
Two unrelated failures, both with known workarounds — put both in any agent brief:
This is the **normal** path, not an edge case — §0 requires you to be in a worktree, so
every gate run you do hits all of these. Three known breakages, all with
workarounds, all of them yours to apply every run — put them in any agent brief:
1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
and exists only in the main checkout. Do **not** copy it (it is a credential).
@@ -166,10 +291,17 @@ Two unrelated failures, both with known workarounds — put both in any agent br
the `ENGINE` hook — a shim that intercepts `run` and re-execs
`podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
`tracked_files()` is this bug, not a broken tree.
3. **`spec-immutability` skips** — the skip called tolerated above is also *avoidable*. `dev.sh`
derives the sibling spec path from its own `REPO_ROOT`, which in a worktree is your
workspace, so `../home-network-k3s` does not resolve and it adds no mount. Mount the spec
repo read-only in the same `ENGINE` shim and pass `-e SPEC_REPO=/home-network-k3s`: with
both mounts the stage reports `pass`, not SKIPPED, and R-TEST-06 is actually covered
(measured — `decisions/2026-10-01-render-selftest-runs-in-gate.md`).
## 4. Branch naming
Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
Use `feat/`, `fix/`, `docs/`, `chore/`. **One branch, one workspace, one PR** — if you
need a second branch, make a second worktree (§0); do not re-point an existing one. Note that `.gitea/workflows/gate.yml`'s **push**
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
+17
View File
@@ -11,6 +11,9 @@ specification** at [`../home-network-k3s`](../home-network-k3s).
- **Done = a green, reproducible, requirement-tagged conformance test** — never "merged," never a
manual demo (spec `docs/14`, `agents/definition-of-done.md`).
- **No work on `main`.** Every change is a feature branch merged into `main` (spec `CONTRIBUTING.md`).
- **No work in a shared checkout.** Every agent run gets its own git worktree, outside the repo
root, branched from `origin/main`. This checkout is shared mutable state — see
[`CLAUDE.md` §0](CLAUDE.md).
- **No plaintext secrets** — SOPS-encrypted only (spec `docs/07`).
- **Phase discipline** — only build what the active roadmap phase authorizes (spec `roadmap/phases.md`).
@@ -95,6 +98,20 @@ The running narrative — decisions, root causes, incidents, follow-ups — is
## Working on this repo
**First, get out of this checkout.** It is the operator's working copy and several agents
share it; concurrent runs here have overwritten each other's branches and left each other's
uncommitted files behind. One workspace per change:
```bash
git fetch origin main
git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
cd /path/outside/the/repo/wt-<slug>
```
Outside the repo root specifically — a worktree under `.claude/worktrees/` is a second full
copy that tree-walking gate checks then count twice. The rule, the measurements behind it, and
the three gate workarounds a worktree needs are in [`CLAUDE.md` §0](CLAUDE.md) and §3.
Everything the gate needs is baked into the pinned dev container — **do not install the
toolchain on the host**:
@@ -0,0 +1,147 @@
---
date: 2026-10-05
branch: docs/agents-work-in-their-own-workspace
title: One workspace per agent run — the shared checkout is now documented as off-limits
---
## What changed
Agents have been doing their work in the operator's checkout at
`/var/home/keith/source/home-network-k3s-impl` and clobbering each other. Nothing in the tree
told them not to: worktrees appeared in CLAUDE.md only as §3's *gate workarounds*, i.e. as an
awkward edge case, and `decisions/README.md` asserted worktree isolation as a fact about how
agents already behave rather than as a rule they must follow. Nowhere was the rule stated.
Documentation only — no code, no manifests, no pipeline logic.
- **`CLAUDE.md`: new `## 0. Work in your OWN workspace. Never in the shared checkout.`**
Placed before §1 so the section numbers every other file and handoff entry cites (§1, §2b,
§3, §7) all keep their meaning. It gives the three commands (`git fetch origin main`;
`git worktree add -b <type>/<slug> <path outside the repo> origin/main`; `cd`), an explicit
do-not list for the shared checkout (no `checkout`/`switch`, no edits, no
`add`/`commit`/`stash`/`restore`, no removing others' worktrees), a measurement table, the
"put it outside the repo root" rule, cleanup, what to do if you are already standing in the
shared checkout with edits, and what a PM must put in a brief it writes for another agent.
- **`CLAUDE.md` intro** now says "Start with §0", with the reason: it is the rule you can
break before you have read anything else.
- **`CLAUDE.md` §3** retitled "Running the gate from a worktree" and reframed as the normal
path rather than an edge case, since §0 now makes every gate run a worktree run. Its list
grew from two breakages to three: `spec-immutability`'s skip is *avoidable*, not merely
tolerated, if the `ENGINE` shim also mounts the sibling spec repo read-only and passes
`-e SPEC_REPO=/home-network-k3s`.
- **`CLAUDE.md` §4** gained "One branch, one workspace, one PR".
- **`README.md`**: a fourth non-negotiable rule ("No work in a shared checkout"), and
"Working on this repo" now opens with how to get out of the checkout before describing the
dev container that runs in it.
- **`decisions/README.md`**: the worktree-isolation sentence now cites §0 as the requirement
instead of describing it as something agents happen to do.
I did this work the way the new §0 requires — a worktree at
`<session scratchpad>/wt-workspace-rule`, outside the repo root, branched from `origin/main`,
with the shared checkout left on the branch and with the untracked files I found in it.
## What I verified
Every number in the new §0 table was measured in the shared checkout on 2026-10-05, not
recalled:
| Claim | Command | Value |
|---|---|---|
| shared HEAD is yanked between branches | `git reflog \| grep -c 'checkout: moving'` | **107** |
| it is not sitting on `main` | `git worktree list` | `feat/pm-agent-issue-to-pr` |
| other runs' uncommitted work is in it | `git status --porcelain` | **2** trees: `infra/agent/` (22:04), `platform/pm-agent/` (22:09) — `stat -c %y` |
| its `main` is stale | `git rev-parse main` vs `origin/main` | `4386d14` vs `22c94ef`, **2 behind** |
| worktrees registered / already merged | `git worktree list --porcelain` + `git merge-base --is-ancestor <b> main` | **42** branches, **31** merged |
| nested worktrees are a gate hazard | `grep -rl '\.claude' conformance/tests platform/monitoring/tests` | **9** scripts carry an explicit exclusion |
| the 18× double-count | read `platform/monitoring/tests/run.sh:115-123` | in-tree comment, observed 2026-09-12 |
**The failure mode was observed live, mid-session, not just reconstructed.** The two untracked
trees I measured at 22:04 and 22:09 were committed at 22:20 by another run — `5334abe`
`feat(pm-agent): an in-cluster AI Program Manager...`, 1,000+ lines across `infra/agent/`,
`clusters/`, `catalog/`, `apps/`, `docs/plans/` and its own `decisions/` entry — onto the shared
checkout's checked-out branch. Measured after the fact: the shared checkout's HEAD moved
`22c94ef` → `5334abe` and its working tree went clean between my first `git status` and my last,
with `git log -1` naming a commit that is not mine. That is §0's "shared HEAD" row happening
during the session that wrote §0, and it is now cited in the doc.
**A claim of mine was wrong and is corrected in the doc.** I had it recorded that
`git worktree remove` is blocked by the agent sandbox. Provoked rather than assumed: created
a throwaway detached worktree in the scratchpad and removed it — **rc 0**, and
`git worktree list` no longer shows it. §0 therefore tells agents to remove their own, and
scopes the "blocked" note to what it actually covers: *other* sessions'
`.claude/worktrees/agent-*`, some of which are additionally `locked` on git's own terms. The
`rework/<pr-number>` escape hatch is now scoped to someone else's stale worktree holding a
branch, which is a real and separate problem (31 of them). Both of my own worktrees in this
session were removed cleanly at rc 0 after pushing.
`spec-immutability` is avoidable-not-tolerated: taken from
`decisions/2026-10-01-render-selftest-runs-in-gate.md`, which records
`stage spec-immutability: pass` under the two-mount shim plus `-e SPEC_REPO=/home-network-k3s`.
**Gate — `rc 3`, UNDETERMINED, and I am not calling it green.** Pinned toolbox, from this
worktree, `DEV_KUBECONFIG=/dev/null` and an `ENGINE` shim bind-mounting both the main
checkout's `.git` (at its own host path) and the sibling spec repo read-only with
`-e SPEC_REPO=/home-network-k3s`:
ENGINE=<two-mount shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' # rc 3
| stage | result |
|---|---|
| spec-immutability | **pass** — it genuinely RAN; the second mount is what makes it run from a worktree |
| approval-policy | pass, but **N/A**: `event='none'`. It cannot see a PR locally, so this green says nothing. See below |
| decisions-log | pass — the three required headings and the frozen-archive pin |
| lint · l0 · render-golden · l1 | pass (L1: 3 offline checks, 0 failures, 0 skips, 0 unwired) |
| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset; it exists only in CI |
| conformance (L2/L3) | SKIPPED — `CLUSTER_AVAILABLE!=true`; the decision notes no cluster-needing requirement was touched |
`gitea-settings` is the *only* stage that did not run, and it cannot be made to run from a
workstation, so rc 3 is the ceiling for any local run — not a defect in this branch. The
first gate attempt returned rc 3 for a different and real reason (`HEAD is not ahead of
origin/main`: I had not committed yet, so the touched-requirement set was empty by default
rather than by verification); it was re-run after committing, and the table above is the
post-commit run. §3 now records both expected skips, because it previously named only
`spec-immutability`.
## What I did NOT verify
- **GATE: UNVERIFIED at the time of writing.** The pinned-toolbox result for this branch and
its exit code are reported in the PR, not predicted here. **A local rc 3 is not green** and
will not be described as one.
- **CI gate: RED, measured — `MERGE GATE: BLOCKED`, one failing stage.** Run 245 on PR #56,
head `b988da8b`, 1m47s. The only failure is `approval-policy`: *"a sensitive change lacks a
non-author approval"*, `sensitive paths touched: CLAUDE.md`, author `gitea-admin`. Every other
stage passed — `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, and `gitea-settings`
(which can only run in CI, and reported `required_approvals = 0`,
`dismiss_stale_approvals = true`). `spec-immutability` SKIPPED in CI (no sibling spec repo in
the CI workspace — the inverse of the local run, where the two-mount shim made it pass) and
`conformance` SKIPPED. **Note `gate / conformance (pull_request)` reports status `success`
with description "Has been skipped"** — CLAUDE.md §2b's trap, live on this very PR. Nothing
in this branch is broken; the red is the structural human gate, below. *Measured at
`b988da8b`; this entry was then amended to record it, so the head SHA in the PR is newer than
the run quoted here and CI re-ran on the amended commit.*
- **This PR is a sensitive-path PR and is CI-RED on `approval-policy` until a human acts.**
`CLAUDE.md` is in `infra/gitea/approval-policy.json`'s `sensitive_paths` ("the delivery
contract agents follow"), so the stage requires an approving review from someone who is not
the author, or an `/approve-override <reason>` posted by an administrator **as a PR review
of type "Comment"**, not in the plain comment box. `ci/approval-policy.sh` run locally
reports N/A (it cannot see a PR), so the local gate cannot catch this and a green local run
says nothing about it. **I have not posted an override and must not** — CLAUDE.md §1 forbids
an agent clearing this on its own PR even though `gitea-admin` holds `owner` and could.
- **Nothing here is enforced.** §0 is prose. There is no gate stage that detects work done in
the shared checkout, no check that a worktree is outside the repo root, and no check that a
branch was cut from `origin/main`. Every other rule in CLAUDE.md that agents kept breaking
(`DECISIONS-LOG.md` appends, the three handoff headings) ended up as a gate stage, and that
is the honest follow-up here. I did not write one: the ask was documentation, and an
enforcement stage lands in `ci/**`, which is itself a sensitive path and deserves its own PR.
- **I did not clean up the 31 stale merged worktrees or the 35 nested under
`.claude/worktrees/`.** They are other sessions' workspaces; removing them is an operator
action, and §0 explicitly tells agents not to remove worktrees they did not create.
- **I did not touch the untracked `infra/agent/` and `platform/pm-agent/` trees** sitting in the
shared checkout. They are another run's in-flight work. Whose, and whether they are wanted,
is unknown to me.
- **`docs/plans/ssd-etcd-execution-runbook.md:18` still points at a `.claude/worktrees/` path**
inside the repo. Left as-is deliberately: that runbook is marked COMPLETED/historical and
describes the now-decommissioned dev cluster, so the path is a record of what was done, not
an instruction. It does mean a grep for in-repo worktree paths is not yet clean.
- **No runtime, cluster or Flux behaviour was touched or checked.** This branch changes three
Markdown files; `kubectl` was not run.
@@ -0,0 +1,112 @@
---
date: 2026-10-05
branch: fix/vpn-runbook-ansible-command
title: vpn-remote-access runbook — step 1 named a script that does not exist, and step 2 cited the wrong reason the key is gitignored
---
## What changed
Two defects in `runbooks/vpn-remote-access.md` (merged in #55), both authored by me, both reported
by the operator the first time they tried to follow it.
**1. Step 1 named a command that does not exist.** It said:
cd infra/ansible && ./site.sh --tags system --limit k3s_prod
There is **no `site.sh` anywhere in this repo** (`find . -name site.sh` → nothing). The invocation
was wrong in four separate ways at once, which is what makes it worth recording rather than just
patching: wrong kind of thing (this repo drives Ansible with `ansible-playbook <playbook>`, never a
wrapper script — `playbooks/site.yml` exists and is the full bring-up chain, not this); wrong
playbook (the module work lives in `playbooks/node-baseline.yml`); wrong targeting flag (`--limit`
instead of this repo's `-e k3s_hosts=...` convention, which `infra/ansible/README.md` "Targeting
prod" documents for every playbook); and no toolbox wrapper, though Ansible exists only inside the
pinned container and `ansible.cfg` is only read from `infra/ansible`.
Replaced with the real command, plus the three things about it that are load-bearing:
./infra/devcontainer/dev.sh run bash -c \
'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
- `-e k3s_hosts=k3s_prod` — the playbook defaults to `k3s_servers`, which is the **dev** group;
dev is decommissioned, so an unqualified run does nothing useful.
- `--tags system`, never untagged — the two kernel-module tasks carry that tag, and an untagged run
would also hit the `disks` tasks, which assert on block-device variables prod deliberately leaves
as placeholders (the playbook's own header warns about this). `--tags system` is broader than the
modules alone (sysctls, iscsid, qemu-guest-agent, swap masking) but each is idempotent and
already in its desired state on a live node.
**2. Step 2 gave the wrong reason the server key is safe from git — and the right reason is
fragile.** It told the operator to stage the key at `/work/.cache/server.key` "inside the repo's
gitignored cache". `.cache/` is **not** gitignored: `git check-ignore -v .cache/serverkey.txt`
matches nothing. The file was ignored only incidentally, by `.gitignore` line 11 `*.key` — so the
path happened to be safe for a reason I had not identified, and **any operator who renamed it
(`server.pem`, `wg-server`, `privatekey`) would have had a committable WireGuard private key sitting
in the checkout.** The gate would not have caught it either; `no-plaintext-secrets` does not
recognise a bare WireGuard key.
The step now states the real rule, marks the `.key` suffix as the thing doing the work, and gives
the operator a verification they can run before trusting it (`git check-ignore -v` must print a
rule, and `git status --short` must not list the file).
Nothing outside `runbooks/vpn-remote-access.md` changed; no manifest, no behaviour.
## What I verified
- **`site.sh` does not exist**: `find . -name 'site.sh' -not -path './.git/*'` → no output.
- **The real surface**: `infra/ansible/playbooks/` contains `node-baseline.yml` and `site.yml` (and
11 others); `node-baseline.yml`'s own header documents both
`-e k3s_hosts=k3s_prod` and the tag-scoping hazard. The `-e <playbook>_hosts=` convention is
confirmed by `infra/ansible/README.md` §"Targeting prod" and by the existing
`runbooks/node-memory-swap.md`, which invokes `ansible-playbook playbooks/node-health.yml -e
node_health_hosts=...`.
- **`k3s_prod` is a real inventory group**: `[k3s_prod:children]` in `inventory/hosts.ini`, with
`[k3s_prod_primary]`/`[k3s_prod_secondary]`.
- **`--tags system` covers the kernel-module tasks**: both "Persist kernel modules…" and "Load
kernel modules now" carry `tags: [system]` in `roles/node_baseline/tasks/main.yml`.
- **The gitignore claim, measured both directions**:
`git check-ignore -v .cache/server.key` → `.gitignore:11:*.key`;
`git check-ignore -v .cache/serverkey.txt` → no match. So the suffix is the only thing ignoring it.
- **Every path the component's docs reference resolves**: all six relative markdown links in
`platform/vpn-ingress/README.md` (checked from that file's own directory), and both Ansible
playbook paths.
- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
`lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; the rc 3 is the two
environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no `CLUSTER_AVAILABLE`).
**Not green. Reported as rc 3, not as pass.**
- **This branch touches NO sensitive path** (`runbooks/**` and `decisions/**` are not in
`infra/gitea/approval-policy.json`), so unlike #55 it needs no `/approve-override`.
## What I did NOT verify
- **The corrected command has not been RUN.** Ansible is an operator action, and this is the
decisive gap: I have verified the playbook, the group, the tag and the flag all exist and are
the documented convention, but not that the run succeeds end to end or that
`lsmod | grep '^wireguard'` then reports the module. It is a better-evidenced command, not a
proven one.
- **Whether `--tags system` is entirely side-effect-free on a live prod node.** Each task it runs is
idempotent by construction, but I did not dry-run it (`--check`) to confirm it reports zero
changes beyond the module load.
- **The rest of the runbook's commands past step 2** are unchanged and still unexercised — in
particular `kubectl exec ds/wireguard -- wg show`, which cannot be reached while the pods sit in
`Init` waiting on the server key.
- **Whether any OTHER document I wrote in #55 contains an invented command.** I re-checked every
file path and markdown link in the component (all resolve) and every script the docs tell an
operator to run, but "the referenced file exists" is weaker than "the command is correct" — which
is exactly the gap that produced `site.sh`.
## Post-merge state of #55, recorded here since its own entry predates the merge
Measured on live prod after #55 merged as `ad3f56e`:
- Flux applied it by the real path: Kustomization `vpn-ingress` `lastAppliedRevision:
main@sha1:ad3f56e`, equal to the GitRepository's artifact revision, `Ready=True`.
- All **18** Flux Kustomizations `Ready=True` — the new tier disturbed nothing, and `wait: false`
behaved as intended (`vpn-ingress` is Ready while its own pods are not, so nothing downstream
stalled on it).
- **`hostNetwork` is confirmed in effect, not merely configured**: the three pods report pod IPs
`192.168.2.10/.11/.12` — the node addresses, one per node.
- **It fails closed exactly as designed**, with a legible cause: all three pods `Init:0/1`,
`MountVolume.SetUp failed for volume "wg-server-key" : secret "wg-server-key" not found`, and no
Secret in the namespace. The #55 entry's "the DaemonSet has never been reconciled" is superseded
by this paragraph; its remaining unverified claims (the tunnel carrying a packet, the
iptables/nftables backend question) still stand.
+2 -1
View File
@@ -26,7 +26,8 @@ The fix is the obvious one for concurrent appenders: **stop sharing a file**. Ea
handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
nothing for git to merge and nothing to conflict. This scales to any number of parallel
branches with zero coordination — which is the point, since the agents are deliberately
isolated from each other in separate worktrees and cannot coordinate.
isolated from each other in separate worktrees (required by `CLAUDE.md` §0) and cannot
coordinate.
There is deliberately **no index file** and no counter. An index would be a shared
mutable file — exactly the thing that just failed — and a sequence number would have
+28 -5
View File
@@ -34,10 +34,26 @@ deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module.
`wg-quick` fails outright and the pod CrashLoops.
```bash
# operator action (Ansible is not an agent action)
cd infra/ansible && ./site.sh --tags system --limit k3s_prod
# operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox,
# and ansible.cfg is only picked up from infra/ansible, so cd there inside the container.
./infra/devcontainer/dev.sh run bash -c \
'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
```
Three details in that command are load-bearing, not style:
- **`-e k3s_hosts=k3s_prod`.** `node-baseline.yml` targets `k3s_servers` by default, which is the
**dev** group — and dev was decommissioned, so an unqualified run does nothing useful. This is
the targeting pattern `infra/ansible/README.md` "Targeting prod" documents for every playbook.
- **`--tags system`, and never an untagged run.** The two kernel-module tasks carry the `system`
tag. An untagged run would also execute the `disks` tasks, which assert on block-device variables
prod has deliberately left as placeholders (`infra/PROD-PROVISIONING.md` §2) — the playbook's own
header warns about this. `--tags system` is broader than the modules alone (it also re-asserts
sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and
already in its desired state on a live node.
- **It is a playbook, not a wrapper script.** There is no `site.sh` in this repo; `playbooks/site.yml`
is the full bring-up chain and is the wrong tool here.
Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:
```bash
@@ -66,9 +82,16 @@ lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE
--from-file=privatekey=/work/.cache/server.key'
```
…where `/work/.cache/server.key` is a path you copied the key to **inside the repo's gitignored
cache** (or mount it however you prefer — the requirement is only that the bytes never pass through
a log). Delete that copy afterwards: `shred -u .cache/server.key`.
…where `/work/.cache/server.key` is a path you copied the key to (`.cache/` is not tracked; create
it if absent). Delete that copy as soon as the Secret exists: `shred -u .cache/server.key`.
> **KEEP THE `.key` SUFFIX.** That filename is what keeps the key out of git: `.gitignore` line 11
> is `*.key`, and it is the **only** rule covering this path — `.cache/` itself is NOT ignored
> (`git check-ignore -v .cache/serverkey.txt` matches nothing). So a copy named `server.pem`,
> `wg-server` or `privatekey` sitting in that directory is **committable**, and the gate's
> `no-plaintext-secrets` stage does not know a WireGuard private key when it sees one. Verify before
> you trust it: `git check-ignore -v .cache/<your-filename>` must print a rule, and
> `git status --short` must not list the file.
Confirm it exists **by key name only**: