home-network-k3s-impl/CLAUDE.md

# CLAUDE.md — how work lands in this repo

Read this before doing anything durable. It is the delivery contract for AI coding
sessions and agents. The newest entry in `decisions/` is the current PM handoff and
takes precedence on *what* to work on; this file governs *how* it lands.

**Start with §0.** It is first because it is the rule you can break before you have read
anything else: the very first `git checkout -b` in the wrong directory has already done the
damage.

## 0. Work in your OWN workspace. Never in the shared checkout.

Before you create a branch, edit a file, or run `git` anything: **make yourself a git
worktree outside the repository root, and do all of your work there.** One workspace per
agent run, no exceptions.

    # from the shared checkout, ONCE, at the start of your run
    git fetch origin main
    git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
    cd /path/outside/the/repo/wt-<slug>        # everything after this happens HERE

`/path/outside/the/repo` means outside this repository's tree — your session scratchpad,
or a sibling like `/var/home/keith/source/wt-<slug>`. Branch from **`origin/main`**, never
from `main` and never from `HEAD`; see "stale base" below.

**The shared checkout is `/var/home/keith/source/home-network-k3s-impl`** — the operator's
working copy, the one you are probably started in. In it, do **not**:

- **`git checkout` / `git switch` / `git checkout -b`.** Its HEAD is shared state; moving
  it moves the tree out from under whoever else is using it.
- **edit, create or delete a file.** Including "just one doc", including a file you intend
  to move later.
- **`git add` / `git commit` / `git stash` / `git restore`.** `git add -A` there stages
  whatever another agent left lying around; `git stash` *removes* their uncommitted work
  into a place they will not look.
- **`git worktree remove` anything you did not create.**

Read-only commands there are fine, and the one write you do need is creating your own
worktree.

### Why — measured in the shared checkout, 2026-10-05

| What was measured | Value |
|---|---|
| `checkout: moving` entries in its reflog | **107** — the tree has been yanked between branches 107 times |
| the branch it was sitting on | `feat/pm-agent-issue-to-pr` — **not `main`** |
| uncommitted trees left in it by other runs | **2** (`infra/agent/`, `platform/pm-agent/`), mtimes five minutes apart the same evening |
| its local `main` vs `origin/main` | **two commits behind** (`4386d14` vs `22c94ef`) |
| registered worktrees | **42**, of which **31** are on branches already merged to `main` |
| gate scripts carrying an explicit `.claude/` exclusion | **9** |

Each row is a distinct way the shared checkout damages work:

- **Dirty tree, foreign files.** Two other runs' uncommitted directories were sitting in
  it. An agent that edits there and runs `git add -A` ships someone else's half-finished
  work inside its own PR, and the PR's author has no idea what those files are.
- **Stale base.** Its `main` was two commits behind `origin/main`, so
  `git checkout -b x main` there branches off a tree that is already not what CI will merge
  into — the gate you run locally proves a base nobody else has. Worse, it was not even *on*
  `main`: a bare `git checkout -b x` would have branched off `feat/pm-agent-issue-to-pr` and
  silently inherited that work into your PR. `git fetch` first, and name `origin/main`
  explicitly as the start point.
- **Shared HEAD.** 107 branch switches in one tree is 107 chances to switch it while
  another agent is mid-edit. The loser's changes are either carried onto the wrong branch
  or lost to a checkout conflict — and the symptom surfaces in *their* session, not yours.

### Put the workspace OUTSIDE the repo root

A worktree *inside* the tree (`.claude/worktrees/agent-*/` — which is where Claude Code's
own default puts it) is a full second copy of the repo that every tree-walking check then
finds. Measured 2026-09-12: `find`-based discovery reported **the same manifest under test
once per worktree, up to 18 times**, and the L0 check comparing that listing against the
tree failed on paths that are not authored surface at all
(`platform/monitoring/tests/run.sh:115-123`). Nine gate scripts now carry an explicit
`.claude/` prune for this reason — which means the hazard is *worked around*, not fixed,
and the next check that walks the tree will hit it again. Keep the copy out of the tree and
none of that applies.

### Clean up — and expect not to be able to

31 of the 42 registered worktrees are on branches already merged to `main`. A registered
worktree **pins its branch**: while it exists, that branch cannot be deleted or checked out
anywhere else. So a left-behind workspace is a lock on a branch name, and the cost lands on
whoever next needs it.

Remove yours when you are done:

    git worktree remove /path/outside/the/repo/wt-<slug>

Measured 2026-10-05: that command **succeeds** under the agent sandbox (rc 0, on a throwaway
worktree in the session scratchpad) — earlier sessions recorded it as blocked, and that
record was wrong, or at least no longer true. Try it; it probably works. If it is refused,
say so in your handoff and leave the worktree registered — do **not** `rm -rf` the directory,
which leaves a registered worktree pointing at nothing and still holding the branch.

If someone *else's* stale worktree is holding the branch you need, do not try to reclaim the
branch name. Push a `rework/<pr-number>` branch to the PR's refspec instead.

### If you are already standing in the shared checkout with edits

Stop. Do not `git stash` (that moves other agents' work too). Create your worktree as
above, redo the edits there, and leave the shared checkout's HEAD and working tree exactly
as you found them.

### Gating from a worktree

Three known breakages, all with workarounds, all required every run: see §3, "Running the
gate from a worktree".

### If you are writing a brief for another agent

The brief must name **its own explicit branch** and tell it to create **its own worktree**
from `origin/main`. "Work on the current branch" and "continue in the checkout" are how
branch sprawl and cross-contamination got here: an agent given no workspace uses the one it
is started in, which is the shared one. Also carry §3's three gate workarounds into the
brief — an agent that cannot gate will report rc 3 as green.

## 1. Never push to `main`. Ever.

Every durable change reaches `main` exactly one way:

    branch  ->  push the branch  ->  open a PR  ->  CI gate green  ->  human approval  ->  merge

There is no other path. Specifically:

- **Do not `git push origin main`.**
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
- **Do not merge your own PR.** See the approval model below — and either way, the merge is
  someone else's action, never yours.

### Who approves

**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
branch requires no human approval for one. That is a deliberate transfer of authority, on the
operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
add a human.

**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
approving review from someone who is not its author. The reason is narrow and structural — **a
change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
the weakened check. Everything else merges on green.

The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
sensitive path, so the policy cannot be widened or narrowed without a human.

That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
missing credential is a lock-out, here a skip would be a bypass.

**The administrator override (2026-10-01).** The rule above was unsatisfiable for the one human
who runs this repo: Gitea refuses to let an author approve their own PR, and `jakman85` is the
only collaborator — so a sensitive-path PR the operator opened could never be merged, *including
the PR that would fix that.* An administrator may therefore waive the review by posting this on
the PR:

    /approve-override <reason>

**Post it as a PR *review* of type "Comment", not in the plain comment box.** CI's
`SETTINGS_API_TOKEN` is minted `[read:repository, read:admin]`, and `/issues/{n}/comments` sits
behind Gitea's *issue* scope — so an override typed into the comment box is **invisible to CI**,
and the stage reports it as a missing approval rather than as a permission problem (measured on
PR #38, 2026-10-02). `/pulls/{n}/reviews` is readable, so a review body works today. Both sources
are accepted and the gate names which it used; the comment box starts working once
`infra/gitea/set-ci-token.sh` is re-run to pick up `read:issue`.

The directive must **start a line**, the **reason is mandatory**, the poster's `admin`/`owner`
permission is re-checked **live** when the gate runs, and it must be **newer than the
head commit** — push again and you must override again. The overrider, the timestamp and the
reason are printed in the gate log and the job summary. For an administrator, "a change to the
gate cannot be gated by the gate" is therefore **advisory, not structural**; what is kept is an
attributed record of each waiver. Note that the agent account `gitea-admin` also holds `owner`
here, so an agent *can* clear this on its own PR — **do not**: posting that directive to unblock
your own agent work is exactly the self-approval the stage exists to prevent. Get the operator
to post it, or move the change out of the sensitive paths.

**`required_approvals` on `main` is `0`**, not 1 — the gate is the approver. That drops the
approval *count* only: `block_admin_merge_override` and `enable_status_check` both remain `true`,
so **a red gate is unmergeable by everyone, administrators included.**
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
  anything applied by hand is drift that the next reconcile reverts or, worse, adopts.

> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
> That wording is **stale** — it predates the PR flow. It describes what Flux does
> (main is what the cluster follows), not what you are permitted to do. Honour this
> file, not that line. Fixing those runbooks is tracked work.

## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)

Verified against the live Gitea on 2026-09-29:

| Expected | Actual |
|---|---|
| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
| Gate cannot be overridden, by anyone | ✅ `block_admin_merge_override: true` |
| Human approval required before merge | ⬜ **No — `required_approvals: 0` since 2026-10-01.** The gate is the approver; sensitive paths are gated by `ci/approval-policy.sh` instead. See §1 |

The first three were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
true. They are true now because each was measured, and the push rejection was provoked rather
than assumed. **Re-measure before relying on any of them** — see §2b.

The fourth row deliberately changed on 2026-10-01. `required_approvals: 1` was a **lock-out**, not
a protection: with a single collaborator and Gitea refusing author self-approval, nothing could
satisfy it (measured on PR #35 — green gate, zero reviews, unmergeable). It also contradicted the
model already written into `ci/approval-policy.sh` and `ci/gate.sh`. Rationale and the live-change
script: `infra/gitea/allow-admin-self-merge.sh`.

Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.

**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
strength of a check you have not seen report — and note that a **skipped** Gitea job
reports as `success`, so a green tick is not by itself evidence that anything ran.

## 2b. Never infer platform state from this repository

**This repo is not evidence about the live platform.** Neither is the handoff log
(`decisions/`, and the frozen `DECISIONS-LOG.md` archive). Both describe what someone
believed at the time of writing, and both have been confidently wrong for a month at a
stretch.

On 2026-09-29 the log stated the merge gate was a required status check with a runner
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
was wrong, and nothing in the tree could have revealed it, because the tree was the source
of the error.

**So, before asserting that any gate, protection, runner, secret, or workload is in a
given state: query the thing itself.**

| Claim about... | Check |
|---|---|
| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
| a runner exists | `GET /api/v1/admin/actions/runners` |
| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |

Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
"unverified" is a legitimate report, and a far better one than a confident guess that
matches the docs.

**A green tick is not proof anything ran.** Read what actually executed.

## 3. Gate before you push

    ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'

Exit codes are a contract — do not flatten them:

| rc | Meaning |
|---|---|
| 0 | PASS |
| 1 | BLOCKED — something failed |
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |

`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
beside this one; that single skip is expected and tolerated — and from a worktree it is
avoidable outright (§3's worktree list, item 3). `gitea-settings` is the other expected
skip: it needs `SETTINGS_API_TOKEN`, which exists only in CI, so **a local run cannot
return 0** — the honest best local result is rc 3 with `gitea-settings` as the only stage
that did not run. Any *further* skip is a real gap in your run. Report the rc and name the
skipped stages; never round rc 3 up.

**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.

### Running the gate from a worktree

This is the **normal** path, not an edge case — §0 requires you to be in a worktree, so
every gate run you do hits all of these. Three known breakages, all with
workarounds, all of them yours to apply every run — put them in any agent brief:

1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
   and exists only in the main checkout. Do **not** copy it (it is a credential).
   Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
   normally.
2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
   gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
   the `ENGINE` hook — a shim that intercepts `run` and re-execs
   `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
   `tracked_files()` is this bug, not a broken tree.
3. **`spec-immutability` skips** — the skip called tolerated above is also *avoidable*. `dev.sh`
   derives the sibling spec path from its own `REPO_ROOT`, which in a worktree is your
   workspace, so `../home-network-k3s` does not resolve and it adds no mount. Mount the spec
   repo read-only in the same `ENGINE` shim and pass `-e SPEC_REPO=/home-network-k3s`: with
   both mounts the stage reports `pass`, not SKIPPED, and R-TEST-06 is actually covered
   (measured — `decisions/2026-10-01-render-selftest-runs-in-gate.md`).

## 4. Branch naming

Use `feat/`, `fix/`, `docs/`, `chore/`. **One branch, one workspace, one PR** — if you
need a second branch, make a second worktree (§0); do not re-point an existing one. Note that `.gitea/workflows/gate.yml`'s **push**
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.

## 5. Cluster access

`kubectl` does not work on the operator workstation; the toolchain lives only in the
pinned container. Always name the context explicitly:

    ./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

`dev.sh` merges both clusters and picks a current-context **per invocation**; that
default has already changed once. Never rely on it. Do not run two `dev.sh`
invocations concurrently.

**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**

## 6. Secrets

- **Never `sops -d` a file into a terminal, a log, or a transcript.**
- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
  list key names or object names only. A password has been leaked into a transcript
  this way before.
- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
  the single creation-rules file; do not add a second one in a subtree.
- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
  flow. Do not design around committing one — read the expiry from the issuing page and
  treat a burnt token as free to retry.

## 7. Before you finish — write a handoff entry, in its OWN file

Add **one new file**:

    decisions/YYYY-MM-DD-<slug>.md          <slug> = your branch's last segment

with frontmatter (`date`, `branch`, `title`) and exactly these three sections:

    ## What changed
    ## What I verified
    ## What I did NOT verify

State gaps plainly — an honest "UNVERIFIED" is worth more than an optimistic "green",
and this repo has already been burned by the difference. All three headings are
**enforced** by `ci/decisions-log.sh`, because the third is the one that gets dropped.
Template and full rationale: `decisions/README.md`.

**Do NOT append to `DECISIONS-LOG.md`.** It is a frozen archive (pre-2026-09-29), its
SHA-256 is pinned in `decisions/ARCHIVE.sha256`, and the merge gate rejects any change
to it. Appending to its tail is what every agent used to do, and because every branch
appended to the *same last hunk*, no two concurrent branches could auto-merge. Measured
on 2026-09-29: **four of the five open PRs were `mergeable: false`, with
`DECISIONS-LOG.md` the only conflicting path in all four** — none of those changes had
anything to do with each other. Separate files have nothing to merge, which is the
entire fix. Note that the damage from getting this wrong lands on *other people's*
branches, later, which is why it is a gate stage and not a request.

Many older handoff entries and runbooks say "append to `DECISIONS-LOG.md`". That
wording is **stale**, in the same way §1's `git push origin main` wording is stale.