docs(decisions-log): PM handoff — two fail-open holes in the gate itself

Records the session's three pieces of work and, more importantly, the theme
joining them: the gate was reporting green over things it had not checked.

- runbooks/node-memory-swap.md, with the three claims I wrote and then removed
  because they were false (EtcdIsVoter does not exist here; node-drain-policy is
  not pinned so the runbook reads it live; the 5-vs-1 rebuild-limit confusion).
- The nested-worktree scan: six discovery sites auditing 18 copies of the repo.
- L1 rendering never running on a requirement-scoped gate, with the external-dns
  incident reproduced via --print-selection (0 tests before, 3 after).
- The two defects that surfaced in the meta-check guarding all of it.

PM HANDOFF lists the open items in priority order. The first is a judgement call
worth flagging rather than burying: the planned memory swap on k3s-prod-1 IS a
supervised single-node loss on prod, which is exactly the event R-STATE-02
describes and which is still proven only on the decommissioned dev cluster.
Capturing evidence during the swap costs nothing; manufacturing the event later
costs a second maintenance window.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -4265,3 +4265,134 @@ recreating the VMI. Then the leaked value never becomes a working password anywh
 `.data` answers a question nobody asked. Every previous secret check in this session used a
 keys-only or boolean form — this one regressed because I appended a convenience field to an
 otherwise-safe command without re-reading what it expanded to.
+
+## 2026-09-18 — Two fail-open holes in the gate itself: nested worktrees, and L1 never running
+
+Three pieces of work, one theme: the gate was reporting green over things it had not checked.
+
+### 1. `runbooks/node-memory-swap.md` — the procedure the operator asked for
+
+Keith has a matched 32 GB pair to install and asked which node needs it. **`k3s-prod-1`**: `DIMM1` is a
+Micron `8ATF2G64HZ-3G2E1` (16 GB, DDR4-3200, 1Rx8), `DIMM3` a `16ATF2G64HZ-2G6E1` (16 GB, DDR4-2667,
+2Rx8). Capacity is correct; the pair runs at the slower 2667. Nothing is faulty. `k3s-prod-3` is a
+matched Crucial pair.
+
+The runbook front-loads the go/no-go checks because this is the awkward node to lose: 24 Longhorn
+replicas, one of three etcd members (drain leaves quorum at 2 of 3, no further margin), and its
+rebuilds are the load that starved etcd once. A BLOCKED drain is treated as the safety property
+working, never something to force past.
+
+Three things I had written and then removed, because they were not true:
+
+- **`EtcdIsVoter` does not exist in this cluster.** I had an assertion built on it; it appears nowhere
+  in the repo. The house check is `curl http://127.0.0.1:2381/metrics | grep etcd_server_has_leader`
+  (`runbooks/node-down.md:93`).
+- **`node-drain-policy` is not pinned** in `platform/longhorn/helmrelease.yaml`. The runbook has the
+  operator READ the live Setting and set it before draining rather than assuming the chart default —
+  that Setting is the entire safety story for 24 replicas.
+- I had written "rebuild concurrency is deliberately 1". That is the purge/restore limits, not replica
+  rebuild — the same 5-vs-1 confusion already corrected once in `longhorn-degraded.md`.
+
+Verification hinges on **`Configured Memory Speed`**, not `Speed`: a matched 3200 pair being clocked
+down still reports `Speed: 3200 MT/s`, so matched part numbers alone do not prove the swap worked.
+
+Also recorded: the `node-health` artifacts de-duplicate DIMM part numbers, so they can prove "two
+different modules" but cannot distinguish one module from two identical ones — verification collects
+`dmidecode` live. **Node 2's inventory remains stale and self-contradictory** (documented as the only
+single-DIMM node at 16 GB, yet reporting ~31.1 GiB) and needs re-inventory before anyone claims
+three-way symmetry. This is NOT the §0b "do not buy RAM" case, which was an EFI memattr kernel bug.
+
+### 2. The whole-repo L0 checks were auditing 18 copies of the repo
+
+Six discovery sites walked `ROOT` without pruning dot-directories, so they descended into
+`.claude/worktrees/agent-*/` — full repo copies from isolated agent runs, gitignored at
+`.gitignore:43`.
+
+The visible half was false FAILs: four ServiceMonitors "selecting no Service", nine PrometheusRules
+"under test but not in the repo", every path a worktree path. The whole-repo gate was red for reasons
+with nothing to do with authored surface.
+
+The dangerous half is quieter, and is why all six were fixed rather than only the failing ones: a check
+asking *"does anything in the repo declare X"* can be satisfied by a worktree copy instead of real
+source. The same bug that invents failures can manufacture a pass.
+
+`egress-paths-selected-by-rules` already had the fix and documents hitting it (it once saw 5
+`CiliumEgressGatewayPolicy` objects instead of the 1 authored). Its dot-directory predicate is now
+applied everywhere, plus both `find` calls in `platform/monitoring/tests/run.sh`, whose `--list`
+reported each manifest once per worktree — 21 real pairs where it had been returning them ~19x over.
+
+### 3. L1 rendering never ran on a requirement-scoped gate — closed, with the incident reproduced
+
+`ci/l1.sh` has passed `--include-scope whole-repo` since it was written, with a comment saying no L1
+test declared the scope yet and it was wired ahead of time. Nothing ever declared it, so **the entire
+L1 rendering layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or
+R-NET-05/06/07.**
+
+All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`. The justification is one
+shared code path — `catalog/render.py` against `schemas/servicedefinition.schema.json` — which
+requirement-scoped selection cannot see.
+
+The 2026-09-04 external-dns merge is the proof, reproduced rather than asserted.
+`ci/touched-requirements.sh` attributes `platform/externaldns/helmrelease.yaml` to R-ENV-02,
+R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — **none** of the rendering requirements — yet that
+change added `external-dns.alpha.kubernetes.io/controller: none` to the rendered output of all three
+goldens. Measured with `--print-selection`:
+
+| selection for that change | L1 tests selected |
+|---|---|
+| requirement-scoped only (the 2026-09-04 behaviour) | **0** |
+| with `--include-scope whole-repo` (how `ci/l1.sh` invokes it) | **3** |
+
+### 4. …which exposed two defects in the meta-check that guards all of this
+
+Declaring the scope at L1 made `whole-repo-invariants-wired` (R-TEST-03) fail, and it was right to
+change behaviour — but for the wrong reason, and it could not say so:
+
+- **It was layer-agnostic where the design is layer-bounded.** It compared *every* scope-declaring dir
+  against an `--layer l0` selection, assuming only L0 tests would ever declare the scope — while
+  `ci/l1.sh` was written from the opposite premise, that a whole-repo invariant could appear "at any
+  layer". So it blamed the union for correctly excluding another layer. Now partitioned by layer, with
+  the L1 selection and L1 leakage assertions added — which is what makes the L1 declaration
+  load-bearing instead of decorative.
+- **Every assertion in it failed silently.** The idiom was `<test>` on one line, `chk $?` on the next,
+  under `set -euo pipefail` — so a false test aborted the script *before* `chk` could print. The real
+  failure I hit emitted 26 `ok` lines, rc=1, and no message. All 12 sites now capture the status
+  (`chk $arc`), proven by forcing `MIN_DECLARED=999`: the failure prints its full diagnostic and the
+  run continues to the remaining assertions.
+
+A meta-check that cannot report why it failed is worse than no meta-check, because the silence reads
+as a tooling glitch rather than a finding.
+
+### Gate
+
+Offline `l0,l1,meta --include-scope whole-repo` in the pinned toolbox: **25 tests, 0 failures, 0
+errors, exit 0**. L0 whole-repo went 4 failures → 1 → 0 across the two fixes;
+`whole-repo-invariants-wired` now runs 35 assertions. Host runs report one skip because `promtool` is
+not on `$PATH` there — the toolbox run is the real result, and that difference is worth remembering
+before trusting a host-only gate run.
+
+### PM HANDOFF
+
+Open, in the order I would take them:
+
+1. **The memory swap is a free R-STATE-02 proof.** `node-loss` is Phase 7's last exit item and is still
+   proven only on the decommissioned dev cluster. The swap is a planned, supervised single-node loss on
+   prod — exactly the event R-STATE-02 describes. Capturing evidence during it costs nothing extra;
+   manufacturing the event later costs a second maintenance window.
+2. **Node 2 DIMM re-inventory** — blocked on the vault become password for `node_health`. Ansible only;
+   never `ansible-inventory --list` (it prints decrypted vault values).
+3. **Workstation DNS, one sudo command.** `/etc/dnsmasq.d/10-home.keithjackson.dev.conf` contains
+   `local=/home.keithjackson.dev/` plus `address=/.home.keithjackson.dev/192.168.1.53` — the whole zone
+   hardcoded to the decommissioned dev cluster, with forwarding forbidden so no upstream could correct
+   it. That is one root cause for both the sonarr failure and `gitea` not resolving. Pi-hole answers
+   correctly (`gitea` → `192.168.2.20`); only this file is wrong. Correction to an earlier claim of
+   mine: there is no separate `20-prod.*` file — it is a replace of this one file. Router DHCP option
+   6 → `.23` is the fix that makes it unnecessary per-machine.
+4. **UniFi guest has no IP** — needs console access; `cat /etc/netplan/*.yaml` is the confirming command.
+5. **Five `public` services have no working public path** — waiting on WAN forwarding, which Keith has
+   said is deliberately not ready.
+6. **18 stale agent worktrees** on already-merged branches remain on disk. No longer load-bearing for
+   the gate; cleaning them up is cosmetic and should check each for uncommitted work first.
+7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, 103 unmapped
+   paths, and two rendered manifests nothing applies (the keycloak one would stand up a conflicting
+   instance).