diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 689df16..b6b4f64 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -4265,3 +4265,134 @@ recreating the VMI. Then the leaked value never becomes a working password anywh `.data` answers a question nobody asked. Every previous secret check in this session used a keys-only or boolean form — this one regressed because I appended a convenience field to an otherwise-safe command without re-reading what it expanded to. + +## 2026-09-18 — Two fail-open holes in the gate itself: nested worktrees, and L1 never running + +Three pieces of work, one theme: the gate was reporting green over things it had not checked. + +### 1. `runbooks/node-memory-swap.md` — the procedure the operator asked for + +Keith has a matched 32 GB pair to install and asked which node needs it. **`k3s-prod-1`**: `DIMM1` is a +Micron `8ATF2G64HZ-3G2E1` (16 GB, DDR4-3200, 1Rx8), `DIMM3` a `16ATF2G64HZ-2G6E1` (16 GB, DDR4-2667, +2Rx8). Capacity is correct; the pair runs at the slower 2667. Nothing is faulty. `k3s-prod-3` is a +matched Crucial pair. + +The runbook front-loads the go/no-go checks because this is the awkward node to lose: 24 Longhorn +replicas, one of three etcd members (drain leaves quorum at 2 of 3, no further margin), and its +rebuilds are the load that starved etcd once. A BLOCKED drain is treated as the safety property +working, never something to force past. + +Three things I had written and then removed, because they were not true: + +- **`EtcdIsVoter` does not exist in this cluster.** I had an assertion built on it; it appears nowhere + in the repo. The house check is `curl http://127.0.0.1:2381/metrics | grep etcd_server_has_leader` + (`runbooks/node-down.md:93`). +- **`node-drain-policy` is not pinned** in `platform/longhorn/helmrelease.yaml`. The runbook has the + operator READ the live Setting and set it before draining rather than assuming the chart default — + that Setting is the entire safety story for 24 replicas. +- I had written "rebuild concurrency is deliberately 1". That is the purge/restore limits, not replica + rebuild — the same 5-vs-1 confusion already corrected once in `longhorn-degraded.md`. + +Verification hinges on **`Configured Memory Speed`**, not `Speed`: a matched 3200 pair being clocked +down still reports `Speed: 3200 MT/s`, so matched part numbers alone do not prove the swap worked. + +Also recorded: the `node-health` artifacts de-duplicate DIMM part numbers, so they can prove "two +different modules" but cannot distinguish one module from two identical ones — verification collects +`dmidecode` live. **Node 2's inventory remains stale and self-contradictory** (documented as the only +single-DIMM node at 16 GB, yet reporting ~31.1 GiB) and needs re-inventory before anyone claims +three-way symmetry. This is NOT the §0b "do not buy RAM" case, which was an EFI memattr kernel bug. + +### 2. The whole-repo L0 checks were auditing 18 copies of the repo + +Six discovery sites walked `ROOT` without pruning dot-directories, so they descended into +`.claude/worktrees/agent-*/` — full repo copies from isolated agent runs, gitignored at +`.gitignore:43`. + +The visible half was false FAILs: four ServiceMonitors "selecting no Service", nine PrometheusRules +"under test but not in the repo", every path a worktree path. The whole-repo gate was red for reasons +with nothing to do with authored surface. + +The dangerous half is quieter, and is why all six were fixed rather than only the failing ones: a check +asking *"does anything in the repo declare X"* can be satisfied by a worktree copy instead of real +source. The same bug that invents failures can manufacture a pass. + +`egress-paths-selected-by-rules` already had the fix and documents hitting it (it once saw 5 +`CiliumEgressGatewayPolicy` objects instead of the 1 authored). Its dot-directory predicate is now +applied everywhere, plus both `find` calls in `platform/monitoring/tests/run.sh`, whose `--list` +reported each manifest once per worktree — 21 real pairs where it had been returning them ~19x over. + +### 3. L1 rendering never ran on a requirement-scoped gate — closed, with the incident reproduced + +`ci/l1.sh` has passed `--include-scope whole-repo` since it was written, with a comment saying no L1 +test declared the scope yet and it was wired ahead of time. Nothing ever declared it, so **the entire +L1 rendering layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or +R-NET-05/06/07.** + +All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`. The justification is one +shared code path — `catalog/render.py` against `schemas/servicedefinition.schema.json` — which +requirement-scoped selection cannot see. + +The 2026-09-04 external-dns merge is the proof, reproduced rather than asserted. +`ci/touched-requirements.sh` attributes `platform/externaldns/helmrelease.yaml` to R-ENV-02, +R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — **none** of the rendering requirements — yet that +change added `external-dns.alpha.kubernetes.io/controller: none` to the rendered output of all three +goldens. Measured with `--print-selection`: + +| selection for that change | L1 tests selected | +|---|---| +| requirement-scoped only (the 2026-09-04 behaviour) | **0** | +| with `--include-scope whole-repo` (how `ci/l1.sh` invokes it) | **3** | + +### 4. …which exposed two defects in the meta-check that guards all of this + +Declaring the scope at L1 made `whole-repo-invariants-wired` (R-TEST-03) fail, and it was right to +change behaviour — but for the wrong reason, and it could not say so: + +- **It was layer-agnostic where the design is layer-bounded.** It compared *every* scope-declaring dir + against an `--layer l0` selection, assuming only L0 tests would ever declare the scope — while + `ci/l1.sh` was written from the opposite premise, that a whole-repo invariant could appear "at any + layer". So it blamed the union for correctly excluding another layer. Now partitioned by layer, with + the L1 selection and L1 leakage assertions added — which is what makes the L1 declaration + load-bearing instead of decorative. +- **Every assertion in it failed silently.** The idiom was `` on one line, `chk $?` on the next, + under `set -euo pipefail` — so a false test aborted the script *before* `chk` could print. The real + failure I hit emitted 26 `ok` lines, rc=1, and no message. All 12 sites now capture the status + (`chk $arc`), proven by forcing `MIN_DECLARED=999`: the failure prints its full diagnostic and the + run continues to the remaining assertions. + +A meta-check that cannot report why it failed is worse than no meta-check, because the silence reads +as a tooling glitch rather than a finding. + +### Gate + +Offline `l0,l1,meta --include-scope whole-repo` in the pinned toolbox: **25 tests, 0 failures, 0 +errors, exit 0**. L0 whole-repo went 4 failures → 1 → 0 across the two fixes; +`whole-repo-invariants-wired` now runs 35 assertions. Host runs report one skip because `promtool` is +not on `$PATH` there — the toolbox run is the real result, and that difference is worth remembering +before trusting a host-only gate run. + +### PM HANDOFF + +Open, in the order I would take them: + +1. **The memory swap is a free R-STATE-02 proof.** `node-loss` is Phase 7's last exit item and is still + proven only on the decommissioned dev cluster. The swap is a planned, supervised single-node loss on + prod — exactly the event R-STATE-02 describes. Capturing evidence during it costs nothing extra; + manufacturing the event later costs a second maintenance window. +2. **Node 2 DIMM re-inventory** — blocked on the vault become password for `node_health`. Ansible only; + never `ansible-inventory --list` (it prints decrypted vault values). +3. **Workstation DNS, one sudo command.** `/etc/dnsmasq.d/10-home.keithjackson.dev.conf` contains + `local=/home.keithjackson.dev/` plus `address=/.home.keithjackson.dev/192.168.1.53` — the whole zone + hardcoded to the decommissioned dev cluster, with forwarding forbidden so no upstream could correct + it. That is one root cause for both the sonarr failure and `gitea` not resolving. Pi-hole answers + correctly (`gitea` → `192.168.2.20`); only this file is wrong. Correction to an earlier claim of + mine: there is no separate `20-prod.*` file — it is a replace of this one file. Router DHCP option + 6 → `.23` is the fix that makes it unnecessary per-machine. +4. **UniFi guest has no IP** — needs console access; `cat /etc/netplan/*.yaml` is the confirming command. +5. **Five `public` services have no working public path** — waiting on WAN forwarding, which Keith has + said is deliberately not ready. +6. **18 stale agent worktrees** on already-merged branches remain on disk. No longer load-bearing for + the gate; cleaning them up is cosmetic and should check each for uncommitted work first. +7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, 103 unmapped + paths, and two rendered manifests nothing applies (the keycloak one would stand up a conflicting + instance).