diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index a1956ce..95f7348 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1407,3 +1407,82 @@ definition of done, only a green requirement-tagged test is acceptance — so th to make `dr-restore/chainsaw-test.yaml` and the node-loss test actually assert the behaviour the operator has already demonstrated, and let the runner enforce it. Recording the attestation here does not close the requirement, and the gate must not be softened to make it appear closed. + +--- + +## 2026-09-01 — Phase-7 prep session complete. **PM HANDOFF.** + +Everything below is on branch `docs/phase-7-open-prod-node-1` / **PR #4**, 13 commits, the +enforced Gitea gate **green**. Nothing has been installed on prod hardware. `main` is untouched +until the operator merges. + +### Delivered +1. **Phase 7 opened (prep only)** with the entry-gate shortfall recorded, not waived. +2. **T-7.1 checklist** (`infra/PROD-PROVISIONING.md`) written against the real hardware. +3. **R-ENV-02 tokenization** — 6 `cluster-vars` knobs; prod would otherwise have come up + announcing DEV addresses. Dev proven byte-identical. +4. **Prod Ansible**: inventory, vaulted `become` (prod gets NO passwordless sudo), and + **target-group parameterisation** so prod reuses the roles unchanged. +5. **`node_health` role** — read-only, portable dev+prod, run against both. +6. **Repo-derived spec status report** (`run.sh --status`). +7. **CI false-green killed** + L0 static tests for R-SEC-01/02 and R-TEST-06. +8. **dind readiness fix**; **DNS resilience re-authored** for the current stack. +9. **`dr-restore` now asserts** — safety-critical scaffolds 0. + +### Numbers +Coverage 37/51 → **40/51**. Executable tests 32 → **36**. Scaffold-only 5 → **4**. +Safety-critical without an executable L3 1 → **0**. **PROVEN: 0 — still not derivable**, by +design; proof-of-pass is not recoverable from repo state, so a green-run ledger is the only +thing that ever closes a requirement. + +### Corrections made during the session (recorded so they are not re-litigated) +- **NOPASSWD sudo on prod: REJECTED** by the operator. Prod uses vaulted `become`. My original + checklist copied dev's convenience onto prod; that was wrong. +- **The "`$i` envsubst bug" was FALSE.** An assessment claimed Flux envsubst ate bare `$i`, + making the dind cert-wait bound a no-op. Running the real `flux envsubst` disproved it: bare + `$VAR` passes through; only `${VAR}` expands — matching this repo's own contract in + `clusters/base/apps-vms.yaml`, itself written after empirical probing. **No escaping was + applied**; `$${i}` would have been an active regression (correct only on the envsubst path — + on a plain `kustomize build`, `sh` reads `$$` as its PID). Agent reports are not evidence; + the tool is. +- **The stranded DNS branch was NOT landed.** Its netplan mechanism no longer exists on Debian + 13; landing it would have broken `node-baseline.yml` on every node. + +### Near-misses worth remembering +- A YAML **comment** nearly broke every merge in the repo: Flux envsubst runs over comments too, + so a braced name with punctuation is a fatal parse error that fails the whole `apps-wave-1` + Kustomization. +- `roles/cilium`, `cluster_verify`, `flux_bootstrap` all hardcoded `.cache/dev.kubeconfig` — the + file `dev.sh` exports as `$KUBECONFIG`. The moment prod became targetable, a prod run would + have silently overwritten it and left the operator's `kubectl` answering from **prod**. +- Prod's `GATEWAY_VIP` outside the LB pool would have brought the Gateway up on `.21` while + split-horizon DNS answered `.20` — blackholing in-cluster SSO while reporting healthy. + +### BLOCKED ON THE OPERATOR — nothing else moves without these +1. **Create `inventory/group_vars/k3s_prod/vault.yml`** (`ansible_become_password`). The + passphrase file already exists. This gates SMART, the packages-only baseline, and every + future prod play. +2. **Drive health decision** — run `smartctl -a /dev/sda` once `smartmontools` lands via + `node-baseline.yml -e k3s_hosts=k3s_prod --tags packages`. Prod state on an unmeasured + ~2011 consumer SSD is the open risk. +3. **Prod's second DNS upstream** — `192.168.2.1` is both gateway and only IPv4 resolver. +4. **Prod backup target** — `S3_BACKUP_ENDPOINT` is still `TBD-PROD-*`; dev's versitygw is on + the far side of the pfSense boundary. +5. **`UNIFI_L4_VIP` / `CRAFTY_L4_VIP`** — allocatable from `.20-.30`, not invented here. +6. **Merge PR #4.** + +### Known, documented, NOT fixed +- `roles/flux_bootstrap` shells to `bootstrap/flux/bootstrap.sh` with a hardcoded + `--path=clusters/dev` — retargeting prod needs a `bootstrap/` change. +- Prod has no VPN egress group (needs its own vaulted PIA credentials). +- **`dr-restore` is AUTHORED, NEVER EXECUTED** (`conformance.home.k3s/never-executed`). It is + destructive; converting it to proven needs an operator maintenance-window run. The log must + show `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`, **not** `DR_RESTORE_SKIPPED_OK` — green but + proving nothing. +- `dind` → native sidecar `initContainer` with a `docker info` startupProbe (NOT a bare + tcpSocket probe) is the right shape on k3s 1.36; deferred so a merge-gate failure is + unambiguous. +- **ECC is unknowable on this hardware** — no EDAC controller bound; non-ECC consumer RAM. + Silent memory corruption on a prod node would be invisible. Hardware property, not fixable. +- L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs + the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI.