docs: PM handoff — Phase-7 prep session complete (PR #4, 13 commits, CI green)
Records what landed, the numbers (coverage 37->40/51, safety-critical scaffolds 1->0, PROVEN still 0 by design), the corrections made mid-session so they are not re-litigated (prod NOPASSWD rejected; the '$i envsubst bug' disproven by running the real tool; the stranded DNS branch dropped rather than landed), three near-misses worth remembering, the six items blocked on the operator, and what is knowingly left undone. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
c1f24e4a6d
commit
ce1966acb0
1 file changed
+79
@@ -1407,3 +1407,82 @@ definition of done, only a green requirement-tagged test is acceptance — so th
|
||||
to make `dr-restore/chainsaw-test.yaml` and the node-loss test actually assert the behaviour the
|
||||
operator has already demonstrated, and let the runner enforce it. Recording the attestation here
|
||||
does not close the requirement, and the gate must not be softened to make it appear closed.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-01 — Phase-7 prep session complete. **PM HANDOFF.**
|
||||
|
||||
Everything below is on branch `docs/phase-7-open-prod-node-1` / **PR #4**, 13 commits, the
|
||||
enforced Gitea gate **green**. Nothing has been installed on prod hardware. `main` is untouched
|
||||
until the operator merges.
|
||||
|
||||
### Delivered
|
||||
1. **Phase 7 opened (prep only)** with the entry-gate shortfall recorded, not waived.
|
||||
2. **T-7.1 checklist** (`infra/PROD-PROVISIONING.md`) written against the real hardware.
|
||||
3. **R-ENV-02 tokenization** — 6 `cluster-vars` knobs; prod would otherwise have come up
|
||||
announcing DEV addresses. Dev proven byte-identical.
|
||||
4. **Prod Ansible**: inventory, vaulted `become` (prod gets NO passwordless sudo), and
|
||||
**target-group parameterisation** so prod reuses the roles unchanged.
|
||||
5. **`node_health` role** — read-only, portable dev+prod, run against both.
|
||||
6. **Repo-derived spec status report** (`run.sh --status`).
|
||||
7. **CI false-green killed** + L0 static tests for R-SEC-01/02 and R-TEST-06.
|
||||
8. **dind readiness fix**; **DNS resilience re-authored** for the current stack.
|
||||
9. **`dr-restore` now asserts** — safety-critical scaffolds 0.
|
||||
|
||||
### Numbers
|
||||
Coverage 37/51 → **40/51**. Executable tests 32 → **36**. Scaffold-only 5 → **4**.
|
||||
Safety-critical without an executable L3 1 → **0**. **PROVEN: 0 — still not derivable**, by
|
||||
design; proof-of-pass is not recoverable from repo state, so a green-run ledger is the only
|
||||
thing that ever closes a requirement.
|
||||
|
||||
### Corrections made during the session (recorded so they are not re-litigated)
|
||||
- **NOPASSWD sudo on prod: REJECTED** by the operator. Prod uses vaulted `become`. My original
|
||||
checklist copied dev's convenience onto prod; that was wrong.
|
||||
- **The "`$i` envsubst bug" was FALSE.** An assessment claimed Flux envsubst ate bare `$i`,
|
||||
making the dind cert-wait bound a no-op. Running the real `flux envsubst` disproved it: bare
|
||||
`$VAR` passes through; only `${VAR}` expands — matching this repo's own contract in
|
||||
`clusters/base/apps-vms.yaml`, itself written after empirical probing. **No escaping was
|
||||
applied**; `$${i}` would have been an active regression (correct only on the envsubst path —
|
||||
on a plain `kustomize build`, `sh` reads `$$` as its PID). Agent reports are not evidence;
|
||||
the tool is.
|
||||
- **The stranded DNS branch was NOT landed.** Its netplan mechanism no longer exists on Debian
|
||||
13; landing it would have broken `node-baseline.yml` on every node.
|
||||
|
||||
### Near-misses worth remembering
|
||||
- A YAML **comment** nearly broke every merge in the repo: Flux envsubst runs over comments too,
|
||||
so a braced name with punctuation is a fatal parse error that fails the whole `apps-wave-1`
|
||||
Kustomization.
|
||||
- `roles/cilium`, `cluster_verify`, `flux_bootstrap` all hardcoded `.cache/dev.kubeconfig` — the
|
||||
file `dev.sh` exports as `$KUBECONFIG`. The moment prod became targetable, a prod run would
|
||||
have silently overwritten it and left the operator's `kubectl` answering from **prod**.
|
||||
- Prod's `GATEWAY_VIP` outside the LB pool would have brought the Gateway up on `.21` while
|
||||
split-horizon DNS answered `.20` — blackholing in-cluster SSO while reporting healthy.
|
||||
|
||||
### BLOCKED ON THE OPERATOR — nothing else moves without these
|
||||
1. **Create `inventory/group_vars/k3s_prod/vault.yml`** (`ansible_become_password`). The
|
||||
passphrase file already exists. This gates SMART, the packages-only baseline, and every
|
||||
future prod play.
|
||||
2. **Drive health decision** — run `smartctl -a /dev/sda` once `smartmontools` lands via
|
||||
`node-baseline.yml -e k3s_hosts=k3s_prod --tags packages`. Prod state on an unmeasured
|
||||
~2011 consumer SSD is the open risk.
|
||||
3. **Prod's second DNS upstream** — `192.168.2.1` is both gateway and only IPv4 resolver.
|
||||
4. **Prod backup target** — `S3_BACKUP_ENDPOINT` is still `TBD-PROD-*`; dev's versitygw is on
|
||||
the far side of the pfSense boundary.
|
||||
5. **`UNIFI_L4_VIP` / `CRAFTY_L4_VIP`** — allocatable from `.20-.30`, not invented here.
|
||||
6. **Merge PR #4.**
|
||||
|
||||
### Known, documented, NOT fixed
|
||||
- `roles/flux_bootstrap` shells to `bootstrap/flux/bootstrap.sh` with a hardcoded
|
||||
`--path=clusters/dev` — retargeting prod needs a `bootstrap/` change.
|
||||
- Prod has no VPN egress group (needs its own vaulted PIA credentials).
|
||||
- **`dr-restore` is AUTHORED, NEVER EXECUTED** (`conformance.home.k3s/never-executed`). It is
|
||||
destructive; converting it to proven needs an operator maintenance-window run. The log must
|
||||
show `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`, **not** `DR_RESTORE_SKIPPED_OK` — green but
|
||||
proving nothing.
|
||||
- `dind` → native sidecar `initContainer` with a `docker info` startupProbe (NOT a bare
|
||||
tcpSocket probe) is the right shape on k3s 1.36; deferred so a merge-gate failure is
|
||||
unambiguous.
|
||||
- **ECC is unknowable on this hardware** — no EDAC controller bound; non-ECC consumer RAM.
|
||||
Silent memory corruption on a prod node would be invisible. Hardware property, not fixable.
|
||||
- L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs
|
||||
the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI.
|
||||
Reference in new issue
Block a user