From 577c62136b5d76a126b401d0509ee639f3096584 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 5 Sep 2026 14:17:41 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20DECISIONS-LOG=20=E2=80=94=20PM=20autono?= =?UTF-8?q?mous=20session=20handoff?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Backups moved to R2 (blocked on the operator's token), CDI cold-start deadlock repaired in the graph, six L2 tests made environment-agnostic, conformance 82.4% -> 88.2%. Records the guard I broke and how it was caught, the R-TLS-02 'revoked' gap with the prune/rate-limit trade-off that makes it an operator call, and the merge preconditions for both the CDI split and the R2 switch. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 76 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index ccc77cf..f5c5982 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1860,3 +1860,79 @@ healthy reading during the incident. Never verify DNS or TLS here through a loca change stalls dev's fetching (cluster keeps serving; fix is a `kubectl patch`). Static reservation. - **Six L2 tests hardcode the apex** and must be parameterized per environment. - Nothing merged to `main`; nothing pushed. `infra/PROD-STANDUP.md` is the command sequence. + +## 2026-09-05 (afternoon) — **PM autonomous session.** Backups to R2, CDI deadlock repaired, coverage 82%→88% — and a guard I broke + +Operator away ~2h, PM directed to work autonomously through agents. Four agents ran; +every result was re-verified by the PM before commit. **Nothing merged, nothing pushed.** +Both clusters healthy throughout — dev 13/13, prod 12/13 (velero deliberately suspended). + +Commits: `32d6766` R2 · `765ad03` CDI · `59ffbaf` L2 params · `8f9b2e4` s3check · `fe45d0f` coverage. + +### THE MISTAKE, AND WHY IT MATTERS MORE THAN THE FEATURES + +In `32d6766` I changed the backup placeholder from `TBD-PROD-S3-BACKUP-ENDPOINT` to +`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`. The guards in `standup.sh` matched +`TBD-*` — a **prefix** glob. `fnmatch('https://TBD-R2-…','TBD-*')` is **False**, so all three +guards silently died, and **`resume` would have enabled velero on prod against a nonexistent +hostname**: installs clean, reports healthy, no backup exists. Precisely the failure the guard +existed to prevent, disarmed by a value change in a *different file*. + +**The class, now seen three times today** (this; the CRD count that read a blank line as `1`; +two coverage checks that passed when perturbed): *a check that degrades to success*. The +structural lesson is narrower and worth keeping — **a guard keyed on a value's SHAPE, where the +value lives in another file, has no test tying the two together.** Fixed by substring match; +a test for the coupling is **not** yet written and should be. + +### Operator decisions waiting + +1. **R-TLS-02 "revoked" is NOT satisfied — a real platform gap.** The spec requires certs be + "issued, renewed, **and revoked** without manual steps"; docs/08 reads revocation as a + declarative removal. That needs `enableCertificateOwnerRef: true`, which + `platform/cert-manager/helmrelease.yaml` does not set (chart default false). **Verified on + both live clusters: `gateway/wildcard-home-tls` has ZERO ownerReferences** — deleting the + Certificate leaves the private key serving indefinitely. + **Deliberately not flipped.** The wildcard Certificate is under `./platform`, which runs + `prune: true`. With owner refs ON, a transient prune of that Certificate — *exactly what + happened to the CDI StorageProfiles today* — deletes the SERVING TLS Secret for every host, + and reissuance is bounded by LE's duplicate-certificate limit (5/week per identical name set), + risking **no trusted cert for a week**. Security gap vs plausible week-long outage: operator's + call. The L2 test asserts the requirement as written and **will fail until decided** — it was + not softened. +2. **R2 needs the two token values.** `infra/BACKUP-R2.md` is the procedure. The **account id is + already committed** (`infra/tofu/backend.r2.hcl` → `7ce452a1b3d6c86daf2c020edcb82d37`), so if + the buckets go in that account only the token is missing. Placeholder kept on purpose. +3. **The apex flip is also the legacy-prod 443 cutover.** The SOHO router NATs inbound 443 to the + pfSense WAN for the non-k3s prod. §10a sequence P does not cover repointing it. + +### Merge preconditions — read before merging anything here + +- **CDI split:** `flux get kustomization platform-config` must be **Ready on the target first**. + Otherwise the two StorageProfiles are pruned and sit at CDI defaults until it recovers (prod was + in that state hours earlier). Measured exposure: CDI unaided infers **ReadWriteOnce** + (`longhorn-static`) where we author **ReadWriteMany** (`longhorn-vm`) — so a DataVolume created + in the ~30–60s window is **permanently non-live-migratable**. `cloneStrategy` is *not* at risk. +- **R2 switch RESETS dev's PITR window** — fresh kopia repo, new CNPG base backups, the 30-day + Barman window rebuilds from zero. The MinIO copy is retained and restorable. +- **R-STATE-03 is PROVEN MinIO-ONLY** and its ledger row predates the R2 move. `BACKUP-R2.md` §5 + is the re-proof and it ends in a **restore**, not an upload. + +### Findings worth keeping + +- **Second-order CRDs need a third pass.** A CR whose CRD is created by a *controller at runtime* + cannot share a Kustomization with resources of that CRD. `cdi-operator` registers exactly one + CRD; `storageprofiles`/`datavolumes`/`datasources` appear only after it reconciles the CDI CR. + Audit found `StorageProfile` was the only second-order kind in `platform-config`. +- **`dns-exposure`'s teardown is ASYNCHRONOUS.** It reports PASS when chainsaw deletes the + namespace; the Cloudflare record is reaped later by external-dns's 1-minute sync. A green run + does **not** prove cleanup — if external-dns were suspended the record would persist silently. + (Observed, then observed clearing.) +- **The apex was not the only per-environment literal in the L2 suite** — two DNS tests hardcoded + dev's LB range `192.168.1.53-.60`. They now assert the *observed* `GATEWAY_VIP` / assigned LB IP, + which is stricter as well as portable. +- **Velero's kopia backend has no checksum knob**, so BSL metadata and PVC-data R2 compatibility + are **separate questions**. CNPG's boto3 1.35.60 is below the 1.36 default-checksum line — clear + today, but **a CNPG image bump is an R2 re-verification event** (barman-cloud has no escape hatch). +- **`ansible-inventory --list` prints DECRYPTED vault values** (the passphrase file is mounted into + the dev container). It emitted the prod become-password to an agent's stdout. Never put it in CI. + Confirmed no plaintext reached disk.