docs: DECISIONS-LOG — PM autonomous session handoff
Backups moved to R2 (blocked on the operator's token), CDI cold-start deadlock repaired in the graph, six L2 tests made environment-agnostic, conformance 82.4% -> 88.2%. Records the guard I broke and how it was caught, the R-TLS-02 'revoked' gap with the prune/rate-limit trade-off that makes it an operator call, and the merge preconditions for both the CDI split and the R2 switch. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
fe45d0f5c5
commit
577c62136b
1 file changed
+76
@@ -1860,3 +1860,79 @@ healthy reading during the incident. Never verify DNS or TLS here through a loca
|
||||
change stalls dev's fetching (cluster keeps serving; fix is a `kubectl patch`). Static reservation.
|
||||
- **Six L2 tests hardcode the apex** and must be parameterized per environment.
|
||||
- Nothing merged to `main`; nothing pushed. `infra/PROD-STANDUP.md` is the command sequence.
|
||||
|
||||
## 2026-09-05 (afternoon) — **PM autonomous session.** Backups to R2, CDI deadlock repaired, coverage 82%→88% — and a guard I broke
|
||||
|
||||
Operator away ~2h, PM directed to work autonomously through agents. Four agents ran;
|
||||
every result was re-verified by the PM before commit. **Nothing merged, nothing pushed.**
|
||||
Both clusters healthy throughout — dev 13/13, prod 12/13 (velero deliberately suspended).
|
||||
|
||||
Commits: `32d6766` R2 · `765ad03` CDI · `59ffbaf` L2 params · `8f9b2e4` s3check · `fe45d0f` coverage.
|
||||
|
||||
### THE MISTAKE, AND WHY IT MATTERS MORE THAN THE FEATURES
|
||||
|
||||
In `32d6766` I changed the backup placeholder from `TBD-PROD-S3-BACKUP-ENDPOINT` to
|
||||
`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`. The guards in `standup.sh` matched
|
||||
`TBD-*` — a **prefix** glob. `fnmatch('https://TBD-R2-…','TBD-*')` is **False**, so all three
|
||||
guards silently died, and **`resume` would have enabled velero on prod against a nonexistent
|
||||
hostname**: installs clean, reports healthy, no backup exists. Precisely the failure the guard
|
||||
existed to prevent, disarmed by a value change in a *different file*.
|
||||
|
||||
**The class, now seen three times today** (this; the CRD count that read a blank line as `1`;
|
||||
two coverage checks that passed when perturbed): *a check that degrades to success*. The
|
||||
structural lesson is narrower and worth keeping — **a guard keyed on a value's SHAPE, where the
|
||||
value lives in another file, has no test tying the two together.** Fixed by substring match;
|
||||
a test for the coupling is **not** yet written and should be.
|
||||
|
||||
### Operator decisions waiting
|
||||
|
||||
1. **R-TLS-02 "revoked" is NOT satisfied — a real platform gap.** The spec requires certs be
|
||||
"issued, renewed, **and revoked** without manual steps"; docs/08 reads revocation as a
|
||||
declarative removal. That needs `enableCertificateOwnerRef: true`, which
|
||||
`platform/cert-manager/helmrelease.yaml` does not set (chart default false). **Verified on
|
||||
both live clusters: `gateway/wildcard-home-tls` has ZERO ownerReferences** — deleting the
|
||||
Certificate leaves the private key serving indefinitely.
|
||||
**Deliberately not flipped.** The wildcard Certificate is under `./platform`, which runs
|
||||
`prune: true`. With owner refs ON, a transient prune of that Certificate — *exactly what
|
||||
happened to the CDI StorageProfiles today* — deletes the SERVING TLS Secret for every host,
|
||||
and reissuance is bounded by LE's duplicate-certificate limit (5/week per identical name set),
|
||||
risking **no trusted cert for a week**. Security gap vs plausible week-long outage: operator's
|
||||
call. The L2 test asserts the requirement as written and **will fail until decided** — it was
|
||||
not softened.
|
||||
2. **R2 needs the two token values.** `infra/BACKUP-R2.md` is the procedure. The **account id is
|
||||
already committed** (`infra/tofu/backend.r2.hcl` → `7ce452a1b3d6c86daf2c020edcb82d37`), so if
|
||||
the buckets go in that account only the token is missing. Placeholder kept on purpose.
|
||||
3. **The apex flip is also the legacy-prod 443 cutover.** The SOHO router NATs inbound 443 to the
|
||||
pfSense WAN for the non-k3s prod. §10a sequence P does not cover repointing it.
|
||||
|
||||
### Merge preconditions — read before merging anything here
|
||||
|
||||
- **CDI split:** `flux get kustomization platform-config` must be **Ready on the target first**.
|
||||
Otherwise the two StorageProfiles are pruned and sit at CDI defaults until it recovers (prod was
|
||||
in that state hours earlier). Measured exposure: CDI unaided infers **ReadWriteOnce**
|
||||
(`longhorn-static`) where we author **ReadWriteMany** (`longhorn-vm`) — so a DataVolume created
|
||||
in the ~30–60s window is **permanently non-live-migratable**. `cloneStrategy` is *not* at risk.
|
||||
- **R2 switch RESETS dev's PITR window** — fresh kopia repo, new CNPG base backups, the 30-day
|
||||
Barman window rebuilds from zero. The MinIO copy is retained and restorable.
|
||||
- **R-STATE-03 is PROVEN MinIO-ONLY** and its ledger row predates the R2 move. `BACKUP-R2.md` §5
|
||||
is the re-proof and it ends in a **restore**, not an upload.
|
||||
|
||||
### Findings worth keeping
|
||||
|
||||
- **Second-order CRDs need a third pass.** A CR whose CRD is created by a *controller at runtime*
|
||||
cannot share a Kustomization with resources of that CRD. `cdi-operator` registers exactly one
|
||||
CRD; `storageprofiles`/`datavolumes`/`datasources` appear only after it reconciles the CDI CR.
|
||||
Audit found `StorageProfile` was the only second-order kind in `platform-config`.
|
||||
- **`dns-exposure`'s teardown is ASYNCHRONOUS.** It reports PASS when chainsaw deletes the
|
||||
namespace; the Cloudflare record is reaped later by external-dns's 1-minute sync. A green run
|
||||
does **not** prove cleanup — if external-dns were suspended the record would persist silently.
|
||||
(Observed, then observed clearing.)
|
||||
- **The apex was not the only per-environment literal in the L2 suite** — two DNS tests hardcoded
|
||||
dev's LB range `192.168.1.53-.60`. They now assert the *observed* `GATEWAY_VIP` / assigned LB IP,
|
||||
which is stricter as well as portable.
|
||||
- **Velero's kopia backend has no checksum knob**, so BSL metadata and PVC-data R2 compatibility
|
||||
are **separate questions**. CNPG's boto3 1.35.60 is below the 1.36 default-checksum line — clear
|
||||
today, but **a CNPG image bump is an R2 re-verification event** (barman-cloud has no escape hatch).
|
||||
- **`ansible-inventory --list` prints DECRYPTED vault values** (the passphrase file is mounted into
|
||||
the dev container). It emitted the prod become-password to an agent's stdout. Never put it in CI.
|
||||
Confirmed no plaintext reached disk.
|
||||
Reference in new issue
Block a user