From 5ec14605326d7660249b86260094fa28f21295ad Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 6 Sep 2026 18:51:00 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20DECISIONS-LOG=20=E2=80=94=20prod=20soli?= =?UTF-8?q?dified,=20four=20cold-start=20bugs=20of=20one=20shape?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Node 2 at full 32GB; storage recovered 17/17 after the cordon that was silently blocking Longhorn rebuilds; Kyverno/CDI unblocked by satisfying the policy rather than exempting it; conformance 82.4%->88.2%. Names the recurring class — a thing that looks applied and isn't — and lists what is staged for one push versus what waits on the operator. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 98 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index f5c5982..20ae8db 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1936,3 +1936,101 @@ a test for the coupling is **not** yet written and should be. - **`ansible-inventory --list` prints DECRYPTED vault values** (the passphrase file is mounted into the dev container). It emitted the prod become-password to an agent's stdout. Never put it in CI. Confirmed no plaintext reached disk. + +## 2026-09-06 — Prod solidified. Node 2 at full 32 GB, storage recovered, four latent cold-start bugs closed + +Operator away; PM directed to oversee, delegate, and pay down debt while skipping anything needing +their input. Five agents ran; every result re-verified by the PM before commit. **Nothing merged, +nothing pushed. Both clusters healthy.** + +Commits: `04569ab` node-maintenance runbook · `da3e9cc` Kyverno/CDI · `a984442` timezone + guard test. + +### PROD IS FUNCTIONING + +| | | +|---|---| +| Nodes | 3/3 Ready, **all three at ~`32638256Ki`** — node 2's DIMM upgrade landed, matching 1 and 3 to within 4 KiB | +| Flux | **12/13 tiers Ready** (`velero` suspended: no backup endpoint) | +| Pods | **112 Running**, 0 Running-but-NotReady | +| Storage | **17/17 volumes healthy**, replicas symmetric 17-running+1-stopped per node | +| TLS | LE wildcard `*.prod.home.keithjackson.dev` Ready | +| etcd | 3 members, survived an ungraceful power-down without losing a member | + +The memattr saga (§0b) is **closed on the hardware side**: with the 7.1.8 backports kernel, all three +nodes address their full 32 GB. + +### THE FINDING THAT MATTERS MOST — a failed drain silently stops storage self-healing + +`kubectl drain` **cordons before it fails**, and Longhorn's `disable-scheduling-on-cordoned-node` is +`true`. After prod-2's memory-upgrade reboot, **all 17 volumes sat `attached/degraded` with prod-2 +holding 2 of 18 replicas and ZERO rebuilds in flight** — nothing driving recovery, every pod +`Running`, no alert. `kubectl uncordon` started rebuilds immediately; converged 17/17 in ~6 minutes. + +**Uncordon is not tidying up — it is what re-enables redundancy.** Full runbook + post-maintenance +checklist in `infra/PROD-STANDUP.md` §Node maintenance, including why `drain` can never complete here +(instance-manager PDBs at 0 allowed disruptions; at 3 nodes/3 replicas replica eviction has nowhere to +go, which is structural, not a setting). + +### FOUR BUGS OF ONE SHAPE — *a thing that looks applied and isn't* + +Worth naming as a class, because it has now appeared four times in two days: + +1. **Kyverno's own CRDs** — `crds.install: false`, nothing else installed them; pods crash-looped 55×. +2. **CDI StorageProfiles** — CR and its dependent resource in the same Flux pass; the dry-run aborts + the tier, so the CR never applies and its CRD never appears. *Repaired in the graph* (`765ad03`). +3. **The placeholder guard** — a `TBD-*` prefix glob stopped matching when the value it guards changed + *in a different file*; `resume` would have enabled velero against a nonexistent host. Now tested + (`a984442`). +4. **PolicyException** — the CRD is installed but every controller runs + `--enablePolicyException=false`. An exception would apply cleanly through Flux and be **silently + ignored**. Verified live on prod. + +Two more of the same family: a coverage check that passed when perturbed (asked a question always +satisfiable), and `ci/touched-requirements.sh` having **no `bootstrap/*` entry**, which scoped the L0 +run away from the very check guarding the placeholder class. Both fixed. + +**The rule:** dev's age hides cold-start defects. Anything that only runs on a fresh cluster has never +been exercised on dev, and prod is the first honest test. And: *a guard keyed on the shape of a value +that lives in another file needs a test tying the two together.* + +### Kyverno blocking CDI — fixed by SATISFYING the policy, not exempting it + +Prod's UniFi VM had no disk: Kyverno denied CDI's generated `importer-*` pod for missing the +`auth-mode` / `egress-class` labels, which CDI cannot set. Fixed with a **mutate** policy that adds +them when the requester is ServiceAccount `cdi/cdi-sa` — identity from AdmissionReview userInfo, **not +a name prefix** (a prefix is attacker-choosable; the test proves it with a forged +`importer-prime-forged-by-an-app` fixture that is still denied). Two-way proof: same pod, different +requester, opposite outcomes. Kyverno L0 17 → 29 tests. + +Rejected `exclude` scoped to the same SA because any userInfo match forces `background: false`, which +would drop background scanning of R-NET-05/R-IDP-02 for **all existing pods** to fix one pod class. + +### Conformance 82.4% → 88.2% + +45/51 scaffolded, 0 safety-critical gaps. Six L2 tests made environment-agnostic (they derive +`BASE_DOMAIN`/`GATEWAY_VIP` from the cluster they run against, so they cannot drift); R-TLS-02, +R-ENV-01 and R-ENV-03 covered — the last two matter because the Phase-7 entry-gate note recorded +R-ENV-01/02/03 as uncovered while prod provisioning was proceeding. + +### STAGED FOR ONE PUSH — `prod/deliver-fixes` + +Prod reconciles bootstrap-Gitea `main@c8c1d29` and **cannot take the working branch** — that carries +the DNS inversion, which would move prod to the apex and collide with dev. `prod/deliver-fixes` is +`c8c1d29` + the CDI graph fix + the Kyverno mutate policy: a clean fast-forward, verified free of any +cluster-vars/externaldns/overlay change, prod's `BASE_DOMAIN` untouched, render checked. **Not pushed +— that changes what a live cluster reconciles.** + +### Waiting on the operator (deliberately skipped) + +- **R2 credentials.** Two token values; the account id is already committed in + `infra/tofu/backend.r2.hcl`. `infra/BACKUP-R2.md` is the procedure. **Prod has no backup coverage.** +- **`enableCertificateOwnerRef`** — R-TLS-02's "revoked" clause is unmet; both clusters' TLS Secrets + have zero ownerReferences. Not flipped because the wildcard Certificate lives under a `prune: true` + tier, so a transient prune would delete the serving Secret and reissuance is capped at 5/week. +- **Node timezone** — authored (`Etc/UTC`, matching dev), `--check` confirms the drift, **not applied**: + a live clock change during storage resync is not an unattended action. +- **Database HA** — all three CNPG clusters are `instances: 1` and all three primaries are on prod-1. + Losing that node takes Keycloak, Nextcloud and Semaphore together. `instances: 2` spends resources. +- **The apex flip** is also the legacy-prod 443 cutover; §10a sequence P does not cover the SOHO NAT. +- **R-STATE-04** stays a gap: `bulk` is a non-provisioning stub pending **OQ-2**. +- Orphaned `prime-be3f9d1c-…` 40 GiB scratch PVC from the blocked imports — operator cleanup.