docs: DECISIONS-LOG — prod solidified, four cold-start bugs of one shape

Node 2 at full 32GB; storage recovered 17/17 after the cordon that was silently
blocking Longhorn rebuilds; Kyverno/CDI unblocked by satisfying the policy rather
than exempting it; conformance 82.4%->88.2%. Names the recurring class — a thing
that looks applied and isn't — and lists what is staged for one push versus what
waits on the operator.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -1936,3 +1936,101 @@ a test for the coupling is **not** yet written and should be.
 - **`ansible-inventory --list` prints DECRYPTED vault values** (the passphrase file is mounted into
   the dev container). It emitted the prod become-password to an agent's stdout. Never put it in CI.
   Confirmed no plaintext reached disk.
+
+## 2026-09-06 — Prod solidified. Node 2 at full 32 GB, storage recovered, four latent cold-start bugs closed
+
+Operator away; PM directed to oversee, delegate, and pay down debt while skipping anything needing
+their input. Five agents ran; every result re-verified by the PM before commit. **Nothing merged,
+nothing pushed. Both clusters healthy.**
+
+Commits: `04569ab` node-maintenance runbook · `da3e9cc` Kyverno/CDI · `a984442` timezone + guard test.
+
+### PROD IS FUNCTIONING
+
+| | |
+|---|---|
+| Nodes | 3/3 Ready, **all three at ~`32638256Ki`** — node 2's DIMM upgrade landed, matching 1 and 3 to within 4 KiB |
+| Flux | **12/13 tiers Ready** (`velero` suspended: no backup endpoint) |
+| Pods | **112 Running**, 0 Running-but-NotReady |
+| Storage | **17/17 volumes healthy**, replicas symmetric 17-running+1-stopped per node |
+| TLS | LE wildcard `*.prod.home.keithjackson.dev` Ready |
+| etcd | 3 members, survived an ungraceful power-down without losing a member |
+
+The memattr saga (§0b) is **closed on the hardware side**: with the 7.1.8 backports kernel, all three
+nodes address their full 32 GB.
+
+### THE FINDING THAT MATTERS MOST — a failed drain silently stops storage self-healing
+
+`kubectl drain` **cordons before it fails**, and Longhorn's `disable-scheduling-on-cordoned-node` is
+`true`. After prod-2's memory-upgrade reboot, **all 17 volumes sat `attached/degraded` with prod-2
+holding 2 of 18 replicas and ZERO rebuilds in flight** — nothing driving recovery, every pod
+`Running`, no alert. `kubectl uncordon` started rebuilds immediately; converged 17/17 in ~6 minutes.
+
+**Uncordon is not tidying up — it is what re-enables redundancy.** Full runbook + post-maintenance
+checklist in `infra/PROD-STANDUP.md` §Node maintenance, including why `drain` can never complete here
+(instance-manager PDBs at 0 allowed disruptions; at 3 nodes/3 replicas replica eviction has nowhere to
+go, which is structural, not a setting).
+
+### FOUR BUGS OF ONE SHAPE — *a thing that looks applied and isn't*
+
+Worth naming as a class, because it has now appeared four times in two days:
+
+1. **Kyverno's own CRDs** — `crds.install: false`, nothing else installed them; pods crash-looped 55×.
+2. **CDI StorageProfiles** — CR and its dependent resource in the same Flux pass; the dry-run aborts
+   the tier, so the CR never applies and its CRD never appears. *Repaired in the graph* (`765ad03`).
+3. **The placeholder guard** — a `TBD-*` prefix glob stopped matching when the value it guards changed
+   *in a different file*; `resume` would have enabled velero against a nonexistent host. Now tested
+   (`a984442`).
+4. **PolicyException** — the CRD is installed but every controller runs
+   `--enablePolicyException=false`. An exception would apply cleanly through Flux and be **silently
+   ignored**. Verified live on prod.
+
+Two more of the same family: a coverage check that passed when perturbed (asked a question always
+satisfiable), and `ci/touched-requirements.sh` having **no `bootstrap/*` entry**, which scoped the L0
+run away from the very check guarding the placeholder class. Both fixed.
+
+**The rule:** dev's age hides cold-start defects. Anything that only runs on a fresh cluster has never
+been exercised on dev, and prod is the first honest test. And: *a guard keyed on the shape of a value
+that lives in another file needs a test tying the two together.*
+
+### Kyverno blocking CDI — fixed by SATISFYING the policy, not exempting it
+
+Prod's UniFi VM had no disk: Kyverno denied CDI's generated `importer-*` pod for missing the
+`auth-mode` / `egress-class` labels, which CDI cannot set. Fixed with a **mutate** policy that adds
+them when the requester is ServiceAccount `cdi/cdi-sa` — identity from AdmissionReview userInfo, **not
+a name prefix** (a prefix is attacker-choosable; the test proves it with a forged
+`importer-prime-forged-by-an-app` fixture that is still denied). Two-way proof: same pod, different
+requester, opposite outcomes. Kyverno L0 17 → 29 tests.
+
+Rejected `exclude` scoped to the same SA because any userInfo match forces `background: false`, which
+would drop background scanning of R-NET-05/R-IDP-02 for **all existing pods** to fix one pod class.
+
+### Conformance 82.4% → 88.2%
+
+45/51 scaffolded, 0 safety-critical gaps. Six L2 tests made environment-agnostic (they derive
+`BASE_DOMAIN`/`GATEWAY_VIP` from the cluster they run against, so they cannot drift); R-TLS-02,
+R-ENV-01 and R-ENV-03 covered — the last two matter because the Phase-7 entry-gate note recorded
+R-ENV-01/02/03 as uncovered while prod provisioning was proceeding.
+
+### STAGED FOR ONE PUSH — `prod/deliver-fixes`
+
+Prod reconciles bootstrap-Gitea `main@c8c1d29` and **cannot take the working branch** — that carries
+the DNS inversion, which would move prod to the apex and collide with dev. `prod/deliver-fixes` is
+`c8c1d29` + the CDI graph fix + the Kyverno mutate policy: a clean fast-forward, verified free of any
+cluster-vars/externaldns/overlay change, prod's `BASE_DOMAIN` untouched, render checked. **Not pushed
+— that changes what a live cluster reconciles.**
+
+### Waiting on the operator (deliberately skipped)
+
+- **R2 credentials.** Two token values; the account id is already committed in
+  `infra/tofu/backend.r2.hcl`. `infra/BACKUP-R2.md` is the procedure. **Prod has no backup coverage.**
+- **`enableCertificateOwnerRef`** — R-TLS-02's "revoked" clause is unmet; both clusters' TLS Secrets
+  have zero ownerReferences. Not flipped because the wildcard Certificate lives under a `prune: true`
+  tier, so a transient prune would delete the serving Secret and reissuance is capped at 5/week.
+- **Node timezone** — authored (`Etc/UTC`, matching dev), `--check` confirms the drift, **not applied**:
+  a live clock change during storage resync is not an unattended action.
+- **Database HA** — all three CNPG clusters are `instances: 1` and all three primaries are on prod-1.
+  Losing that node takes Keycloak, Nextcloud and Semaphore together. `instances: 2` spends resources.
+- **The apex flip** is also the legacy-prod 443 cutover; §10a sequence P does not cover the SOHO NAT.
+- **R-STATE-04** stays a gap: `bulk` is a non-provisioning stub pending **OQ-2**.
+- Orphaned `prime-be3f9d1c-…` 40 GiB scratch PVC from the blocked imports — operator cleanup.