diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 8c3aa3b..01b46de 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3244,3 +3244,57 @@ standing as a deliberate red.** Writing a test tagged R-TLS-02 that silently ver and renewal would report the requirement as met while a third of its normative text is unimplemented — which is the one thing this harness exists to prevent. The choice is the architect's, not an agent's. + +--- + +## 2026-09-13 (evening) — HANDOFF RESOLVED: all five WIP branches finished and merged + +The five branches parked earlier today after the session rate limit are **all complete, gated and +merged**. That handoff table is superseded; nothing from it remains pending. + +| branch | outcome | +|---|---| +| `fix/scrape-targets-for-inert-alerts` | merged — `longhorn_*` 8717 series / `longhorn-backend` up=3, `certmanager_*` 104 series / 1 target, verified through the Flux+Helm path, not a hand-apply | +| `fix/gate-does-not-pass-on-skipped-stages` | merged — PASS/0, BLOCKED/1, UNDETERMINED/3; independently re-verified both branches | +| `fix/postbuild-substitution-catches-dollar-mangling` | merged — 357 assertions; rejects the exact pre-16b7fcd manifest | +| `fix/qbittorrent-durable-webui-credential` | merged — arrs restored from `isValid: false` to `true`, survived a real pod recreation, hook logged `VERIFIED:` on a genuine postStart | +| `feat/close-c2-coverage-gaps` | merged — coverage **51/51, 0 gaps** | + +**The new guard earned its place within minutes.** `flux-postbuild-substitution` BLOCKED the very +next merge on `apps/wave-0/qbittorrent.yaml:155` — a doubled dollar inside the postStart hook's own +comment *explaining the doubled-dollar hazard*. That is the second time in one day the trap caught an +explanation of itself: the original fix (`16b7fcd`) shipped the same mistake in its comment and was +only caught by re-reading. Machine 1, humans 0. + +**Two regressions I introduced today, both found and fixed:** +- **kube-state-metrics OOMKilled 91 times.** Enabling `customResourceState` for the Velero Schedule + metrics added a CRD informer plus the mandatory CRD list/watch, crossing a 128Mi limit that already + sat ~4 MiB above a 111 MiB working set (6h peak 123.3 MiB). Thirteen `kube_*` rules were therefore + *intermittently* inert — the quiet form of the same class — with `KubePodNotReady` and + `KubeDeploymentReplicasMismatch` stuck pending on it. Raised to 384Mi (~3x peak; nodes report + 18.7–22.3 GiB free). Now 0 restarts. +- **A plaintext password in Secret metadata.** `qbittorrent-bootstrap` was hand-created with + `kubectl apply`, so its `kubectl.kubernetes.io/last-applied-configuration` annotation carried + `webui-password` in the clear. Annotations surface where `data` does not. Removed; data intact. + +**Also caught by the gate, worth remembering:** an agent left a *self-referential symlink* +(`home-network-k3s -> .`) inside the read-only spec repo while trying to make it visible to a gate +clone. Spec content was untouched (0 tracked modifications, HEAD still the pinned commit) but +`spec-immutability` correctly refused to call the tree clean and BLOCKED the merge. Agent briefs +should say how to expose the spec repo to a clone without writing into it. + +### Open, characterised, not fixed +- **`LonghornVolumeCapacityHigh` will fire permanently.** It reads + `longhorn_volume_actual_size_bytes / capacity` — ALLOCATION, not usage. Prometheus's TSDB volume + (50Gi, `retentionSize: 45GiB`) reads 0.973 while the kubelet sees the same filesystem ~20% used, + because Longhorn never returns freed blocks without discard/TRIM. By-design and unclearable, i.e. + the permanently-firing class. Delegated: split the "filesystem filling" alarm (kubelet stats) from + the "pool exhaustion" one (actual_size, at node/disk level), and assess TRIM plus whether 45GiB of + 50Gi is sane — both owner-facing. +- **Twelve `apps/*` `-down` SLI rules discover zero targets**: their ServiceMonitors select + `port: metrics` and no app Service defines that port. `PrometheusTargetsMissing` cannot see it + either — a never-discovered target has no `up` series to be 0. Delegated, including the likelihood + that several of these apps export no Prometheus metrics at all and need a different liveness signal. +- **R-TLS-02 still needs the owner** (revocation is not implementable with cert-manager). +- **`path_requirements()` leaves 159 of 613 tracked paths (26%) unmapped** — 56 prose, 103 non-prose. + Measured today; smaller than the 255/476 an older branch claimed, still the larger fail-open.