diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index d3b44b7..92e1abe 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3580,3 +3580,52 @@ decommissioned, and it is a second reason the port would not have been cheap. `main` (as `efc6db6`), with `runbooks/node-io.md` alongside them. Verified live rather than by reading the file: both rules are loaded in the `platform-health.nodes` group, `health: ok`, `state: inactive`, and `ALERTS{alertname=~"NodeIOPressure.*"}` returns no samples. Nothing to port, no test added. + +--- + +## 2026-09-13 (close) — corrections to my own claims, and what is left + +Two things I asserted during this session were wrong, and both were caught by agents doing the work +rather than by me: + +1. **"Longhorn storage alerting now works" was premature.** After the scrape fix landed I said + Longhorn alerting was live. It was *scraped* but still could not alert: + `longhorn_volume_robustness` on 1.12.1 is **one-hot over a `state` label with values 0/1**, not + the 0–3 enum `LonghornVolumeDegraded`/`LonghornVolumeFaulted` and their comment assumed. A faulted + volume would have alerted nobody. Scraping a metric and being able to alert on it are two + different claims, and I collapsed them. +2. **"velero runs a nightly volume backup" was wrong**, and it was load-bearing in a brief I wrote. + Both velero schedules use `defaultVolumesToFsBackup: true` — kopia through the node-agent, not + Longhorn volume backups — and Longhorn has **no backup target configured at all**. So + `concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing today; I had reasoned it was a + throughput tradeoff against nightly backups. Revisit it the day a backup target lands. + +A third correction, to a figure I relayed: the path→requirement fail-open is **159 of 613 tracked +paths (26%)** on today's main — 56 prose, 103 non-prose — not the 255 of 476 (54%) an older branch +claimed. Smaller than I passed on; still the largest remaining fail-open. + +**Pattern worth keeping.** Every genuine find this session came from asking "can this signal actually +reach a human?" rather than "is this configured?". The answers were: no scrape target (Longhorn, +cert-manager, Flux), no series name (the robustness enum), no discovered target (twelve SLIs), a +crashlooping exporter (kube-state-metrics), a permanently-firing rule nobody would read (the +allocation alarm), a dead link at the end of 42 annotations, and a runbook asserting a cap that was +never applied. Configuration was present in almost every case. + +### Backlog, characterised, none urgent +- `feat/egress-gateway-staleness-alert` and the two small `rescued/*` fixes (act-runner dind + readiness; resilient node DNS) — all stale trees, cherry-pick only. +- The 103 non-prose unmapped paths in `path_requirements()`. +- `apps/wave-1/keycloak.yaml` / `apps/wave-0/vm-demo.yaml`: rendered, golden-checked, applied by + nothing; the keycloak one would stand up a second conflicting instance if "fixed" by wiring it in. +- `infra/ansible`'s `etcd_io_priority` role: deliberately not ported. The real fix is §2's exit + condition — give etcd its own device. + +### Needs the owner +- **R-TLS-02**: revocation is not implementable with cert-manager. Amend the requirement, or record a + reviewed `DOCUMENTED_GAPS` entry. Both drafts are in this log. +- **Three ~1 GB files named `.exe`/`.zipx` in `/media/downloads`**, plus a dangling + `downloads -> /shared/downloads` symlink. Untouched. +- **Radarr has no root folder**, so it cannot import anything; Sonarr shows 0 series against ~100 + folders on disk. +- Deleting the orphaned kopia blobs from the two R2-tagging failures. +- The stale workstation dnsmasq override (`sudo rm` — blocks local name resolution).