docs: the §2 shared-device measurement came back, and it does not hold

infra/PROD-PROVISIONING.md §2 recorded the etcd/Longhorn shared-partition risk as
"accepted to be MEASURED, not assumed", with an exit condition. The measurement
arrived on 2026-09-07 in production, 2 d 7 h after bring-up, as an outage.

§2 now carries "THE MEASUREMENT CAME BACK": the journal evidence, the three-node
IO PSI table (11.36 / 0.75 / 1.29), and the honest verdict that the section's own
premise — "every prod node is SSD or NVMe, where rebuild I/O costs latency rather
than collapsing it" — was right about node 2 and wrong about node 1. A 2011 C400
with 61,440 reallocated sectors and no power-loss protection is not the "SSD" that
reasoning assumed, and §0 had already named the three-way drive asymmetry as the
fault line. The exit condition is OPEN and the second bullet (dedicated device)
applies; that remains the operator's.

Also records what the three software mitigations in this branch do and do not buy,
including the point that no cgroup or scheduler control arbitrates the ext4 journal
both tenants share — and the gap that etcd's own fsync histograms, which the exit
condition literally asks for, are not collected because kubeEtcd is disabled for
k3s. PSI is the signal that exists.

DECISIONS-LOG gains the full entry: the four I/O mechanisms found INERT on these
nodes (IOSchedulingClass, IOSchedulingPriority, io.latency, bare IOWeight), the
finding that concurrentReplicaRebuildPerNodeLimit was already set and already held
during the outage, the third instance of config that looked applied and was not
(orphanAutoDeletion), and the three things still requiring the operator.

infra/ansible/README.md: CORRECTED a stale claim that k3s-prod-2/-3 "stay commented
out" of the inventory. They have been live in [k3s_prod_secondary] since
2026-09-03. That sentence is exactly the kind used to reason about blast radius —
`-e k3s_hosts=k3s_prod` targets THREE hosts, not one — so it is replaced with the
--list-hosts command that settles it. The new etcd-io-priority play is added to the
run order, the layer table and the prod command list.

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit a00a27a2567ea6a805c8c196b75df2c92a7c63a4)

DECISIONS-LOG.md

@@ -1847,3 +1847,110 @@ down in `platform/headlamp/README.md`: moving one without the other 401s every r
 `headlamp-oidc-credentials.sops.yaml.example` used to *teach* the broken pattern — it listed
 `OIDC_ISSUER_URL` and `OIDC_CALLBACK_URL` as keys to fill in. It now says, in the file, that adding
 either key back re-breaks prod.
+## 2026-09-07 — k3s-prod-1 NotReady: etcd starved of disk I/O by Longhorn (branch `fix/etcd-io-priority`)
+
+**06:05:21Z, `k3s-prod-1` went NotReady** (`Kubelet stopped posting node status`) with the host
+up, pingable and SSH-able. The embedded etcd could not complete a WAL `fdatasync`:
+
+```
+slow fdatasync                took=3.56s / 6.71s / 4.95s / 3.40s   (etcd expects 1s)
+apply request took too long   took=8.30s                           (expects 100ms)
+etcdserver: too many requests
+Failed to check local etcd status for learner management: context deadline exceeded
+```
+
+No OOM, no MCE, no hung task — **this is not the §0b memattr bug and not memory**. It is the §2
+shared-device risk, which §2 recorded as "accepted to be MEASURED, not assumed". The measurement
+came back, in production, 2 d 7 h after bring-up. `/proc/pressure/io` **full avg300**: node 1
+**11.36**, node 2 (NVMe) 0.75, node 3 (WD Blue SATA) 1.29. Node 1's drive is a 2011
+`C400-MTFDDAC512MAM` with 61,440 reallocated sectors and no power-loss protection. §2's premise —
+"every prod node is SSD or NVMe, where rebuild I/O costs latency rather than collapsing it" — was
+right about node 2 and wrong about node 1. §2 now records this under "THE MEASUREMENT CAME BACK".
+
+**The exit condition is OPEN and the fix is hardware: replace node 1's drive, or give etcd its own
+device (`node_baseline_etcd_source: device`, the path the role still supports). That is the
+operator's.** Three software mitigations shipped meanwhile, none of which is the fix.
+
+### 1. The outage was invisible to the monitoring stack — fixed first
+
+There were alerts for **CPU** PSI and memory and **none for I/O**, so the first signal was a node
+already out of the cluster. Added `NodeIOPressureHigh` (full IO PSI > 5% for 10m) and
+`NodeIOPressureCritical` (> 10% for 5m), thresholds taken from the numbers above against a ~1%
+steady state, plus an I/O row on the node-pressure dashboard. Verified live that node-exporter
+already exports `node_pressure_io_{stalled,waiting}_seconds_total` on all three nodes.
+
+**Gap left open for the operator:** etcd's own `etcd_disk_wal_fsync_duration_seconds` — the metric
+§2's exit condition literally asks for — is **not collected**. `kubeEtcd` is deliberately disabled
+in `platform/monitoring/kube-prometheus-stack.yaml` for k3s. Enabling it (k3s
+`--etcd-expose-metrics` + a scrape target) would make §2 directly measurable instead of inferred
+from PSI. Not done here: it is a k3s server-flag change requiring a rolling restart, which is not
+something to bundle into an incident fix.
+
+### 2. Longhorn was already throttled as far as Longhorn goes
+
+`concurrentReplicaRebuildPerNodeLimit: 1` was **in force during the outage and it held** — the 18
+volumes rebuilt one at a time and the node stalled anyway. Recorded plainly because it would be
+easy to "fix" this incident by setting a knob that was already set. For the v1 data engine there is
+no further rebuild lever: `replicaRebuildingBandwidthLimit` is v2/SPDK only (live Setting reads
+`{"v2":"0"}`, no v1 key) and `rebuildConcurrentSyncLimit` is already at its floor of 1.
+
+So what was tightened is every *other* Longhorn I/O source, each defaulting to 5 concurrent
+operations per node: `concurrentVolumeBackupRestorePerNodeLimit` 5→1 (an R-STATE-03 restore of 18
+volumes would otherwise run five parallel full-volume writes onto the disk etcd fsyncs to),
+`snapshotHeavyTaskConcurrentLimit` 5→1, plus pins on `replicaReplenishmentWaitInterval`,
+`backup`/`restoreConcurrentLimit` and `concurrentAutomaticEngineUpgradePerNodeLimit`.
+
+**And one setting in that file was inert.** `orphanAutoDeletion: true` has been silently dropped
+since the 1.11→1.12 upgrade — Longhorn 1.9 renamed it `orphanResourceAutoDeletion` (a list, not a
+bool) and the chart drops unknown `defaultSettings` keys with no error. Confirmed from both ends:
+absent from chart v1.12.1's `templates/default-setting.yaml`, and absent from the rendered
+`longhorn-default-setting` ConfigMap in the live cluster. Removed, deliberately not replaced. That
+is the **third** time this repo has shipped config that looked applied and was not.
+
+### 3. Block-layer arbitration — and four mechanisms that are inert on these nodes
+
+`roles/etcd_io_priority` + `playbooks/etcd-io-priority.yml`: BFQ on the shared disk, `nr_requests`
+64→32, `IOWeight=10000` on `system.slice` **and** `k3s.service`, then every value read back out of
+the kernel and asserted. Enabled for `[k3s_prod]`; **applies nothing until the operator runs it**.
+
+The investigation's main value is the negative results, all checked on the live nodes:
+
+- **`IOSchedulingClass=realtime` on `k3s.service` is a footgun, not a fix.** `mq-deadline` does
+  honour the priority class, so it looks correct — but ioprio is inherited across `fork` and kept
+  across `exec` (`block/blk-ioc.c` `__copy_io` copies `ioc->ioprio`), and **containerd is a child
+  of the k3s server process** (verified: containerd pid 84553, ppid 84333 = `k3s-server`; and
+  `ionice -c2 -n0 -- sh -c 'ionice -p $$; sh -c "ionice -p \$\$"'` reports best-effort prio 0 at
+  both levels). Realtime for k3s would be realtime for every container including Longhorn.
+- **`IOSchedulingPriority` (level within a class) is a no-op** — `mq-deadline` buckets by class only.
+- **`io.latency` is unavailable** — `CONFIG_BLK_CGROUP_IOLATENCY` is not built in Debian trixie's
+  7.1.8 kernel (only `IOCOST` is), and no `io.latency` exists under `/sys/fs/cgroup`.
+- **A bare `IOWeight=` drop-in is decoration.** systemd writes it to `io.weight` (blk-iocost —
+  compiled in but disabled; `io.cost.qos` is empty) and `io.bfq.weight` (BFQ — not loaded). Neither
+  reader exists on the stock nodes. And `IOWeight=1000` would map to `io.bfq.weight` ≈ 101 against
+  BFQ's unweighted default of 100 — hence 10000, and an asserted floor of 500.
+
+**What none of it fixes, stated so it is not forgotten:** etcd and Longhorn still share one ext4
+filesystem, and no cgroup or scheduler control arbitrates the **jbd2 journal** that etcd's
+`fdatasync` waits on and Longhorn's writes dirty. Software raises the load level at which this
+happens; it does not remove it.
+
+**Unproven.** The role fails loudly if its settings do not *land*; it cannot tell you they helped.
+Apply to `k3s-prod-3` first (WD Blue, PSI 1.29 — same code path, far lower stakes), watch
+`NodeIOPressureHigh` for a day, then node 1.
+
+### Doc correction found on the way
+
+`infra/ansible/README.md` still claimed `k3s-prod-2`/`-3` "stay commented out" of the inventory.
+They have been live in `[k3s_prod_secondary]` since 2026-09-03. Corrected, because that sentence is
+exactly the kind used to reason about blast radius: **`-e k3s_hosts=k3s_prod` targets three hosts,
+not one.** Confirmed by `--list-hosts` (dev default → 3 dev hosts, where the new role is a no-op;
+`-e k3s_hosts=k3s_prod` → 3 prod hosts; `--limit k3s-prod-3` → 1).
+
+### Still requires the operator
+
+1. **The hardware fix.** Replace node 1's C400, or add an M.2 per node and set
+   `node_baseline_etcd_source: device`. Everything above is a delay, not a repair.
+2. **Run `playbooks/etcd-io-priority.yml`** — node 3 first, `--limit`, `--list-hosts` confirmed.
+   Nothing was run against either cluster for this change.
+3. **Decide on `kubeEtcd` metrics**, so §2's exit condition can be measured with etcd's own fsync
+   histograms rather than inferred from node PSI.