docs: the §2 shared-device measurement came back, and it does not hold

infra/PROD-PROVISIONING.md §2 recorded the etcd/Longhorn shared-partition risk as
"accepted to be MEASURED, not assumed", with an exit condition. The measurement
arrived on 2026-09-07 in production, 2 d 7 h after bring-up, as an outage.

§2 now carries "THE MEASUREMENT CAME BACK": the journal evidence, the three-node
IO PSI table (11.36 / 0.75 / 1.29), and the honest verdict that the section's own
premise — "every prod node is SSD or NVMe, where rebuild I/O costs latency rather
than collapsing it" — was right about node 2 and wrong about node 1. A 2011 C400
with 61,440 reallocated sectors and no power-loss protection is not the "SSD" that
reasoning assumed, and §0 had already named the three-way drive asymmetry as the
fault line. The exit condition is OPEN and the second bullet (dedicated device)
applies; that remains the operator's.

Also records what the three software mitigations in this branch do and do not buy,
including the point that no cgroup or scheduler control arbitrates the ext4 journal
both tenants share — and the gap that etcd's own fsync histograms, which the exit
condition literally asks for, are not collected because kubeEtcd is disabled for
k3s. PSI is the signal that exists.

DECISIONS-LOG gains the full entry: the four I/O mechanisms found INERT on these
nodes (IOSchedulingClass, IOSchedulingPriority, io.latency, bare IOWeight), the
finding that concurrentReplicaRebuildPerNodeLimit was already set and already held
during the outage, the third instance of config that looked applied and was not
(orphanAutoDeletion), and the three things still requiring the operator.

infra/ansible/README.md: CORRECTED a stale claim that k3s-prod-2/-3 "stay commented
out" of the inventory. They have been live in [k3s_prod_secondary] since
2026-09-03. That sentence is exactly the kind used to reason about blast radius —
`-e k3s_hosts=k3s_prod` targets THREE hosts, not one — so it is replaced with the
--list-hosts command that settles it. The new etcd-io-priority play is added to the
run order, the layer table and the prod command list.

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit a00a27a2567ea6a805c8c196b75df2c92a7c63a4)

infra/PROD-PROVISIONING.md

@@ -779,6 +779,74 @@ disk each, already fully partitioned, no free slot verified populated.
   physically) and flip `node_baseline_etcd_source` back to `device` — the role still
   supports that path unchanged, and it is the dev shape.
 
+#### THE MEASUREMENT CAME BACK — 2026-09-07. It does not hold on node 1.
+
+Not under a deliberate test: in production, 2 days 7 hours after the cluster came up.
+**06:05:21Z `k3s-prod-1` went NotReady** (`Kubelet stopped posting node status`) while the
+host stayed up, pingable and SSH-able. From the k3s journal:
+
+```
+slow fdatasync                took=3.56s / 6.71s / 4.95s / 3.40s   (etcd expects 1s)
+apply request took too long   took=8.30s                           (expects 100ms)
+etcdserver: too many requests
+Failed to check local etcd status for learner management: context deadline exceeded
+```
+
+No OOM, no MCE, no hung task in the kernel log — **this is not §0b and not memory**. Recovery
+required a manual `systemctl restart k3s`; 18 Longhorn volumes then rebuilt, and I/O pressure
+only fell once the rebuilds finished. The failure is **self-reinforcing**: the node dropping
+out degrades volumes → Longhorn rebuilds → more write load on the disk etcd needs → etcd
+stalls further.
+
+`/proc/pressure/io` **full avg300**, all three nodes, during the incident:
+
+| node | device | drive | IO PSI full avg300 |
+|---|---|---|---|
+| k3s-prod-1 | `/dev/sda5` | C400-MTFDDAC512MAM (2011 consumer SATA SSD, no PLP) | **11.36** |
+| k3s-prod-2 | `/dev/nvme0n1p5` | Samsung PM9A1 NVMe | 0.75 |
+| k3s-prod-3 | `/dev/sda5` | WDC WDS500G2B0A (WD Blue SATA) | 1.29 |
+
+The prediction in the paragraph above — "every prod node is SSD or NVMe end-to-end, where
+concurrent rebuild I/O costs latency rather than collapsing it" — was **right about node 2 and
+wrong about node 1**. A 2011 consumer SATA SSD with 61,440 reallocated sectors, 31% of its
+endurance consumed and no power-loss protection is not the "SSD" that reasoning assumed. §0's
+"three nodes, three different drive models" is the fault line, exactly where §0 said it would
+be.
+
+**The exit condition is therefore OPEN and the second bullet applies.** Replacing node 1's
+drive, or giving etcd its own device, is the fix, and it is the operator's.
+
+Pending that, three SOFTWARE mitigations shipped on 2026-09-07 (branch
+`fix/etcd-io-priority`). **None of them is the fix**, and each says so where it lives:
+
+1. **Observability first** — `NodeIOPressureHigh` / `NodeIOPressureCritical` in
+   `platform/monitoring/alerts.yaml`, plus an I/O row on the node-pressure dashboard. The
+   stack alerted on CPU and memory PSI and had **no I/O alert at all**, so the first signal
+   of this outage was a node already out of the cluster. Thresholds (5% / 10% full IO PSI)
+   come from the numbers in the table above. Note the etcd histograms this section's first
+   bullet asks for are **not available**: kube-prometheus-stack's `kubeEtcd` scrape job is
+   deliberately disabled for k3s (`platform/monitoring/kube-prometheus-stack.yaml`), so PSI
+   is the signal that actually exists. Enabling the etcd scrape would make that bullet
+   directly measurable and is worth doing.
+2. **Longhorn concurrency ceilings** — `platform/longhorn/helmrelease.yaml`. Honestly:
+   `concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during this outage** and
+   held; the 18 rebuilds ran one at a time and the node stalled anyway. Longhorn offers no
+   further rebuild lever for the v1 data engine (`replicaRebuildingBandwidthLimit` is
+   v2/SPDK only; `rebuildConcurrentSyncLimit` is already at its floor of 1). What was
+   tightened is every *other* Longhorn I/O source — restore, snapshot purge, backup, engine
+   upgrade — each of which defaulted to 5 concurrent operations per node and each of which
+   could reproduce this on its own.
+3. **Block-layer arbitration** — `roles/etcd_io_priority`, enabled for `[k3s_prod]`, applied
+   only when the operator runs `playbooks/etcd-io-priority.yml`. BFQ + `IOWeight` so etcd's
+   cgroup outweighs `kubepods.slice`. Its README is explicit that most of the obvious knobs
+   here (`IOSchedulingClass`, `io.latency`, bare `IOWeight`) are **inert on these nodes**,
+   and that even this one cannot arbitrate the ext4 journal both tenants share.
+
+The last point is the one that matters for this section: **no cgroup or scheduler control
+separates two tenants on one filesystem.** etcd's `fdatasync` waits on a jbd2 journal commit,
+jbd2 is a kernel thread in the root cgroup, and Longhorn's writes dirty the same journal. The
+software mitigations raise the load level at which this happens. They do not remove it.
+
 **~~Known, separate consequence — `/var` is only 12 GiB.~~ RESOLVED 2026-09-04 by moving
 k3s's data dir.** This was recorded here on 2026-09-03 as a capacity item "to watch or
 repartition". Re-measuring it against dev turned it from a watch item into a blocker: the

infra/ansible/README.md

@@ -27,6 +27,7 @@ ansible-playbook playbooks/site.yml        # core bring-up, in order (baseline
 # — or step by step —
 ansible-playbook playbooks/node-baseline.yml    # 0. packages, kernel modules, Longhorn disk, swap-off
 ansible-playbook playbooks/k3s.yml         # 1. HA k3s (primary --cluster-init, then joins)
+ansible-playbook playbooks/etcd-io-priority.yml # 1b. shared etcd/Longhorn device I/O weights (PROD ONLY; a no-op on dev)
 ansible-playbook playbooks/cilium.yml      # 2. Cilium CNI (cluster is NotReady until this)
 ansible-playbook playbooks/verify.yml      # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
 ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOPS then Flux)
@@ -71,6 +72,7 @@ should return a PIA exit IP (not the home ISP).
 |---|---|
 | `k3s.yml` / `k3s_server` | k3s server install with `--flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb`; primary `--cluster-init`, secondaries join via the primary's node-token (fetched, never pre-stored) |
 | `cilium.yml` / `cilium` | Gateway API CRDs, then Helm `cilium` with `kubeProxyReplacement`, `k8sServiceHost/Port`, `gatewayAPI`, `egressGateway`, `hubble`, `l2announcements` |
+| `etcd-io-priority.yml` / `etcd_io_priority` | **prod only** (master switch defaults false; `group_vars/k3s_prod` sets it true). Switches the disk carrying `k3s_data_dir` to **BFQ**, drops `nr_requests` to 32, and gives `system.slice` + `k3s.service` `IOWeight=10000` so the embedded etcd outweighs `kubepods.slice` — then reads every value back out of the kernel and **fails** if it did not land. Mitigation for the 2026-09-07 k3s-prod-1 etcd WAL-fsync starvation. Restarts nothing. NVMe-backed nodes are skipped, not decorated. See `roles/etcd_io_priority/README.md` for the mechanisms in this space that are **inert** here and must not be "fixed" back in |
 | `verify.yml` / `cluster_verify` | waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
 | `flux-bootstrap.yml` / `flux_bootstrap` | calls `bootstrap/` (SOPS age key, then `flux bootstrap --path=clusters/dev`) |
 
@@ -120,10 +122,20 @@ for PM reconciliation.
 ## Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)
 
 `inventory/hosts.ini` carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
-`[k3s_prod:children]` section for the three planned bare-metal nodes (`k3s-prod-1` at
-`192.168.2.10` exists and is reachable today; `k3s-prod-2`/`-3` at `.11`/`.12` are addressed but
-not yet racked, so their inventory lines stay commented out), with its own `group_vars/k3s_prod/`
-(vaulted `ansible_become_password` + non-secret overrides).
+`[k3s_prod:children]` section for the three bare-metal nodes — `k3s-prod-1` `192.168.2.10`,
+`k3s-prod-2` `.11`, `k3s-prod-3` `.12` — with its own `group_vars/k3s_prod/` (vaulted
+`ansible_become_password` + non-secret overrides).
+
+**All three are live in the inventory.** This paragraph used to say `k3s-prod-2`/`-3` "are
+addressed but not yet racked, so their inventory lines stay commented out"; that stopped being
+true on 2026-09-03 when node 3 came online, and the group has been complete since. Corrected
+2026-09-07 because the stale sentence is the kind that gets used to reason about blast radius:
+`-e k3s_hosts=k3s_prod` targets **three** hosts, not one. Confirm it, every time, before running
+anything:
+
+```bash
+ansible-playbook playbooks/<play>.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3 --list-hosts
+```
 
 **Nothing here is forked for prod.** Every node-host play takes its target group from a variable
 that defaults to the dev group, so the dev commands above are unchanged, and prod is the same
@@ -153,6 +165,11 @@ ansible-playbook playbooks/kube-vip.yml        -e k3s_hosts=k3s_prod
 ansible-playbook playbooks/verify.yml          -e k3s_hosts=k3s_prod
 ansible-playbook playbooks/flux-bootstrap.yml  -e k3s_hosts=k3s_prod   # see caveat below
 
+# prod — I/O arbitration for the shared etcd/Longhorn disk. ONE NODE AT A TIME, node 3 first
+# (WD Blue, IO PSI 1.29 during the incident) before node 1 (the 2011 C400, 11.36).
+ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3 --list-hosts
+ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3
+
 # prod — k3s.yml needs both halves of the primary/secondary split
 ansible-playbook playbooks/k3s.yml \
   -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary