home-network-k3s-impl/infra/ansible/roles/etcd_io_priority

roles/etcd_io_priority

Give the k3s server process — and therefore the embedded etcd it hosts — a larger share of the block device it shares with Longhorn.

The incident

2026-09-07 06:05:21Z, k3s-prod-1 went NotReady (Kubelet stopped posting node status) while the host stayed up, pingable and SSH-able. From the k3s journal:

slow fdatasync                took=3.56s / 6.71s / 4.95s / 3.40s   (etcd expects 1s)
apply request took too long   took=8.30s                           (expects 100ms)
etcdserver: too many requests
Failed to check local etcd status for learner management: context deadline exceeded

No OOM, no MCE, no hung task. All three prod nodes put /data/k3s (etcd's WAL and the containerd image store) and /data/longhorn (every replica) in two directories on one partition:

node	device	drive	/proc/pressure/io full avg300
k3s-prod-1	/dev/sda5	C400-MTFDDAC512MAM — 2011 consumer SATA SSD, no PLP, 61,440 reallocated sectors	11.36
k3s-prod-2	/dev/nvme0n1p5	Samsung PM9A1 NVMe	0.75
k3s-prod-3	/dev/sda5	WDC WDS500G2B0A (WD Blue SATA)	1.29

Recovery took a manual systemctl restart k3s; 18 Longhorn volumes then rebuilt onto the same disk etcd was trying to fsync on. The failure is self-reinforcing.

What this role does

Two things, both instantly reversible, neither requiring a k3s restart:

1. k3s-etcd-io-priority.service (oneshot, ordered Before=k3s.service) switches the shared device to the BFQ I/O scheduler and lowers nr_requests from 64 to 32.
2. IOWeight=10000 drop-ins on system.slice and k3s.service, which BFQ turns into io.bfq.weight=1000 — a 10:1 share against the default-weighted kubepods.slice where Longhorn's instance-managers live.

Then it reads every one of those values back out of the kernel and asserts on them. If the scheduler did not switch, or the weight did not land, the play fails. A mitigation that looks applied and is not is worse than none.

Why not the obvious things — all checked on the live nodes, all inert

IOSchedulingClass=realtime on k3s.service. The scheduler here (mq-deadline, and BFQ too) does honour the I/O priority class, so this looks like the answer. It is not: I/O priority is inherited across fork and preserved across exec — block/blk-ioc.c's __copy_io() copies ioc->ioprio into the child — and containerd is a child of the k3s server process. Verified on k3s-prod-1: ps -eo pid,ppid,comm shows containerd's PPID is the k3s-server PID (84553 → 84333), and

$ ionice -c2 -n0 -- sh -c 'ionice -p $$; sh -c "ionice -p \$\$"'
best-effort: prio 0
best-effort: prio 0

confirms inheritance through two levels of fork+exec on that node. Putting k3s in the realtime class therefore puts every container on the node in the realtime class, Longhorn included — no relative gain, and it demotes the rest of system.slice (sshd, journald) for nothing. Getting this to work would need containerd out of k3s's process tree, i.e. an external containerd via --container-runtime-endpoint. That is an architecture change, not a tweak.

IOSchedulingPriority (the level within a class). mq-deadline buckets requests by priority class only — IOPRIO_CLASS_RT/BE/IDLE map to three dd_per_prio queues and the level inside a class is not read. So best-effort 0 versus best-effort 4 is a no-op here.

io.latency (blk-iolatency). The ideal mechanism for this problem — scheduler-independent, protects one cgroup's latency target by throttling its siblings. Not available: Debian trixie's kernel does not build it. grep CONFIG_BLK_CGROUP /boot/config-7.1.8+deb13-amd64 returns CONFIG_BLK_CGROUP_IOCOST=y and no IOLATENCY, and there is no io.latency file anywhere under /sys/fs/cgroup.

IOWeight= on the stock configuration. systemd writes IOWeight= to both io.weight (blk-iocost) and io.bfq.weight (BFQ). On these nodes as shipped, neither reader exists: the scheduler is mq-deadline, which has no cgroup awareness, and blk-iocost is compiled in but disabled (/sys/fs/cgroup/io.cost.qos is empty). Adding a drop-in without switching the scheduler would be pure decoration. That is why the scheduler change and the weights are one role and one verification, not two independent knobs.

IOWeight=1000 instead of 10000. systemd rescales IOWeight's 1..10000 range onto io.bfq.weight's 1..1000, so 1000 maps to roughly 101 — BFQ's default for an unweighted cgroup is 100. Only the top of the range separates anything. The role asserts the resulting io.bfq.weight is ≥ etcd_io_priority_min_bfq_weight (500) rather than trusting that mapping.

A udev rule instead of a unit. ATTR{queue/scheduler}="bfq" on the boot disk fires its add event in the initramfs, where /etc/udev/rules.d is absent and bfq.ko is not loaded — it silently does nothing and the disk comes up on mq-deadline with no error anywhere.

A drop-in on kubepods.slice (to lower Longhorn instead of raising etcd). kubepods.slice is a transient unit the kubelet creates through the systemd API (systemctl cat kubepods.slice shows /run/systemd/transient/kubepods.slice), and whether systemd merges an /etc drop-in into a transient unit was not verified on these nodes. Raising system.slice gets the same ratio through a unit systemd definitely owns.

What this role does NOT fix

• It does not separate etcd from Longhorn. Both still write to one ext4 filesystem on one partition. Weights arbitrate the device queue; they cannot arbitrate the shared ext4 journal. fdatasync on the WAL waits on a jbd2 commit, jbd2 is a kernel thread in the root cgroup, and Longhorn's replica writes dirty the same journal. That serialization point is invisible to every cgroup control in this file and is a hard limit of co-tenancy on one filesystem.
• It is unproven on this hardware. BFQ is the right mechanism on paper and it is what systemd's IOWeight was built for, but nothing here has been measured on a C400 under a real Longhorn rebuild. The role fails loudly if the settings do not land; it cannot tell you they helped. Apply to k3s-prod-3 first and watch the NodeIOPressureHigh alert and the "Nodes — CPU / Memory / I-O Pressure" dashboard before touching node 1.
• BFQ has costs. It disables writeback throttling (wbt_lat_usec is 2000 on sda today), spends more CPU per request, and gives up some peak throughput. On a device that is already the bottleneck this is the right trade; it is still a trade.
• k3s-prod-2 gets nothing. Its data dir is on NVMe, BFQ is a poor fit for NVMe, and its IO PSI was 0.75 during the incident. The role detects this and installs nothing rather than leaving inert files behind.
• The real fix is hardware and is the operator's: replace the C400, or give etcd its own device and flip node_baseline_etcd_source back to device — the path roles/node_baseline still supports unchanged, and the exit condition already written into infra/PROD-PROVISIONING.md §2.

Running it

cd infra/ansible

# ALWAYS confirm the target first — a prior mistake in this repo silently hit dev.
ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod \
  --limit k3s-prod-3 --list-hosts

ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3

Rollback, per node, immediate:

systemctl disable --now k3s-etcd-io-priority.service
printf 'mq-deadline\n' > /sys/block/sda/queue/scheduler
printf '64\n'          > /sys/block/sda/queue/nr_requests
rm -f /etc/systemd/system/{system.slice,k3s.service}.d/10-etcd-io-weight.conf
systemctl daemon-reload