--- # Give the k3s server process — and therefore the EMBEDDED ETCD inside it — a larger share of # the block device it shares with Longhorn. Mitigates the 2026-09-07 k3s-prod-1 outage, where # etcd's WAL fdatasync took 3.4-6.7 s (expects 1 s) and the node went NotReady while the host # stayed up. See roles/etcd_io_priority/README.md for the mechanism, the measurements, and the # list of things in this space that do NOT work on these nodes. # # NO-OP BY DEFAULT. The role's master switch (etcd_io_priority_enabled) is false in # defaults/main.yml and true only in inventory/group_vars/k3s_prod/vars.yml, so running this # with no extra vars against dev changes nothing on any node — dev already has a dedicated # etcd device and has nothing to arbitrate. # # TARGET (R-ENV-02): same `k3s_hosts` seam every other whole-cluster play uses. # ansible-playbook playbooks/etcd-io-priority.yml # dev, a no-op # ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod # prod # # RUN IT ON ONE NODE FIRST, AND CONFIRM THE TARGET BEFORE YOU RUN IT AT ALL. A prior mistake in # this repo silently targeted dev's three nodes; --list-hosts is how that is caught: # ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod \ # --limit k3s-prod-3 --list-hosts # k3s-prod-3 first, deliberately: it is the WD Blue SATA node (IO PSI 1.29 during the incident), # so it exercises the identical code path as the failing node 1 (11.36) at a fraction of the # stakes. Watch NodeIOPressureHigh / the "Nodes — CPU / Memory / I-O Pressure" dashboard for a # day before touching node 1. # # ROLLBACK is immediate and needs no playbook — see the header of # /usr/local/sbin/k3s-etcd-io-priority on the node. - name: I/O priority for the shared etcd + Longhorn device hosts: "{{ k3s_hosts | default('k3s_servers') }}" become: true # serial: 1 — one node at a time. Nothing here restarts k3s, but changing a live node's I/O # scheduler is a real change to a cluster member, and the whole point is that a bad outcome # on one node must never be simultaneous across all three. serial: 1 gather_facts: true roles: - role: etcd_io_priority