jakman85
a819c50ffe
roles/etcd_io_priority + playbooks/etcd-io-priority.yml, enabled for [k3s_prod]
only. Mitigation for the 2026-09-07 k3s-prod-1 outage: etcd's WAL fdatasync took
3.4-6.7 s (expects 1 s), the node went NotReady while the host stayed up, and
/proc/pressure/io full avg300 read 11.36 there against 0.75 (node 2, NVMe) and
1.29 (node 3).
WHAT IT DOES, on the SATA nodes only:
1. k3s-etcd-io-priority.service (oneshot, Before=k3s.service) switches the
shared device to BFQ and drops nr_requests 64 -> 32.
2. IOWeight=10000 drop-ins on system.slice AND k3s.service, which BFQ turns
into io.bfq.weight=1000 — a 10:1 share against the default-weighted
kubepods.slice where Longhorn's instance-managers live.
3. Reads all of it back out of the kernel and ASSERTS. If the scheduler did
not switch or the weight did not land, the play fails.
THE THREE OBVIOUS ANSWERS ARE INERT ON THESE NODES, ALL CHECKED LIVE, ALL
DELIBERATELY ABSENT:
* IOSchedulingClass=realtime on k3s.service. mq-deadline DOES honour the
priority class, so this looks right — but ioprio is inherited across fork
and kept across exec (block/blk-ioc.c __copy_io copies ioc->ioprio) and
CONTAINERD IS A CHILD OF THE K3S SERVER PROCESS. Verified on k3s-prod-1:
containerd pid 84553 has ppid 84333 = k3s-server, and
ionice -c2 -n0 -- sh -c 'ionice -p $$; sh -c "ionice -p \$\$"'
prints best-effort prio 0 at both levels. Realtime for k3s is realtime for
every container on the node, Longhorn included. No relative gain.
* IOSchedulingPriority (the level inside a class): mq-deadline buckets by
CLASS only, so BE-0 vs BE-4 is a no-op.
* io.latency, which would be the ideal scheduler-independent protection:
CONFIG_BLK_CGROUP_IOLATENCY is not built in Debian trixie's 7.1.8 kernel
(only IOCOST is) and no io.latency file exists under /sys/fs/cgroup.
AND ONE NON-OBVIOUS TRAP: bare IOWeight= is DECORATION here. systemd writes it
to io.weight (blk-iocost — compiled in but disabled, io.cost.qos is empty) and
io.bfq.weight (BFQ — not loaded). Neither reader exists on the stock nodes, so
the scheduler change and the weights have to be one role with one verification.
The value is 10000 rather than a "reasonable" 1000 for the same reason: systemd
rescales 1..10000 onto io.bfq.weight's 1..1000, so 1000 maps to ~101 against
BFQ's unweighted default of 100. The role asserts the result is >= 500.
k3s-prod-2 IS SKIPPED ON PURPOSE. The role discovers the device under
k3s_data_dir per node (verified: prod-1/3 -> /dev/sda5 -> sda, prod-2 ->
/dev/nvme0n1p5 -> nvme0n1) and writes NOTHING on NVMe rather than leaving inert
files behind — BFQ is a poor fit for NVMe and node 2 measured 0.75 IO PSI.
DEV IS UNTOUCHED. Master switch defaults false and is set true only in
group_vars/k3s_prod; dev has a dedicated etcd device (docs/plans/ssd-etcd-
isolation.md) and nothing to arbitrate. Verified with --list-hosts: no extra
vars -> the three dev hosts (where the role is a no-op); -e k3s_hosts=k3s_prod
-> the three prod hosts; --limit k3s-prod-3 -> one.
NOTHING IS RESTARTED. The oneshot writes two sysfs files; the weights go in
through daemon-reload. If daemon-reload turns out not to re-realize a running
k3s.service's cgroup, the verification FAILS and names the rolling restart as
the fix rather than quietly leaving the weight unapplied.
WHAT IT CANNOT DO, stated in the role README and worth repeating: it does not
separate the two tenants. Both still write one ext4 filesystem, and no cgroup
control arbitrates the shared jbd2 journal that etcd's fdatasync waits on. It is
also UNPROVEN on this hardware — it fails loudly if the settings do not land, it
cannot tell you they helped. Apply to k3s-prod-3 first and watch
NodeIOPressureHigh. Replacing the C400 / giving etcd its own device remains the
real fix and the operator's (PROD-PROVISIONING.md §2 exit condition).
Syntax-checked, and the fact-derivation and assert expressions were exercised
offline against all three node shapes plus negative controls (a node still on
mq-deadline fails the gate; IOWeight=1000 -> 101 fails the floor).
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit de8c1f832724c011b03466842e18296de07a6776)
68 lines
3.8 KiB
YAML
68 lines
3.8 KiB
YAML
---
|
|
# Full dev/test cluster bring-up, in order (docs/02 sequencing; run AFTER
|
|
# `tofu apply` has created and cloud-init has baselined the VMs):
|
|
# k3s HA install -> kube-vip control-plane VIP -> Cilium CNI -> verify readiness
|
|
# -> Flux + SOPS hand-off.
|
|
# (kube-vip joined the chain 2026-09-06 — see the note at its import below; it is a
|
|
# prerequisite of the CNI now, not a separate operator step.)
|
|
#
|
|
# TARGET (R-ENV-02): every imported play takes its host pattern from a var that
|
|
# defaults to the dev group, so `ansible-playbook playbooks/site.yml` with no extra
|
|
# vars is byte-for-byte the dev run it always was. Extra vars are global and reach
|
|
# imported playbooks, so the whole chain retargets with three flags:
|
|
# ansible-playbook playbooks/site.yml \
|
|
# -e k3s_hosts=k3s_prod \
|
|
# -e k3s_primary_hosts=k3s_prod_primary \
|
|
# -e k3s_secondary_hosts=k3s_prod_secondary
|
|
# DO NOT run that yet — prod is blocked on the disk-layout decision
|
|
# (infra/PROD-PROVISIONING.md §2) and the §7 repo blockers, and the operator has ruled
|
|
# the prod cluster stays uninitialized until all three nodes are present (§8).
|
|
- name: Import - node baseline (packages, kernel modules, Longhorn disk, swap-off)
|
|
ansible.builtin.import_playbook: node-baseline.yml
|
|
|
|
- name: Import - install k3s HA control plane
|
|
ansible.builtin.import_playbook: k3s.yml
|
|
|
|
# ADDED 2026-09-07, after the k3s-prod-1 etcd I/O-starvation outage. It must come AFTER
|
|
# k3s.yml because it writes an IOWeight drop-in for k3s.service and verifies the value in
|
|
# that unit's LIVE cgroup — on a cold build the cgroup does not exist until k3s runs, and
|
|
# the role reports that rather than asserting on it. It restarts nothing.
|
|
#
|
|
# A NO-OP FOR DEV, by design: roles/etcd_io_priority defaults its master switch to false and
|
|
# only inventory/group_vars/k3s_prod/vars.yml sets it true. Dev has a dedicated etcd device
|
|
# and must not be dragged onto a different I/O scheduler by a shared role.
|
|
- name: Import - I/O arbitration for the shared etcd + Longhorn device (prod only)
|
|
ansible.builtin.import_playbook: etcd-io-priority.yml
|
|
|
|
# ADDED 2026-09-06, AND ITS POSITION IS LOAD-BEARING: kube-vip MUST come up between k3s and
|
|
# Cilium. Cilium's k8sServiceHost is now the control-plane VIP (k3s_apiserver_endpoint =
|
|
# kube_vip_address) rather than the founder's own node IP, so installing Cilium before the
|
|
# VIP is announced would point every agent at an address nothing answers on — the exact
|
|
# shape of the 2026-09-01 incident, where agents that could not reach the API server
|
|
# deadlocked on a cold cluster.
|
|
#
|
|
# IT CANNOT COME EARLIER. kube-vip ships as a k3s auto-deploy manifest
|
|
# (<data-dir>/server/manifests), so a cluster must exist before it can be applied — hence
|
|
# after k3s.yml, not before.
|
|
#
|
|
# IT DOES NOT NEED THE CNI, which is what makes this ordering possible at all: the kube-vip
|
|
# DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 (roles/kube_vip's template
|
|
# explains why), so it reaches the node-local apiserver with no CNI and no kube-proxy, and
|
|
# DaemonSet pods carry the default not-ready/network-unavailable tolerations that let them
|
|
# schedule onto a still-NotReady node. roles/kube_vip blocks until the VIP actually answers
|
|
# on 6443, so cilium.yml below cannot start against a VIP that is not up.
|
|
#
|
|
# Re-running this against the LIVE dev cluster is a no-op for this play: dev already runs
|
|
# kube-vip with the same rendered manifest, and the wait finds the VIP already answering.
|
|
- name: Import - float the control-plane HA VIP (kube-vip) BEFORE the CNI dials it
|
|
ansible.builtin.import_playbook: kube-vip.yml
|
|
|
|
- name: Import - install Cilium CNI
|
|
ansible.builtin.import_playbook: cilium.yml
|
|
|
|
- name: Import - verify cluster readiness
|
|
ansible.builtin.import_playbook: verify.yml
|
|
|
|
- name: Import - Flux + SOPS bootstrap hand-off
|
|
ansible.builtin.import_playbook: flux-bootstrap.yml
|