docs: the §2 shared-device measurement came back, and it does not hold
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled

infra/PROD-PROVISIONING.md §2 recorded the etcd/Longhorn shared-partition risk as
"accepted to be MEASURED, not assumed", with an exit condition. The measurement
arrived on 2026-09-07 in production, 2 d 7 h after bring-up, as an outage.

§2 now carries "THE MEASUREMENT CAME BACK": the journal evidence, the three-node
IO PSI table (11.36 / 0.75 / 1.29), and the honest verdict that the section's own
premise — "every prod node is SSD or NVMe, where rebuild I/O costs latency rather
than collapsing it" — was right about node 2 and wrong about node 1. A 2011 C400
with 61,440 reallocated sectors and no power-loss protection is not the "SSD" that
reasoning assumed, and §0 had already named the three-way drive asymmetry as the
fault line. The exit condition is OPEN and the second bullet (dedicated device)
applies; that remains the operator's.

Also records what the three software mitigations in this branch do and do not buy,
including the point that no cgroup or scheduler control arbitrates the ext4 journal
both tenants share — and the gap that etcd's own fsync histograms, which the exit
condition literally asks for, are not collected because kubeEtcd is disabled for
k3s. PSI is the signal that exists.

DECISIONS-LOG gains the full entry: the four I/O mechanisms found INERT on these
nodes (IOSchedulingClass, IOSchedulingPriority, io.latency, bare IOWeight), the
finding that concurrentReplicaRebuildPerNodeLimit was already set and already held
during the outage, the third instance of config that looked applied and was not
(orphanAutoDeletion), and the three things still requiring the operator.

infra/ansible/README.md: CORRECTED a stale claim that k3s-prod-2/-3 "stay commented
out" of the inventory. They have been live in [k3s_prod_secondary] since
2026-09-03. That sentence is exactly the kind used to reason about blast radius —
`-e k3s_hosts=k3s_prod` targets THREE hosts, not one — so it is replaced with the
--list-hosts command that settles it. The new etcd-io-priority play is added to the
run order, the layer table and the prod command list.

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit a00a27a2567ea6a805c8c196b75df2c92a7c63a4)
This commit is contained in:
jakman85 committed 2026-09-07 02:46:26 -05:00
1 parent f7a9229d81
commit 863af01062
7 files changed
+760 -4

No files matched your search

+21 -4
View File
@@ -27,6 +27,7 @@ ansible-playbook playbooks/site.yml # core bring-up, in order (baseline
# — or step by step —
ansible-playbook playbooks/node-baseline.yml # 0. packages, kernel modules, Longhorn disk, swap-off
ansible-playbook playbooks/k3s.yml # 1. HA k3s (primary --cluster-init, then joins)
ansible-playbook playbooks/etcd-io-priority.yml # 1b. shared etcd/Longhorn device I/O weights (PROD ONLY; a no-op on dev)
ansible-playbook playbooks/cilium.yml # 2. Cilium CNI (cluster is NotReady until this)
ansible-playbook playbooks/verify.yml # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOPS then Flux)
@@ -71,6 +72,7 @@ should return a PIA exit IP (not the home ISP).
|---|---|
| `k3s.yml` / `k3s_server` | k3s server install with `--flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb`; primary `--cluster-init`, secondaries join via the primary's node-token (fetched, never pre-stored) |
| `cilium.yml` / `cilium` | Gateway API CRDs, then Helm `cilium` with `kubeProxyReplacement`, `k8sServiceHost/Port`, `gatewayAPI`, `egressGateway`, `hubble`, `l2announcements` |
| `etcd-io-priority.yml` / `etcd_io_priority` | **prod only** (master switch defaults false; `group_vars/k3s_prod` sets it true). Switches the disk carrying `k3s_data_dir` to **BFQ**, drops `nr_requests` to 32, and gives `system.slice` + `k3s.service` `IOWeight=10000` so the embedded etcd outweighs `kubepods.slice` — then reads every value back out of the kernel and **fails** if it did not land. Mitigation for the 2026-09-07 k3s-prod-1 etcd WAL-fsync starvation. Restarts nothing. NVMe-backed nodes are skipped, not decorated. See `roles/etcd_io_priority/README.md` for the mechanisms in this space that are **inert** here and must not be "fixed" back in |
| `verify.yml` / `cluster_verify` | waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
| `flux-bootstrap.yml` / `flux_bootstrap` | calls `bootstrap/` (SOPS age key, then `flux bootstrap --path=clusters/dev`) |
@@ -120,10 +122,20 @@ for PM reconciliation.
## Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)
`inventory/hosts.ini` carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
`[k3s_prod:children]` section for the three planned bare-metal nodes (`k3s-prod-1` at
`192.168.2.10` exists and is reachable today; `k3s-prod-2`/`-3` at `.11`/`.12` are addressed but
not yet racked, so their inventory lines stay commented out), with its own `group_vars/k3s_prod/`
(vaulted `ansible_become_password` + non-secret overrides).
`[k3s_prod:children]` section for the three bare-metal nodes — `k3s-prod-1` `192.168.2.10`,
`k3s-prod-2` `.11`, `k3s-prod-3` `.12` — with its own `group_vars/k3s_prod/` (vaulted
`ansible_become_password` + non-secret overrides).
**All three are live in the inventory.** This paragraph used to say `k3s-prod-2`/`-3` "are
addressed but not yet racked, so their inventory lines stay commented out"; that stopped being
true on 2026-09-03 when node 3 came online, and the group has been complete since. Corrected
2026-09-07 because the stale sentence is the kind that gets used to reason about blast radius:
`-e k3s_hosts=k3s_prod` targets **three** hosts, not one. Confirm it, every time, before running
anything:
```bash
ansible-playbook playbooks/<play>.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3 --list-hosts
```
**Nothing here is forked for prod.** Every node-host play takes its target group from a variable
that defaults to the dev group, so the dev commands above are unchanged, and prod is the same
@@ -153,6 +165,11 @@ ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod # see caveat below
# prod — I/O arbitration for the shared etcd/Longhorn disk. ONE NODE AT A TIME, node 3 first
# (WD Blue, IO PSI 1.29 during the incident) before node 1 (the 2011 C400, 11.36).
ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3 --list-hosts
ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3
# prod — k3s.yml needs both halves of the primary/secondary split
ansible-playbook playbooks/k3s.yml \
-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary