jakman85
a819c50ffe
roles/etcd_io_priority + playbooks/etcd-io-priority.yml, enabled for [k3s_prod]
only. Mitigation for the 2026-09-07 k3s-prod-1 outage: etcd's WAL fdatasync took
3.4-6.7 s (expects 1 s), the node went NotReady while the host stayed up, and
/proc/pressure/io full avg300 read 11.36 there against 0.75 (node 2, NVMe) and
1.29 (node 3).
WHAT IT DOES, on the SATA nodes only:
1. k3s-etcd-io-priority.service (oneshot, Before=k3s.service) switches the
shared device to BFQ and drops nr_requests 64 -> 32.
2. IOWeight=10000 drop-ins on system.slice AND k3s.service, which BFQ turns
into io.bfq.weight=1000 — a 10:1 share against the default-weighted
kubepods.slice where Longhorn's instance-managers live.
3. Reads all of it back out of the kernel and ASSERTS. If the scheduler did
not switch or the weight did not land, the play fails.
THE THREE OBVIOUS ANSWERS ARE INERT ON THESE NODES, ALL CHECKED LIVE, ALL
DELIBERATELY ABSENT:
* IOSchedulingClass=realtime on k3s.service. mq-deadline DOES honour the
priority class, so this looks right — but ioprio is inherited across fork
and kept across exec (block/blk-ioc.c __copy_io copies ioc->ioprio) and
CONTAINERD IS A CHILD OF THE K3S SERVER PROCESS. Verified on k3s-prod-1:
containerd pid 84553 has ppid 84333 = k3s-server, and
ionice -c2 -n0 -- sh -c 'ionice -p $$; sh -c "ionice -p \$\$"'
prints best-effort prio 0 at both levels. Realtime for k3s is realtime for
every container on the node, Longhorn included. No relative gain.
* IOSchedulingPriority (the level inside a class): mq-deadline buckets by
CLASS only, so BE-0 vs BE-4 is a no-op.
* io.latency, which would be the ideal scheduler-independent protection:
CONFIG_BLK_CGROUP_IOLATENCY is not built in Debian trixie's 7.1.8 kernel
(only IOCOST is) and no io.latency file exists under /sys/fs/cgroup.
AND ONE NON-OBVIOUS TRAP: bare IOWeight= is DECORATION here. systemd writes it
to io.weight (blk-iocost — compiled in but disabled, io.cost.qos is empty) and
io.bfq.weight (BFQ — not loaded). Neither reader exists on the stock nodes, so
the scheduler change and the weights have to be one role with one verification.
The value is 10000 rather than a "reasonable" 1000 for the same reason: systemd
rescales 1..10000 onto io.bfq.weight's 1..1000, so 1000 maps to ~101 against
BFQ's unweighted default of 100. The role asserts the result is >= 500.
k3s-prod-2 IS SKIPPED ON PURPOSE. The role discovers the device under
k3s_data_dir per node (verified: prod-1/3 -> /dev/sda5 -> sda, prod-2 ->
/dev/nvme0n1p5 -> nvme0n1) and writes NOTHING on NVMe rather than leaving inert
files behind — BFQ is a poor fit for NVMe and node 2 measured 0.75 IO PSI.
DEV IS UNTOUCHED. Master switch defaults false and is set true only in
group_vars/k3s_prod; dev has a dedicated etcd device (docs/plans/ssd-etcd-
isolation.md) and nothing to arbitrate. Verified with --list-hosts: no extra
vars -> the three dev hosts (where the role is a no-op); -e k3s_hosts=k3s_prod
-> the three prod hosts; --limit k3s-prod-3 -> one.
NOTHING IS RESTARTED. The oneshot writes two sysfs files; the weights go in
through daemon-reload. If daemon-reload turns out not to re-realize a running
k3s.service's cgroup, the verification FAILS and names the rolling restart as
the fix rather than quietly leaving the weight unapplied.
WHAT IT CANNOT DO, stated in the role README and worth repeating: it does not
separate the two tenants. Both still write one ext4 filesystem, and no cgroup
control arbitrates the shared jbd2 journal that etcd's fdatasync waits on. It is
also UNPROVEN on this hardware — it fails loudly if the settings do not land, it
cannot tell you they helped. Apply to k3s-prod-3 first and watch
NodeIOPressureHigh. Replacing the C400 / giving etcd its own device remains the
real fix and the operator's (PROD-PROVISIONING.md §2 exit condition).
Syntax-checked, and the fact-derivation and assert expressions were exercised
offline against all three node shapes plus negative controls (a node still on
mq-deadline fails the gate; IOWeight=1000 -> 101 fails the floor).
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit de8c1f832724c011b03466842e18296de07a6776)
infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)
The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
CNI, verifies readiness, then invokes the bootstrap/ entrypoints.
LIVE. These plays have been executed against the running dev cluster (k3s v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the operational bring-up + re-provision runbook. See Re-provisioning a node below.
Serves R-ENV-01/02, R-PLAT-01, ADR-0001 (k3s), ADR-0002 (Cilium), ADR-0004 (Flux hand-off), ADR-0007 (SOPS).
Prerequisites
tofu applyin../tofu/has created the VMs; cloud-init (../cloud-init/) has baselined them (modules, sysctls, Longhorn disk, SSH user).- Inventory reflects the node IPs.
inventory/hosts.iniships aligned with the LOCKED defaults; regenerate from tofu if overridden:cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini - Run from the dev container (has
kubectl+helm, used by the Cilium/verify/ Flux plays viadelegate_to: localhost).
Run order
cd infra/ansible
ansible-playbook playbooks/site.yml # core bring-up, in order (baseline→k3s→cilium→verify→flux)
# — or step by step —
ansible-playbook playbooks/node-baseline.yml # 0. packages, kernel modules, Longhorn disk, swap-off
ansible-playbook playbooks/k3s.yml # 1. HA k3s (primary --cluster-init, then joins)
ansible-playbook playbooks/cilium.yml # 2. Cilium CNI (cluster is NotReady until this)
ansible-playbook playbooks/verify.yml # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOPS then Flux)
site.ymlis NOT everything. It covers the core bring-up only. Two node-host layers live OUTSIDEsite.ymland must be run separately:ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); defaults to k3s_servers ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3(Both take the same target-group override as the rest — see Targeting prod below.)
Standalone diagnostics (not part of any bring-up; safe to point anywhere):
ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod # read-only hardware health ansible-playbook playbooks/memattr-survey.yml # EFI memattr reboot survey
memattr-survey.ymlis the only play here that reboots its targets, and it will not do so until-e memattr_survey_reboots_authorised=trueis passed. Seeroles/memattr_survey/README.mdandinfra/PROD-PROVISIONING.md§0b.
Re-provisioning a node (tofu -replace / fresh OS)
A node rebuild wipes everything on the host filesystem — including host-level config
that Flux/Helm do NOT manage. After tofu apply/-replace re-creates a node, re-run the
node-host layers or the node comes back missing pieces:
| Re-run | Why it doesn't survive a rebuild |
|---|---|
node-baseline.yml |
packages (incl. cryptsetup), kernel modules, Longhorn disk, swap-off |
k3s.yml |
the k3s server binary + config + join (secondaries re-join the VIP) |
kube-vip.yml |
the kube-vip auto-deploy manifest on the server's host path |
vpn.egress.gateway.yml |
the OpenVPN (PIA) client, tun0, systemd units + nftables kill-switch on k3s-dev-3 — miss this and egress-class=vpn pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet |
Everything Flux owns (platform + apps) self-heals once the node re-joins; only the
node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3,
systemctl is-active vpn-egress-tunnel.service and curl --interface tun0 https://ipinfo.io/ip
should return a PIA exit IP (not the home ISP).
What each layer does
| Playbook / role | Action |
|---|---|
k3s.yml / k3s_server |
k3s server install with --flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb; primary --cluster-init, secondaries join via the primary's node-token (fetched, never pre-stored) |
cilium.yml / cilium |
Gateway API CRDs, then Helm cilium with kubeProxyReplacement, k8sServiceHost/Port, gatewayAPI, egressGateway, hubble, l2announcements |
verify.yml / cluster_verify |
waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
flux-bootstrap.yml / flux_bootstrap |
calls bootstrap/ (SOPS age key, then flux bootstrap --path=clusters/dev) |
Cilium ownership / parity contract (docs/02)
Ansible installs Cilium; the Flux platform/cilium HelmRelease adopts config
afterward. The parity-critical values must match between this role's
templates/cilium-values.yaml.j2 and that HelmRelease:
kubeProxyReplacement: truek8sServiceHost: {{ k3s_apiserver_endpoint }}— the kube-vip CONTROL-PLANE VIP (= kube_vip_address: dev192.168.1.49, prod192.168.2.9), notk3s_api_hostk8sServicePort: {{ k3s_api_port }}(6443 in both environments)
Two API addresses, split 2026-09-06 — do not re-merge them
k3s_api_host and k3s_apiserver_endpoint were one variable, and the conflation put every
prod cilium-agent on 192.168.2.10 (k3s-prod-1's own address) on a cluster whose point is
HA — verified live. Lose that node and the datapath keeps serving from Cilium's cache while no
service or policy update ever lands, which is a silent failure, not an outage.
| variable | value | consumed by | why it must be that address |
|---|---|---|---|
k3s_api_host |
the founder's own node IP (dev 192.168.1.50, prod 192.168.2.10) |
roles/k3s_server: the secondary --server https://…:6443 join URL, and one k3s_tls_sans entry |
on a cold build no VIP exists yet (kube-vip is a k3s auto-deploy manifest, so it needs a cluster first). The founder just ran --cluster-init and is up |
k3s_apiserver_endpoint |
{{ kube_vip_address }} (dev 192.168.1.49, prod 192.168.2.9) |
roles/cilium → k8sServiceHost; any steady-state client |
it must survive losing any single server |
Consequences already handled: playbooks/site.yml imports kube-vip.yml between k3s.yml
and cilium.yml (the VIP has to be floating before the CNI dials it), and roles/kube_vip's
last task blocks until the VIP answers on 6443. k3s_tls_sans already carried both addresses,
so no cert reissue and no k3s restart are involved.
The Flux side of the same contract is ${K3S_API_HOST} in platform/cilium/helmrelease.yaml,
set per overlay in clusters/<env>/kustomization.yaml; [parity] means it must equal
k3s_apiserver_endpoint for the SAME environment. Asserted by
conformance/tests/l0-static/ha-control-plane-as-code.
Interface assumed of bootstrap/ (authored by other workers, NOT here)
roles/flux_bootstrap invokes two entrypoints via overridable command vars:
flux_bootstrap_sops_cmd→bootstrap/sops/install-age-key.sh(Security Engineer, T-1.4): installs the age private key as thesops-ageSecret influx-system. Key never committed (.gitignoreguardsage.key/*.key).flux_bootstrap_flux_cmd→bootstrap/flux/bootstrap.sh(GitOps Engineer, T-1.4): runsflux bootstrap ... --path=clusters/dev.
These script names/args are assumptions — if the bootstrap deliverables
expose different entrypoints, override the two vars in
roles/flux_bootstrap/defaults/main.yml; the task graph is unchanged. Flagged
for PM reconciliation.
Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)
inventory/hosts.ini carries a [k3s_prod_primary] / [k3s_prod_secondary] /
[k3s_prod:children] section for the three planned bare-metal nodes (k3s-prod-1 at
192.168.2.10 exists and is reachable today; k3s-prod-2/-3 at .11/.12 are addressed but
not yet racked, so their inventory lines stay commented out), with its own group_vars/k3s_prod/
(vaulted ansible_become_password + non-secret overrides).
Nothing here is forked for prod. Every node-host play takes its target group from a variable that defaults to the dev group, so the dev commands above are unchanged, and prod is the same command plus an override. Two separate mechanisms, for two separate reasons:
| Layer | Knob | Set via | Why |
|---|---|---|---|
| Which hosts a play runs on | k3s_hosts, k3s_primary_hosts, k3s_secondary_hosts, vpn_egress_gateway_hosts |
-e extra vars only |
Ansible templates a play's hosts: before it has any host, so it cannot see group_vars at all |
| Which group a role means by "my primary" / "my servers" | k3s_primary_group, k3s_servers_group (+ cluster_kubeconfig_name) |
group_vars — dev in all.yml, prod in k3s_prod/vars.yml |
Host context exists by then, so each environment answers for itself with no per-run flag — same mechanism k3s_api_host already used |
playbooks/node-health.yml established the first pattern (-e node_health_hosts=k3s_prod); it
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
bring-up chain.
playbooks/memattr-survey.yml follows it (-e memattr_survey_hosts=…, defaulting to k3s_prod)
for the same reason, and adds a second interlock on top: because it reboots its targets, the
role also refuses any host that is not a member of memattr_survey_required_group. A host pattern
can be mistyped; inventory membership cannot.
# dev — completely unchanged, no extra vars
ansible-playbook playbooks/site.yml
# prod — one flag for the whole-cluster plays
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod # see caveat below
# prod — k3s.yml needs both halves of the primary/secondary split
ansible-playbook playbooks/k3s.yml \
-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
# prod — the whole chain (extra vars are global, so they reach imported playbooks)
ansible-playbook playbooks/site.yml \
-e k3s_hosts=k3s_prod \
-e k3s_primary_hosts=k3s_prod_primary \
-e k3s_secondary_hosts=k3s_prod_secondary
What the roles then resolve by themselves for a prod host, from group_vars/k3s_prod/vars.yml:
k3s_primary_group=k3s_prod_primary (the --cluster-init founder, the node-token delegate_to
target, and the node Cilium fetches the kubeconfig from), k3s_servers_group=k3s_prod (the tls-san
node-IP list), k3s_api_host=192.168.2.10 (the founder, bootstrap-join only),
kube_vip_address=192.168.2.9 — and therefore k3s_apiserver_endpoint=192.168.2.9, which is what
Cilium dials — and
cluster_kubeconfig_name=prod.kubeconfig — so a prod run can never overwrite the
.cache/dev.kubeconfig that infra/devcontainer/dev.sh exports as $KUBECONFIG.
Packages-only baseline (--tags packages)
roles/node_baseline is tagged packages / system / disks. An untagged run is unchanged (all
three, in order). --tags packages runs only the apt task:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
This exists to break a circular dependency: prod's disk layout (infra/PROD-PROVISIONING.md §2) is
blocked on a SMART read, SMART needs smartmontools, smartmontools ships in
node_baseline_packages — but the role's disk tasks need block devices that decision hasn't chosen
yet. With --tags packages no disks-tagged task is selected, so the deliberate
/dev/BLOCKED-pending-disk-decision-… placeholders in group_vars/k3s_prod/vars.yml are never
even templated, let alone handed to mkfs. A full run against a node whose devices are still
placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never
hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).
Still not runnable end to end
- The disk-layout (§2) and the
node_baseline_*_deviceoverrides remain BLOCKED placeholders. group_vars/k3s_prod/vault.yml(the become password) does not exist yet, so no prod play can escalate — and--checkagainst prod fails for the same reason.flux-bootstrap.ymlretargets, butroles/flux_bootstrapshells out tobootstrap/flux/bootstrap.sh, which bootstraps--path=clusters/dev. Prod needs--path=clusters/prod: abootstrap/change, not a target-group change (§7).vpn.egress.gateway.ymlis parameterised (-e vpn_egress_gateway_hosts=…) but prod has no egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own vaulted PIA credentials.- The operator has ruled the prod cluster stays uninitialized until all three nodes are present
(§8).
192.168.2.20is prod's ingress Gateway VIP — a Flux/cluster-varsconcern, not an Ansible one; the kube-vip control-plane VIP is192.168.2.9(kube_vip_address).
Everything above this section describes the dev runbook; the values in it (192.168.1.x,
.cache/dev.kubeconfig, clusters/dev) are dev's, supplied by group_vars, not baked into the
roles.
Static validation (PM, in dev container — NOT run here)
yamllint -c ../../ci/yamllint.yaml infra/ansible
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
ansible-lint playbooks/ roles/ # if the profile is enforced for infra/