jakman85andClaude Opus 5
9d00a3fcd3
The WARN read "roles/node_baseline runs swapoff and neutralises the fstab entry; this node has not been baselined yet". That was a MIS-DIAGNOSIS. All three prod nodes HAD been baselined, had their fstab swap line commented out by the role, and were still swapping 977 MiB each after a reboot. Anyone reading that WARN would have re-run the role that had already run, and watched swap come back next boot. The detail now names the actual mechanism — systemd-gpt-auto-generator re-creating a .swap unit from the partition TYPE GUID 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f at every boot, without reading /etc/fstab — says in as many words that this is not "the node has not been baselined", and points at `node-baseline.yml --tags system`. The check was also blind in the other direction: SwapTotal == 0 is not evidence that swap is disabled, only that nothing has been swapped on yet this boot, so a node freshly swapoff-ed PASSed while still set to swap again on reboot — exactly the state the old role left behind. The collector now reports memory.swap_persistence (swap.target masked?, .swap units the generators emitted this boot, which of them are masked, leftover active fstab swap lines — all world-readable, no root, no systemctl parsing), and the check WARNs on "clean now, swapping after the next reboot". Rendered against every fact shape (swap active + unmasked, swap off + unmasked, after the fix, a dev VM with no swap partition at all, per-unit mask only, persistence unreadable, an older collector's output with the key absent, and /proc/meminfo unreadable) and verified live on k3s-prod-1 post-reboot: [PASS] memory.swap — no swap active and none can be activated: swap.target is masked Co-Authored-By: Claude Opus 5 <[email protected]>
infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)
The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
CNI, verifies readiness, then invokes the bootstrap/ entrypoints.
LIVE. These plays have been executed against the running dev cluster (k3s v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the operational bring-up + re-provision runbook. See Re-provisioning a node below.
Serves R-ENV-01/02, R-PLAT-01, ADR-0001 (k3s), ADR-0002 (Cilium), ADR-0004 (Flux hand-off), ADR-0007 (SOPS).
Prerequisites
tofu applyin../tofu/has created the VMs; cloud-init (../cloud-init/) has baselined them (modules, sysctls, Longhorn disk, SSH user).- Inventory reflects the node IPs.
inventory/hosts.iniships aligned with the LOCKED defaults; regenerate from tofu if overridden:cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini - Run from the dev container (has
kubectl+helm, used by the Cilium/verify/ Flux plays viadelegate_to: localhost).
Run order
cd infra/ansible
ansible-playbook playbooks/site.yml # core bring-up, in order (baseline→k3s→cilium→verify→flux)
# — or step by step —
ansible-playbook playbooks/node-baseline.yml # 0. packages, kernel modules, Longhorn disk, swap-off
ansible-playbook playbooks/k3s.yml # 1. HA k3s (primary --cluster-init, then joins)
ansible-playbook playbooks/etcd-io-priority.yml # 1b. shared etcd/Longhorn device I/O weights (PROD ONLY; a no-op on dev)
ansible-playbook playbooks/cilium.yml # 2. Cilium CNI (cluster is NotReady until this)
ansible-playbook playbooks/verify.yml # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOPS then Flux)
site.ymlis NOT everything. It covers the core bring-up only. Two node-host layers live OUTSIDEsite.ymland must be run separately:ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); defaults to k3s_servers ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3(Both take the same target-group override as the rest — see Targeting prod below.)
Standalone diagnostics (not part of any bring-up; safe to point anywhere):
ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod # read-only hardware health ansible-playbook playbooks/memattr-survey.yml # EFI memattr reboot survey
memattr-survey.ymlis the only play here that reboots its targets, and it will not do so until-e memattr_survey_reboots_authorised=trueis passed. Seeroles/memattr_survey/README.mdandinfra/PROD-PROVISIONING.md§0b.
Re-provisioning a node (tofu -replace / fresh OS)
A node rebuild wipes everything on the host filesystem — including host-level config
that Flux/Helm do NOT manage. After tofu apply/-replace re-creates a node, re-run the
node-host layers or the node comes back missing pieces:
| Re-run | Why it doesn't survive a rebuild |
|---|---|
node-baseline.yml |
packages (incl. cryptsetup), kernel modules, Longhorn disk, swap-off |
k3s.yml |
the k3s server binary + config + join (secondaries re-join the VIP) |
kube-vip.yml |
the kube-vip auto-deploy manifest on the server's host path |
vpn.egress.gateway.yml |
the OpenVPN (PIA) client, tun0, systemd units + nftables kill-switch on k3s-dev-3 — miss this and egress-class=vpn pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet |
Everything Flux owns (platform + apps) self-heals once the node re-joins; only the
node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3,
systemctl is-active vpn-egress-tunnel.service and curl --interface tun0 https://ipinfo.io/ip
should return a PIA exit IP (not the home ISP).
What each layer does
| Playbook / role | Action |
|---|---|
k3s.yml / k3s_server |
k3s server install with --flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb; primary --cluster-init, secondaries join via the primary's node-token (fetched, never pre-stored) |
cilium.yml / cilium |
Gateway API CRDs, then Helm cilium with kubeProxyReplacement, k8sServiceHost/Port, gatewayAPI, egressGateway, hubble, l2announcements |
etcd-io-priority.yml / etcd_io_priority |
prod only (master switch defaults false; group_vars/k3s_prod sets it true). Switches the disk carrying k3s_data_dir to BFQ, drops nr_requests to 32, and gives system.slice + k3s.service IOWeight=10000 so the embedded etcd outweighs kubepods.slice — then reads every value back out of the kernel and fails if it did not land. Mitigation for the 2026-09-07 k3s-prod-1 etcd WAL-fsync starvation. Restarts nothing. NVMe-backed nodes are skipped, not decorated. See roles/etcd_io_priority/README.md for the mechanisms in this space that are inert here and must not be "fixed" back in |
verify.yml / cluster_verify |
waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
flux-bootstrap.yml / flux_bootstrap |
calls bootstrap/ (SOPS age key, then flux bootstrap --path=clusters/dev) |
Cilium ownership / parity contract (docs/02)
Ansible installs Cilium; the Flux platform/cilium HelmRelease adopts config
afterward. The parity-critical values must match between this role's
templates/cilium-values.yaml.j2 and that HelmRelease:
kubeProxyReplacement: truek8sServiceHost: {{ k3s_apiserver_endpoint }}— the kube-vip CONTROL-PLANE VIP (= kube_vip_address: dev192.168.1.49, prod192.168.2.9), notk3s_api_hostk8sServicePort: {{ k3s_api_port }}(6443 in both environments)
Two API addresses, split 2026-09-06 — do not re-merge them
k3s_api_host and k3s_apiserver_endpoint were one variable, and the conflation put every
prod cilium-agent on 192.168.2.10 (k3s-prod-1's own address) on a cluster whose point is
HA — verified live. Lose that node and the datapath keeps serving from Cilium's cache while no
service or policy update ever lands, which is a silent failure, not an outage.
| variable | value | consumed by | why it must be that address |
|---|---|---|---|
k3s_api_host |
the founder's own node IP (dev 192.168.1.50, prod 192.168.2.10) |
roles/k3s_server: the secondary --server https://…:6443 join URL, and one k3s_tls_sans entry |
on a cold build no VIP exists yet (kube-vip is a k3s auto-deploy manifest, so it needs a cluster first). The founder just ran --cluster-init and is up |
k3s_apiserver_endpoint |
{{ kube_vip_address }} (dev 192.168.1.49, prod 192.168.2.9) |
roles/cilium → k8sServiceHost; any steady-state client |
it must survive losing any single server |
Consequences already handled: playbooks/site.yml imports kube-vip.yml between k3s.yml
and cilium.yml (the VIP has to be floating before the CNI dials it), and roles/kube_vip's
last task blocks until the VIP answers on 6443. k3s_tls_sans already carried both addresses,
so no cert reissue and no k3s restart are involved.
The Flux side of the same contract is ${K3S_API_HOST} in platform/cilium/helmrelease.yaml,
set per overlay in clusters/<env>/kustomization.yaml; [parity] means it must equal
k3s_apiserver_endpoint for the SAME environment. Asserted by
conformance/tests/l0-static/ha-control-plane-as-code.
Interface assumed of bootstrap/ (authored by other workers, NOT here)
roles/flux_bootstrap invokes two entrypoints via overridable command vars:
flux_bootstrap_sops_cmd→bootstrap/sops/install-age-key.sh(Security Engineer, T-1.4): installs the age private key as thesops-ageSecret influx-system. Key never committed (.gitignoreguardsage.key/*.key).flux_bootstrap_flux_cmd→bootstrap/flux/bootstrap.sh(GitOps Engineer, T-1.4): runsflux bootstrap ... --path=clusters/dev.
These script names/args are assumptions — if the bootstrap deliverables
expose different entrypoints, override the two vars in
roles/flux_bootstrap/defaults/main.yml; the task graph is unchanged. Flagged
for PM reconciliation.
Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)
inventory/hosts.ini carries a [k3s_prod_primary] / [k3s_prod_secondary] /
[k3s_prod:children] section for the three bare-metal nodes — k3s-prod-1 192.168.2.10,
k3s-prod-2 .11, k3s-prod-3 .12 — with its own group_vars/k3s_prod/ (vaulted
ansible_become_password + non-secret overrides).
All three are live in the inventory. This paragraph used to say k3s-prod-2/-3 "are
addressed but not yet racked, so their inventory lines stay commented out"; that stopped being
true on 2026-09-03 when node 3 came online, and the group has been complete since. Corrected
2026-09-07 because the stale sentence is the kind that gets used to reason about blast radius:
-e k3s_hosts=k3s_prod targets three hosts, not one. Confirm it, every time, before running
anything:
ansible-playbook playbooks/<play>.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3 --list-hosts
Nothing here is forked for prod. Every node-host play takes its target group from a variable that defaults to the dev group, so the dev commands above are unchanged, and prod is the same command plus an override. Two separate mechanisms, for two separate reasons:
| Layer | Knob | Set via | Why |
|---|---|---|---|
| Which hosts a play runs on | k3s_hosts, k3s_primary_hosts, k3s_secondary_hosts, vpn_egress_gateway_hosts |
-e extra vars only |
Ansible templates a play's hosts: before it has any host, so it cannot see group_vars at all |
| Which group a role means by "my primary" / "my servers" | k3s_primary_group, k3s_servers_group (+ cluster_kubeconfig_name) |
group_vars — dev in all.yml, prod in k3s_prod/vars.yml |
Host context exists by then, so each environment answers for itself with no per-run flag — same mechanism k3s_api_host already used |
playbooks/node-health.yml established the first pattern (-e node_health_hosts=k3s_prod); it
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
bring-up chain.
playbooks/memattr-survey.yml follows it (-e memattr_survey_hosts=…, defaulting to k3s_prod)
for the same reason, and adds a second interlock on top: because it reboots its targets, the
role also refuses any host that is not a member of memattr_survey_required_group. A host pattern
can be mistyped; inventory membership cannot.
# dev — completely unchanged, no extra vars
ansible-playbook playbooks/site.yml
# prod — one flag for the whole-cluster plays
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod # see caveat below
# prod — I/O arbitration for the shared etcd/Longhorn disk. ONE NODE AT A TIME, node 3 first
# (WD Blue, IO PSI 1.29 during the incident) before node 1 (the 2011 C400, 11.36).
ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3 --list-hosts
ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3
# prod — k3s.yml needs both halves of the primary/secondary split
ansible-playbook playbooks/k3s.yml \
-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
# prod — the whole chain (extra vars are global, so they reach imported playbooks)
ansible-playbook playbooks/site.yml \
-e k3s_hosts=k3s_prod \
-e k3s_primary_hosts=k3s_prod_primary \
-e k3s_secondary_hosts=k3s_prod_secondary
What the roles then resolve by themselves for a prod host, from group_vars/k3s_prod/vars.yml:
k3s_primary_group=k3s_prod_primary (the --cluster-init founder, the node-token delegate_to
target, and the node Cilium fetches the kubeconfig from), k3s_servers_group=k3s_prod (the tls-san
node-IP list), k3s_api_host=192.168.2.10 (the founder, bootstrap-join only),
kube_vip_address=192.168.2.9 — and therefore k3s_apiserver_endpoint=192.168.2.9, which is what
Cilium dials — and
cluster_kubeconfig_name=prod.kubeconfig — so a prod run can never overwrite the
.cache/dev.kubeconfig that infra/devcontainer/dev.sh exports as $KUBECONFIG.
Packages-only baseline (--tags packages)
roles/node_baseline is tagged packages / system / disks. An untagged run is unchanged (all
three, in order). --tags packages runs only the apt task:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
This exists to break a circular dependency: prod's disk layout (infra/PROD-PROVISIONING.md §2) is
blocked on a SMART read, SMART needs smartmontools, smartmontools ships in
node_baseline_packages — but the role's disk tasks need block devices that decision hasn't chosen
yet. With --tags packages no disks-tagged task is selected, so the deliberate
/dev/BLOCKED-pending-disk-decision-… placeholders in group_vars/k3s_prod/vars.yml are never
even templated, let alone handed to mkfs. A full run against a node whose devices are still
placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never
hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).
Still not runnable end to end
- The disk-layout (§2) and the
node_baseline_*_deviceoverrides remain BLOCKED placeholders. group_vars/k3s_prod/vault.yml(the become password) does not exist yet, so no prod play can escalate — and--checkagainst prod fails for the same reason.flux-bootstrap.ymlretargets, butroles/flux_bootstrapshells out tobootstrap/flux/bootstrap.sh, which bootstraps--path=clusters/dev. Prod needs--path=clusters/prod: abootstrap/change, not a target-group change (§7).vpn.egress.gateway.ymlis parameterised (-e vpn_egress_gateway_hosts=…) but prod has no egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own vaulted PIA credentials.- The operator has ruled the prod cluster stays uninitialized until all three nodes are present
(§8).
192.168.2.20is prod's ingress Gateway VIP — a Flux/cluster-varsconcern, not an Ansible one; the kube-vip control-plane VIP is192.168.2.9(kube_vip_address).
Everything above this section describes the dev runbook; the values in it (192.168.1.x,
.cache/dev.kubeconfig, clusters/dev) are dev's, supplied by group_vars, not baked into the
roles.
Static validation (PM, in dev container — NOT run here)
yamllint -c ../../ci/yamllint.yaml infra/ansible
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
ansible-lint playbooks/ roles/ # if the profile is enforced for infra/