home-network-k3s-impl/infra/ansible/README.md

infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)

The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off (T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the CNI, verifies readiness, then invokes the bootstrap/ entrypoints.

LIVE. These plays have been executed against the running dev cluster (k3s v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the operational bring-up + re-provision runbook. See Re-provisioning a node below.

Serves R-ENV-01/02, R-PLAT-01, ADR-0001 (k3s), ADR-0002 (Cilium), ADR-0004 (Flux hand-off), ADR-0007 (SOPS).

Prerequisites

1. tofu apply in ../tofu/ has created the VMs; cloud-init (../cloud-init/) has baselined them (modules, sysctls, Longhorn disk, SSH user).
2. Inventory reflects the node IPs. inventory/hosts.ini ships aligned with the LOCKED defaults; regenerate from tofu if overridden: cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini
3. Run from the dev container (has kubectl + helm, used by the Cilium/verify/ Flux plays via delegate_to: localhost).

Run order

cd infra/ansible
ansible-playbook playbooks/site.yml        # core bring-up, in order (baseline→k3s→cilium→verify→flux)
# — or step by step —
ansible-playbook playbooks/node-baseline.yml    # 0. packages, kernel modules, Longhorn disk, swap-off
ansible-playbook playbooks/k3s.yml         # 1. HA k3s (primary --cluster-init, then joins)
ansible-playbook playbooks/etcd-io-priority.yml # 1b. shared etcd/Longhorn device I/O weights (PROD ONLY; a no-op on dev)
ansible-playbook playbooks/cilium.yml      # 2. Cilium CNI (cluster is NotReady until this)
ansible-playbook playbooks/verify.yml      # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOPS then Flux)

site.yml is NOT everything. It covers the core bring-up only. Two node-host layers live OUTSIDE site.yml and must be run separately:

ansible-playbook playbooks/kube-vip.yml            # control-plane VIP (192.168.1.49); defaults to k3s_servers
ansible-playbook playbooks/vpn.egress.gateway.yml  # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3

(Both take the same target-group override as the rest — see Targeting prod below.)

Standalone diagnostics (not part of any bring-up; safe to point anywhere):

ansible-playbook playbooks/node-health.yml   -e node_health_hosts=k3s_prod   # read-only hardware health
ansible-playbook playbooks/memattr-survey.yml                                 # EFI memattr reboot survey

memattr-survey.yml is the only play here that reboots its targets, and it will not do so until -e memattr_survey_reboots_authorised=true is passed. See roles/memattr_survey/README.md and infra/PROD-PROVISIONING.md §0b.

Re-provisioning a node (tofu -replace / fresh OS)

A node rebuild wipes everything on the host filesystem — including host-level config that Flux/Helm do NOT manage. After tofu apply/-replace re-creates a node, re-run the node-host layers or the node comes back missing pieces:

Re-run	Why it doesn't survive a rebuild
node-baseline.yml	packages (incl. cryptsetup), kernel modules, Longhorn disk, swap-off
k3s.yml	the k3s server binary + config + join (secondaries re-join the VIP)
kube-vip.yml	the kube-vip auto-deploy manifest on the server's host path
vpn.egress.gateway.yml	the OpenVPN (PIA) client, tun0, systemd units + nftables kill-switch on k3s-dev-3 — miss this and egress-class=vpn pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet

Everything Flux owns (platform + apps) self-heals once the node re-joins; only the node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3, systemctl is-active vpn-egress-tunnel.service and curl --interface tun0 https://ipinfo.io/ip should return a PIA exit IP (not the home ISP).

What each layer does

Playbook / role	Action
k3s.yml / k3s_server	k3s server install with --flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb; primary --cluster-init, secondaries join via the primary's node-token (fetched, never pre-stored)
cilium.yml / cilium	Gateway API CRDs, then Helm cilium with kubeProxyReplacement, k8sServiceHost/Port, gatewayAPI, egressGateway, hubble, l2announcements
etcd-io-priority.yml / etcd_io_priority	prod only (master switch defaults false; group_vars/k3s_prod sets it true). Switches the disk carrying k3s_data_dir to BFQ, drops nr_requests to 32, and gives system.slice + k3s.service IOWeight=10000 so the embedded etcd outweighs kubepods.slice — then reads every value back out of the kernel and fails if it did not land. Mitigation for the 2026-09-07 k3s-prod-1 etcd WAL-fsync starvation. Restarts nothing. NVMe-backed nodes are skipped, not decorated. See roles/etcd_io_priority/README.md for the mechanisms in this space that are inert here and must not be "fixed" back in
verify.yml / cluster_verify	waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent
flux-bootstrap.yml / flux_bootstrap	calls bootstrap/ (SOPS age key, then flux bootstrap --path=clusters/dev)

Cilium ownership / parity contract (docs/02)

Ansible installs Cilium; the Flux platform/cilium HelmRelease adopts config afterward. The parity-critical values must match between this role's templates/cilium-values.yaml.j2 and that HelmRelease:

• kubeProxyReplacement: true
• k8sServiceHost: {{ k3s_apiserver_endpoint }} — the kube-vip CONTROL-PLANE VIP (= kube_vip_address: dev 192.168.1.49, prod 192.168.2.9), not k3s_api_host
• k8sServicePort: {{ k3s_api_port }} (6443 in both environments)

Two API addresses, split 2026-09-06 — do not re-merge them

k3s_api_host and k3s_apiserver_endpoint were one variable, and the conflation put every prod cilium-agent on 192.168.2.10 (k3s-prod-1's own address) on a cluster whose point is HA — verified live. Lose that node and the datapath keeps serving from Cilium's cache while no service or policy update ever lands, which is a silent failure, not an outage.

variable	value	consumed by	why it must be that address
k3s_api_host	the founder's own node IP (dev 192.168.1.50, prod 192.168.2.10)	roles/k3s_server: the secondary --server https://…:6443 join URL, and one k3s_tls_sans entry	on a cold build no VIP exists yet (kube-vip is a k3s auto-deploy manifest, so it needs a cluster first). The founder just ran --cluster-init and is up
k3s_apiserver_endpoint	{{ kube_vip_address }} (dev 192.168.1.49, prod 192.168.2.9)	roles/cilium → k8sServiceHost; any steady-state client	it must survive losing any single server

Consequences already handled: playbooks/site.yml imports kube-vip.yml between k3s.yml and cilium.yml (the VIP has to be floating before the CNI dials it), and roles/kube_vip's last task blocks until the VIP answers on 6443. k3s_tls_sans already carried both addresses, so no cert reissue and no k3s restart are involved.

The Flux side of the same contract is ${K3S_API_HOST} in platform/cilium/helmrelease.yaml, set per overlay in clusters/<env>/kustomization.yaml; [parity] means it must equal k3s_apiserver_endpoint for the SAME environment. Asserted by conformance/tests/l0-static/ha-control-plane-as-code.

Interface assumed of bootstrap/ (authored by other workers, NOT here)

roles/flux_bootstrap invokes two entrypoints via overridable command vars:

• flux_bootstrap_sops_cmd → bootstrap/sops/install-age-key.sh (Security Engineer, T-1.4): installs the age private key as the sops-age Secret in flux-system. Key never committed (.gitignore guards age.key/*.key).
• flux_bootstrap_flux_cmd → bootstrap/flux/bootstrap.sh (GitOps Engineer, T-1.4): runs flux bootstrap ... --path=clusters/dev.

These script names/args are assumptions — if the bootstrap deliverables expose different entrypoints, override the two vars in roles/flux_bootstrap/defaults/main.yml; the task graph is unchanged. Flagged for PM reconciliation.

Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)

inventory/hosts.ini carries a [k3s_prod_primary] / [k3s_prod_secondary] / [k3s_prod:children] section for the three bare-metal nodes — k3s-prod-1 192.168.2.10, k3s-prod-2 .11, k3s-prod-3 .12 — with its own group_vars/k3s_prod/ (vaulted ansible_become_password + non-secret overrides).

All three are live in the inventory. This paragraph used to say k3s-prod-2/-3 "are addressed but not yet racked, so their inventory lines stay commented out"; that stopped being true on 2026-09-03 when node 3 came online, and the group has been complete since. Corrected 2026-09-07 because the stale sentence is the kind that gets used to reason about blast radius: -e k3s_hosts=k3s_prod targets three hosts, not one. Confirm it, every time, before running anything:

ansible-playbook playbooks/<play>.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3 --list-hosts

Nothing here is forked for prod. Every node-host play takes its target group from a variable that defaults to the dev group, so the dev commands above are unchanged, and prod is the same command plus an override. Two separate mechanisms, for two separate reasons:

Layer	Knob	Set via	Why
Which hosts a play runs on	k3s_hosts, k3s_primary_hosts, k3s_secondary_hosts, vpn_egress_gateway_hosts	-e extra vars only	Ansible templates a play's hosts: before it has any host, so it cannot see group_vars at all
Which group a role means by "my primary" / "my servers"	k3s_primary_group, k3s_servers_group (+ cluster_kubeconfig_name)	group_vars — dev in all.yml, prod in k3s_prod/vars.yml	Host context exists by then, so each environment answers for itself with no per-run flag — same mechanism k3s_api_host already used

playbooks/node-health.yml established the first pattern (-e node_health_hosts=k3s_prod); it keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the bring-up chain. playbooks/memattr-survey.yml follows it (-e memattr_survey_hosts=…, defaulting to k3s_prod) for the same reason, and adds a second interlock on top: because it reboots its targets, the role also refuses any host that is not a member of memattr_survey_required_group. A host pattern can be mistyped; inventory membership cannot.

# dev — completely unchanged, no extra vars
ansible-playbook playbooks/site.yml

# prod — one flag for the whole-cluster plays
ansible-playbook playbooks/node-baseline.yml   -e k3s_hosts=k3s_prod
ansible-playbook playbooks/cilium.yml          -e k3s_hosts=k3s_prod
ansible-playbook playbooks/kube-vip.yml        -e k3s_hosts=k3s_prod
ansible-playbook playbooks/verify.yml          -e k3s_hosts=k3s_prod
ansible-playbook playbooks/flux-bootstrap.yml  -e k3s_hosts=k3s_prod   # see caveat below

# prod — I/O arbitration for the shared etcd/Longhorn disk. ONE NODE AT A TIME, node 3 first
# (WD Blue, IO PSI 1.29 during the incident) before node 1 (the 2011 C400, 11.36).
ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3 --list-hosts
ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3

# prod — k3s.yml needs both halves of the primary/secondary split
ansible-playbook playbooks/k3s.yml \
  -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary

# prod — the whole chain (extra vars are global, so they reach imported playbooks)
ansible-playbook playbooks/site.yml \
  -e k3s_hosts=k3s_prod \
  -e k3s_primary_hosts=k3s_prod_primary \
  -e k3s_secondary_hosts=k3s_prod_secondary

What the roles then resolve by themselves for a prod host, from group_vars/k3s_prod/vars.yml: k3s_primary_group=k3s_prod_primary (the --cluster-init founder, the node-token delegate_to target, and the node Cilium fetches the kubeconfig from), k3s_servers_group=k3s_prod (the tls-san node-IP list), k3s_api_host=192.168.2.10 (the founder, bootstrap-join only), kube_vip_address=192.168.2.9 — and therefore k3s_apiserver_endpoint=192.168.2.9, which is what Cilium dials — and cluster_kubeconfig_name=prod.kubeconfig — so a prod run can never overwrite the .cache/dev.kubeconfig that infra/devcontainer/dev.sh exports as $KUBECONFIG.

Packages-only baseline (--tags packages)

roles/node_baseline is tagged packages / system / disks. An untagged run is unchanged (all three, in order). --tags packages runs only the apt task:

ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1

This exists to break a circular dependency: prod's disk layout (infra/PROD-PROVISIONING.md §2) is blocked on a SMART read, SMART needs smartmontools, smartmontools ships in node_baseline_packages — but the role's disk tasks need block devices that decision hasn't chosen yet. With --tags packages no disks-tagged task is selected, so the deliberate /dev/BLOCKED-pending-disk-decision-… placeholders in group_vars/k3s_prod/vars.yml are never even templated, let alone handed to mkfs. A full run against a node whose devices are still placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).

Still not runnable end to end

• The disk-layout (§2) and the node_baseline_*_device overrides remain BLOCKED placeholders.
• group_vars/k3s_prod/vault.yml (the become password) does not exist yet, so no prod play can escalate — and --check against prod fails for the same reason.
• flux-bootstrap.yml retargets, but roles/flux_bootstrap shells out to bootstrap/flux/bootstrap.sh, which bootstraps --path=clusters/dev. Prod needs --path=clusters/prod: a bootstrap/ change, not a target-group change (§7).
• vpn.egress.gateway.yml is parameterised (-e vpn_egress_gateway_hosts=…) but prod has no egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own vaulted PIA credentials.
• The operator has ruled the prod cluster stays uninitialized until all three nodes are present (§8). 192.168.2.20 is prod's ingress Gateway VIP — a Flux/cluster-vars concern, not an Ansible one; the kube-vip control-plane VIP is 192.168.2.9 (kube_vip_address).

Everything above this section describes the dev runbook; the values in it (192.168.1.x, .cache/dev.kubeconfig, clusters/dev) are dev's, supplied by group_vars, not baked into the roles.

Static validation (PM, in dev container — NOT run here)

yamllint -c ../../ci/yamllint.yaml infra/ansible
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
ansible-lint playbooks/ roles/          # if the profile is enforced for infra/