home-network-k3s-impl/infra/ansible

infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)

The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off (T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the CNI, verifies readiness, then invokes the bootstrap/ entrypoints.

LIVE. These plays have been executed against the running dev cluster (k3s v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the operational bring-up + re-provision runbook. See Re-provisioning a node below.

Serves R-ENV-01/02, R-PLAT-01, ADR-0001 (k3s), ADR-0002 (Cilium), ADR-0004 (Flux hand-off), ADR-0007 (SOPS).

Prerequisites

1. tofu apply in ../tofu/ has created the VMs; cloud-init (../cloud-init/) has baselined them (modules, sysctls, Longhorn disk, SSH user).
2. Inventory reflects the node IPs. inventory/hosts.ini ships aligned with the LOCKED defaults; regenerate from tofu if overridden: cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini
3. Run from the dev container (has kubectl + helm, used by the Cilium/verify/ Flux plays via delegate_to: localhost).

Run order

cd infra/ansible
ansible-playbook playbooks/site.yml        # core bring-up, in order (baseline→k3s→cilium→verify→flux)
# — or step by step —
ansible-playbook playbooks/node-baseline.yml    # 0. packages, kernel modules, Longhorn disk, swap-off
ansible-playbook playbooks/k3s.yml         # 1. HA k3s (primary --cluster-init, then joins)
ansible-playbook playbooks/cilium.yml      # 2. Cilium CNI (cluster is NotReady until this)
ansible-playbook playbooks/verify.yml      # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOPS then Flux)

site.yml is NOT everything. It covers the core bring-up only. Two node-host layers live OUTSIDE site.yml and must be run separately:

ansible-playbook playbooks/kube-vip.yml            # control-plane VIP (192.168.1.49); targets k3s_servers
ansible-playbook playbooks/vpn.egress.gateway.yml  # node-host OpenVPN split-tunnel + kill-switch; targets k3s-dev-3

Re-provisioning a node (tofu -replace / fresh OS)

A node rebuild wipes everything on the host filesystem — including host-level config that Flux/Helm do NOT manage. After tofu apply/-replace re-creates a node, re-run the node-host layers or the node comes back missing pieces:

Re-run	Why it doesn't survive a rebuild
node-baseline.yml	packages (incl. cryptsetup), kernel modules, Longhorn disk, swap-off
k3s.yml	the k3s server binary + config + join (secondaries re-join the VIP)
kube-vip.yml	the kube-vip auto-deploy manifest on the server's host path
vpn.egress.gateway.yml	the OpenVPN (PIA) client, tun0, systemd units + nftables kill-switch on k3s-dev-3 — miss this and egress-class=vpn pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet

Everything Flux owns (platform + apps) self-heals once the node re-joins; only the node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3, systemctl is-active vpn-egress-tunnel.service and curl --interface tun0 https://ipinfo.io/ip should return a PIA exit IP (not the home ISP).

What each layer does

Playbook / role	Action
k3s.yml / k3s_server	k3s server install with --flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb; primary --cluster-init, secondaries join via the primary's node-token (fetched, never pre-stored)
cilium.yml / cilium	Gateway API CRDs, then Helm cilium with kubeProxyReplacement, k8sServiceHost/Port, gatewayAPI, egressGateway, hubble, l2announcements
verify.yml / cluster_verify	waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent
flux-bootstrap.yml / flux_bootstrap	calls bootstrap/ (SOPS age key, then flux bootstrap --path=clusters/dev)

Cilium ownership / parity contract (docs/02)

Ansible installs Cilium; the Flux platform/cilium HelmRelease adopts config afterward. The parity-critical values must match between this role's templates/cilium-values.yaml.j2 and that HelmRelease:

• kubeProxyReplacement: true
• k8sServiceHost: {{ k3s_api_host }} (= primary node IP, 10.73.0.11 by default)
• k8sServicePort: 6443

The HelmRelease currently has k8sServiceHost: "REPLACE-ME" — WS-NET must set it to the same value this role uses (surfaced by tofu output primary_node_ip).

Interface assumed of bootstrap/ (authored by other workers, NOT here)

roles/flux_bootstrap invokes two entrypoints via overridable command vars:

• flux_bootstrap_sops_cmd → bootstrap/sops/install-age-key.sh (Security Engineer, T-1.4): installs the age private key as the sops-age Secret in flux-system. Key never committed (.gitignore guards age.key/*.key).
• flux_bootstrap_flux_cmd → bootstrap/flux/bootstrap.sh (GitOps Engineer, T-1.4): runs flux bootstrap ... --path=clusters/dev.

These script names/args are assumptions — if the bootstrap deliverables expose different entrypoints, override the two vars in roles/flux_bootstrap/defaults/main.yml; the task graph is unchanged. Flagged for PM reconciliation.

Static validation (PM, in dev container — NOT run here)

yamllint -c ../../ci/yamllint.yaml infra/ansible
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
ansible-lint playbooks/ roles/          # if the profile is enforced for infra/