diff --git a/infra/ansible/README.md b/infra/ansible/README.md index fd966d4..59a74b4 100644 --- a/infra/ansible/README.md +++ b/infra/ansible/README.md @@ -4,8 +4,9 @@ The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off (T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the CNI, verifies readiness, then invokes the `bootstrap/` entrypoints. -> **AUTHOR-ONLY — NOTHING HERE HAS BEEN EXECUTED.** No `ansible-playbook` run, no -> `tofu apply`, no cluster contacted. The PM static-validates (see below). +> **LIVE.** These plays have been executed against the running dev cluster (k3s +> v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the +> operational bring-up + re-provision runbook. See **Re-provisioning a node** below. Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s), ADR-0002 (Cilium), ADR-0004 (Flux hand-off), ADR-0007 (SOPS). @@ -22,14 +23,39 @@ ADR-0004 (Flux hand-off), ADR-0007 (SOPS). ## Run order ``` cd infra/ansible -ansible-playbook playbooks/site.yml # everything, in order +ansible-playbook playbooks/site.yml # core bring-up, in order (baseline→k3s→cilium→verify→flux) # — or step by step — +ansible-playbook playbooks/node-baseline.yml # 0. packages, kernel modules, Longhorn disk, swap-off ansible-playbook playbooks/k3s.yml # 1. HA k3s (primary --cluster-init, then joins) ansible-playbook playbooks/cilium.yml # 2. Cilium CNI (cluster is NotReady until this) ansible-playbook playbooks/verify.yml # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOPS then Flux) ``` +> **`site.yml` is NOT everything.** It covers the core bring-up only. Two node-host +> layers live OUTSIDE `site.yml` and must be run separately: +> ``` +> ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); targets k3s_servers +> ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; targets k3s-dev-3 +> ``` + +## Re-provisioning a node (tofu -replace / fresh OS) +A node rebuild wipes **everything on the host filesystem** — including host-level config +that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the +node-host layers or the node comes back missing pieces: + +| Re-run | Why it doesn't survive a rebuild | +|---|---| +| `node-baseline.yml` | packages (incl. `cryptsetup`), kernel modules, Longhorn disk, swap-off | +| `k3s.yml` | the k3s server binary + config + join (secondaries re-join the VIP) | +| `kube-vip.yml` | the kube-vip auto-deploy manifest on the server's host path | +| `vpn.egress.gateway.yml` | **the OpenVPN (PIA) client, `tun0`, systemd units + nftables kill-switch on k3s-dev-3 — miss this and `egress-class=vpn` pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet** | + +Everything Flux owns (platform + apps) self-heals once the node re-joins; only the +node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3, +`systemctl is-active vpn-egress-tunnel.service` and `curl --interface tun0 https://ipinfo.io/ip` +should return a PIA exit IP (not the home ISP). + ## What each layer does | Playbook / role | Action | |---|---|