docs(ansible): add re-provision runbook + node-host playbook list

A node rebuild (tofu -replace / fresh OS) wipes host-level config Flux/Helm don't manage.
site.yml omits kube-vip.yml and vpn.egress.gateway.yml, so a re-provision missed the node-host
OpenVPN on k3s-dev-3 and dropped all egress-class=vpn pod traffic. Document the node-host layers
that must be re-run after any re-provision, and correct the stale author-only banner (these plays
are live).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-30 21:18:29 -05:00
1 parent 22db5fdb63
commit 1f921fbfed
1 file changed
+29 -3
+29 -3
View File
@@ -4,8 +4,9 @@ The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
CNI, verifies readiness, then invokes the `bootstrap/` entrypoints.
> **AUTHOR-ONLY — NOTHING HERE HAS BEEN EXECUTED.** No `ansible-playbook` run, no
> `tofu apply`, no cluster contacted. The PM static-validates (see below).
> **LIVE.** These plays have been executed against the running dev cluster (k3s
> v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the
> operational bring-up + re-provision runbook. See **Re-provisioning a node** below.
Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s), ADR-0002 (Cilium),
ADR-0004 (Flux hand-off), ADR-0007 (SOPS).
@@ -22,14 +23,39 @@ ADR-0004 (Flux hand-off), ADR-0007 (SOPS).
## Run order
```
cd infra/ansible
ansible-playbook playbooks/site.yml # everything, in order
ansible-playbook playbooks/site.yml # core bring-up, in order (baseline→k3s→cilium→verify→flux)
# — or step by step —
ansible-playbook playbooks/node-baseline.yml # 0. packages, kernel modules, Longhorn disk, swap-off
ansible-playbook playbooks/k3s.yml # 1. HA k3s (primary --cluster-init, then joins)
ansible-playbook playbooks/cilium.yml # 2. Cilium CNI (cluster is NotReady until this)
ansible-playbook playbooks/verify.yml # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOPS then Flux)
```
> **`site.yml` is NOT everything.** It covers the core bring-up only. Two node-host
> layers live OUTSIDE `site.yml` and must be run separately:
> ```
> ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); targets k3s_servers
> ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; targets k3s-dev-3
> ```
## Re-provisioning a node (tofu -replace / fresh OS)
A node rebuild wipes **everything on the host filesystem** — including host-level config
that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
node-host layers or the node comes back missing pieces:
| Re-run | Why it doesn't survive a rebuild |
|---|---|
| `node-baseline.yml` | packages (incl. `cryptsetup`), kernel modules, Longhorn disk, swap-off |
| `k3s.yml` | the k3s server binary + config + join (secondaries re-join the VIP) |
| `kube-vip.yml` | the kube-vip auto-deploy manifest on the server's host path |
| `vpn.egress.gateway.yml` | **the OpenVPN (PIA) client, `tun0`, systemd units + nftables kill-switch on k3s-dev-3 — miss this and `egress-class=vpn` pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet** |
Everything Flux owns (platform + apps) self-heals once the node re-joins; only the
node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3,
`systemctl is-active vpn-egress-tunnel.service` and `curl --interface tun0 https://ipinfo.io/ip`
should return a PIA exit IP (not the home ISP).
## What each layer does
| Playbook / role | Action |
|---|---|