home-network-k3s-impl/infra/ansible/README.md

# infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)

The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
CNI, verifies readiness, then invokes the `bootstrap/` entrypoints.

> **LIVE.** These plays have been executed against the running dev cluster (k3s
> v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the
> operational bring-up + re-provision runbook. See **Re-provisioning a node** below.

Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s), ADR-0002 (Cilium),
ADR-0004 (Flux hand-off), ADR-0007 (SOPS).

## Prerequisites
1. `tofu apply` in `../tofu/` has created the VMs; cloud-init (`../cloud-init/`)
   has baselined them (modules, sysctls, Longhorn disk, SSH user).
2. Inventory reflects the node IPs. `inventory/hosts.ini` ships aligned with the
   LOCKED defaults; regenerate from tofu if overridden:
   `cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini`
3. Run from the dev container (has `kubectl` + `helm`, used by the Cilium/verify/
   Flux plays via `delegate_to: localhost`).

## Run order
```
cd infra/ansible
ansible-playbook playbooks/site.yml        # core bring-up, in order (baseline→k3s→cilium→verify→flux)
# — or step by step —
ansible-playbook playbooks/node-baseline.yml    # 0. packages, kernel modules, Longhorn disk, swap-off
ansible-playbook playbooks/k3s.yml         # 1. HA k3s (primary --cluster-init, then joins)
ansible-playbook playbooks/cilium.yml      # 2. Cilium CNI (cluster is NotReady until this)
ansible-playbook playbooks/verify.yml      # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOPS then Flux)
```

> **`site.yml` is NOT everything.** It covers the core bring-up only. Two node-host
> layers live OUTSIDE `site.yml` and must be run separately:
> ```
> ansible-playbook playbooks/kube-vip.yml            # control-plane VIP (192.168.1.49); targets k3s_servers
> ansible-playbook playbooks/vpn.egress.gateway.yml  # node-host OpenVPN split-tunnel + kill-switch; targets k3s-dev-3
> ```

## Re-provisioning a node (tofu -replace / fresh OS)
A node rebuild wipes **everything on the host filesystem** — including host-level config
that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
node-host layers or the node comes back missing pieces:

| Re-run | Why it doesn't survive a rebuild |
|---|---|
| `node-baseline.yml` | packages (incl. `cryptsetup`), kernel modules, Longhorn disk, swap-off |
| `k3s.yml` | the k3s server binary + config + join (secondaries re-join the VIP) |
| `kube-vip.yml` | the kube-vip auto-deploy manifest on the server's host path |
| `vpn.egress.gateway.yml` | **the OpenVPN (PIA) client, `tun0`, systemd units + nftables kill-switch on k3s-dev-3 — miss this and `egress-class=vpn` pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet** |

Everything Flux owns (platform + apps) self-heals once the node re-joins; only the
node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3,
`systemctl is-active vpn-egress-tunnel.service` and `curl --interface tun0 https://ipinfo.io/ip`
should return a PIA exit IP (not the home ISP).

## What each layer does
| Playbook / role | Action |
|---|---|
| `k3s.yml` / `k3s_server` | k3s server install with `--flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb`; primary `--cluster-init`, secondaries join via the primary's node-token (fetched, never pre-stored) |
| `cilium.yml` / `cilium` | Gateway API CRDs, then Helm `cilium` with `kubeProxyReplacement`, `k8sServiceHost/Port`, `gatewayAPI`, `egressGateway`, `hubble`, `l2announcements` |
| `verify.yml` / `cluster_verify` | waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
| `flux-bootstrap.yml` / `flux_bootstrap` | calls `bootstrap/` (SOPS age key, then `flux bootstrap --path=clusters/dev`) |

## Cilium ownership / parity contract (docs/02)
Ansible installs Cilium; the Flux `platform/cilium` HelmRelease adopts config
afterward. The parity-critical values **must match** between this role's
`templates/cilium-values.yaml.j2` and that HelmRelease:
- `kubeProxyReplacement: true`
- `k8sServiceHost: {{ k3s_api_host }}` (= primary node IP, `10.73.0.11` by default)
- `k8sServicePort: 6443`

The HelmRelease currently has `k8sServiceHost: "REPLACE-ME"` — WS-NET must set it
to the same value this role uses (surfaced by `tofu output primary_node_ip`).

## Interface assumed of `bootstrap/` (authored by other workers, NOT here)
`roles/flux_bootstrap` invokes two entrypoints via overridable command vars:
- `flux_bootstrap_sops_cmd` → `bootstrap/sops/install-age-key.sh` (Security
  Engineer, T-1.4): installs the age **private** key as the `sops-age` Secret in
  `flux-system`. Key never committed (`.gitignore` guards `age.key`/`*.key`).
- `flux_bootstrap_flux_cmd` → `bootstrap/flux/bootstrap.sh` (GitOps Engineer,
  T-1.4): runs `flux bootstrap ... --path=clusters/dev`.

**These script names/args are assumptions** — if the bootstrap deliverables
expose different entrypoints, override the two vars in
`roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged
for PM reconciliation.

## Static validation (PM, in dev container — NOT run here)
```
yamllint -c ../../ci/yamllint.yaml infra/ansible
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
ansible-lint playbooks/ roles/          # if the profile is enforced for infra/
```