home-network-k3s-impl/infra/ansible/README.md

# infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)

The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
CNI, verifies readiness, then invokes the `bootstrap/` entrypoints.

> **LIVE.** These plays have been executed against the running dev cluster (k3s
> v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the
> operational bring-up + re-provision runbook. See **Re-provisioning a node** below.

Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s), ADR-0002 (Cilium),
ADR-0004 (Flux hand-off), ADR-0007 (SOPS).

## Prerequisites
1. `tofu apply` in `../tofu/` has created the VMs; cloud-init (`../cloud-init/`)
   has baselined them (modules, sysctls, Longhorn disk, SSH user).
2. Inventory reflects the node IPs. `inventory/hosts.ini` ships aligned with the
   LOCKED defaults; regenerate from tofu if overridden:
   `cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini`
3. Run from the dev container (has `kubectl` + `helm`, used by the Cilium/verify/
   Flux plays via `delegate_to: localhost`).

## Run order
```
cd infra/ansible
ansible-playbook playbooks/site.yml        # core bring-up, in order (baseline→k3s→cilium→verify→flux)
# — or step by step —
ansible-playbook playbooks/node-baseline.yml    # 0. packages, kernel modules, Longhorn disk, swap-off
ansible-playbook playbooks/k3s.yml         # 1. HA k3s (primary --cluster-init, then joins)
ansible-playbook playbooks/cilium.yml      # 2. Cilium CNI (cluster is NotReady until this)
ansible-playbook playbooks/verify.yml      # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOPS then Flux)
```

> **`site.yml` is NOT everything.** It covers the core bring-up only. Two node-host
> layers live OUTSIDE `site.yml` and must be run separately:
> ```
> ansible-playbook playbooks/kube-vip.yml            # control-plane VIP (192.168.1.49); defaults to k3s_servers
> ansible-playbook playbooks/vpn.egress.gateway.yml  # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3
> ```
> (Both take the same target-group override as the rest — see **Targeting prod** below.)

## Re-provisioning a node (tofu -replace / fresh OS)
A node rebuild wipes **everything on the host filesystem** — including host-level config
that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
node-host layers or the node comes back missing pieces:

| Re-run | Why it doesn't survive a rebuild |
|---|---|
| `node-baseline.yml` | packages (incl. `cryptsetup`), kernel modules, Longhorn disk, swap-off |
| `k3s.yml` | the k3s server binary + config + join (secondaries re-join the VIP) |
| `kube-vip.yml` | the kube-vip auto-deploy manifest on the server's host path |
| `vpn.egress.gateway.yml` | **the OpenVPN (PIA) client, `tun0`, systemd units + nftables kill-switch on k3s-dev-3 — miss this and `egress-class=vpn` pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet** |

Everything Flux owns (platform + apps) self-heals once the node re-joins; only the
node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3,
`systemctl is-active vpn-egress-tunnel.service` and `curl --interface tun0 https://ipinfo.io/ip`
should return a PIA exit IP (not the home ISP).

## What each layer does
| Playbook / role | Action |
|---|---|
| `k3s.yml` / `k3s_server` | k3s server install with `--flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb`; primary `--cluster-init`, secondaries join via the primary's node-token (fetched, never pre-stored) |
| `cilium.yml` / `cilium` | Gateway API CRDs, then Helm `cilium` with `kubeProxyReplacement`, `k8sServiceHost/Port`, `gatewayAPI`, `egressGateway`, `hubble`, `l2announcements` |
| `verify.yml` / `cluster_verify` | waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
| `flux-bootstrap.yml` / `flux_bootstrap` | calls `bootstrap/` (SOPS age key, then `flux bootstrap --path=clusters/dev`) |

## Cilium ownership / parity contract (docs/02)
Ansible installs Cilium; the Flux `platform/cilium` HelmRelease adopts config
afterward. The parity-critical values **must match** between this role's
`templates/cilium-values.yaml.j2` and that HelmRelease:
- `kubeProxyReplacement: true`
- `k8sServiceHost: {{ k3s_api_host }}` (= primary node IP, `10.73.0.11` by default)
- `k8sServicePort: 6443`

The HelmRelease currently has `k8sServiceHost: "REPLACE-ME"` — WS-NET must set it
to the same value this role uses (surfaced by `tofu output primary_node_ip`).

## Interface assumed of `bootstrap/` (authored by other workers, NOT here)
`roles/flux_bootstrap` invokes two entrypoints via overridable command vars:
- `flux_bootstrap_sops_cmd` → `bootstrap/sops/install-age-key.sh` (Security
  Engineer, T-1.4): installs the age **private** key as the `sops-age` Secret in
  `flux-system`. Key never committed (`.gitignore` guards `age.key`/`*.key`).
- `flux_bootstrap_flux_cmd` → `bootstrap/flux/bootstrap.sh` (GitOps Engineer,
  T-1.4): runs `flux bootstrap ... --path=clusters/dev`.

**These script names/args are assumptions** — if the bootstrap deliverables
expose different entrypoints, override the two vars in
`roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged
for PM reconciliation.

## Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)

`inventory/hosts.ini` carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
`[k3s_prod:children]` section for the three planned bare-metal nodes (`k3s-prod-1` at
`192.168.2.10` exists and is reachable today; `k3s-prod-2`/`-3` at `.11`/`.12` are addressed but
not yet racked, so their inventory lines stay commented out), with its own `group_vars/k3s_prod/`
(vaulted `ansible_become_password` + non-secret overrides).

**Nothing here is forked for prod.** Every node-host play takes its target group from a variable
that defaults to the dev group, so the dev commands above are unchanged, and prod is the same
command plus an override. Two separate mechanisms, for two separate reasons:

| Layer | Knob | Set via | Why |
|---|---|---|---|
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts`, `vpn_egress_gateway_hosts` | `-e` **extra vars only** | Ansible templates a play's `hosts:` *before* it has any host, so it cannot see `group_vars` at all |
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group` (+ `cluster_kubeconfig_name`) | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml` | Host context exists by then, so each environment answers for itself with **no per-run flag** — same mechanism `k3s_api_host` already used |

`playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
bring-up chain.

```bash
# dev — completely unchanged, no extra vars
ansible-playbook playbooks/site.yml

# prod — one flag for the whole-cluster plays
ansible-playbook playbooks/node-baseline.yml   -e k3s_hosts=k3s_prod
ansible-playbook playbooks/cilium.yml          -e k3s_hosts=k3s_prod
ansible-playbook playbooks/kube-vip.yml        -e k3s_hosts=k3s_prod
ansible-playbook playbooks/verify.yml          -e k3s_hosts=k3s_prod
ansible-playbook playbooks/flux-bootstrap.yml  -e k3s_hosts=k3s_prod   # see caveat below

# prod — k3s.yml needs both halves of the primary/secondary split
ansible-playbook playbooks/k3s.yml \
  -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary

# prod — the whole chain (extra vars are global, so they reach imported playbooks)
ansible-playbook playbooks/site.yml \
  -e k3s_hosts=k3s_prod \
  -e k3s_primary_hosts=k3s_prod_primary \
  -e k3s_secondary_hosts=k3s_prod_secondary
```

What the roles then resolve **by themselves** for a prod host, from `group_vars/k3s_prod/vars.yml`:
`k3s_primary_group=k3s_prod_primary` (the `--cluster-init` founder, the node-token `delegate_to`
target, and the node Cilium fetches the kubeconfig from), `k3s_servers_group=k3s_prod` (the tls-san
node-IP list), `k3s_api_host=192.168.2.10`, `kube_vip_address=192.168.2.9`, and
`cluster_kubeconfig_name=prod.kubeconfig` — so a prod run can never overwrite the
`.cache/dev.kubeconfig` that `infra/devcontainer/dev.sh` exports as `$KUBECONFIG`.

### Packages-only baseline (`--tags packages`)
`roles/node_baseline` is tagged `packages` / `system` / `disks`. An untagged run is unchanged (all
three, in order). `--tags packages` runs **only** the apt task:

```bash
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
```

This exists to break a circular dependency: prod's disk layout (`infra/PROD-PROVISIONING.md` §2) is
blocked on a SMART read, SMART needs `smartmontools`, `smartmontools` ships in
`node_baseline_packages` — but the role's disk tasks need block devices that decision hasn't chosen
yet. With `--tags packages` no `disks`-tagged task is selected, so the deliberate
`/dev/BLOCKED-pending-disk-decision-…` placeholders in `group_vars/k3s_prod/vars.yml` are never
even templated, let alone handed to `mkfs`. A **full** run against a node whose devices are still
placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never
hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).

### Still not runnable end to end
- The **disk-layout** (§2) and the `node_baseline_*_device` overrides remain BLOCKED placeholders.
- `group_vars/k3s_prod/vault.yml` (the become password) does not exist yet, so **no** prod play can
  escalate — and `--check` against prod fails for the same reason.
- `flux-bootstrap.yml` retargets, but `roles/flux_bootstrap` shells out to
  `bootstrap/flux/bootstrap.sh`, which bootstraps `--path=clusters/dev`. Prod needs
  `--path=clusters/prod`: a `bootstrap/` change, not a target-group change (§7).
- `vpn.egress.gateway.yml` is parameterised (`-e vpn_egress_gateway_hosts=…`) but prod has **no**
  egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own
  vaulted PIA credentials.
- The operator has ruled the prod cluster stays **uninitialized** until all three nodes are present
  (§8). `192.168.2.20` is prod's *ingress* Gateway VIP — a Flux/`cluster-vars` concern, not an
  Ansible one; the kube-vip **control-plane** VIP is `192.168.2.9` (`kube_vip_address`).

Everything above this section describes the **dev** runbook; the values in it (`192.168.1.x`,
`.cache/dev.kubeconfig`, `clusters/dev`) are dev's, supplied by `group_vars`, not baked into the
roles.

## Static validation (PM, in dev container — NOT run here)
```
yamllint -c ../../ci/yamllint.yaml infra/ansible
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
ansible-lint playbooks/ roles/          # if the profile is enforced for infra/
```