home-network-k3s-impl/infra/ansible/README.md

# infra/ansible — k3s HA + Cilium bootstrap (Phase 1 T-1.3)

The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
CNI, verifies readiness, then invokes the `bootstrap/` entrypoints.

> **LIVE.** These plays have been executed against the running dev cluster (k3s
> v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the
> operational bring-up + re-provision runbook. See **Re-provisioning a node** below.

Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s), ADR-0002 (Cilium),
ADR-0004 (Flux hand-off), ADR-0007 (SOPS).

## Prerequisites
1. `tofu apply` in `../tofu/` has created the VMs; cloud-init (`../cloud-init/`)
   has baselined them (modules, sysctls, Longhorn disk, SSH user).
2. Inventory reflects the node IPs. `inventory/hosts.ini` ships aligned with the
   LOCKED defaults; regenerate from tofu if overridden:
   `cd ../tofu && tofu output -raw ansible_inventory > ../ansible/inventory/hosts.ini`
3. Run from the dev container (has `kubectl` + `helm`, used by the Cilium/verify/
   Flux plays via `delegate_to: localhost`).

## Run order
```
cd infra/ansible
ansible-playbook playbooks/site.yml        # core bring-up, in order (baseline→k3s→cilium→verify→flux)
# — or step by step —
ansible-playbook playbooks/node-baseline.yml    # 0. packages, kernel modules, Longhorn disk, swap-off
ansible-playbook playbooks/k3s.yml         # 1. HA k3s (primary --cluster-init, then joins)
ansible-playbook playbooks/etcd-io-priority.yml # 1b. shared etcd/Longhorn device I/O weights (PROD ONLY; a no-op on dev)
ansible-playbook playbooks/cilium.yml      # 2. Cilium CNI (cluster is NotReady until this)
ansible-playbook playbooks/verify.yml      # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOPS then Flux)
```

> **`site.yml` is NOT everything.** It covers the core bring-up only. Two node-host
> layers live OUTSIDE `site.yml` and must be run separately:
> ```
> ansible-playbook playbooks/kube-vip.yml            # control-plane VIP (192.168.1.49); defaults to k3s_servers
> ansible-playbook playbooks/vpn.egress.gateway.yml  # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3
> ```
> (Both take the same target-group override as the rest — see **Targeting prod** below.)

> **Standalone diagnostics** (not part of any bring-up; safe to point anywhere):
> ```
> ansible-playbook playbooks/node-health.yml   -e node_health_hosts=k3s_prod   # read-only hardware health
> ansible-playbook playbooks/memattr-survey.yml                                 # EFI memattr reboot survey
> ```
> `memattr-survey.yml` is the only play here that **reboots** its targets, and it will not do so
> until `-e memattr_survey_reboots_authorised=true` is passed. See
> `roles/memattr_survey/README.md` and `infra/PROD-PROVISIONING.md` §0b.

## Re-provisioning a node (tofu -replace / fresh OS)
A node rebuild wipes **everything on the host filesystem** — including host-level config
that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
node-host layers or the node comes back missing pieces:

| Re-run | Why it doesn't survive a rebuild |
|---|---|
| `node-baseline.yml` | packages (incl. `cryptsetup`), kernel modules, Longhorn disk, swap-off |
| `k3s.yml` | the k3s server binary + config + join (secondaries re-join the VIP) |
| `kube-vip.yml` | the kube-vip auto-deploy manifest on the server's host path |
| `vpn.egress.gateway.yml` | **the OpenVPN (PIA) client, `tun0`, systemd units + nftables kill-switch on k3s-dev-3 — miss this and `egress-class=vpn` pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet** |

Everything Flux owns (platform + apps) self-heals once the node re-joins; only the
node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3,
`systemctl is-active vpn-egress-tunnel.service` and `curl --interface tun0 https://ipinfo.io/ip`
should return a PIA exit IP (not the home ISP).

## What each layer does
| Playbook / role | Action |
|---|---|
| `k3s.yml` / `k3s_server` | k3s server install with `--flannel-backend=none --disable-kube-proxy --disable-network-policy --disable=traefik,servicelb`; primary `--cluster-init`, secondaries join via the primary's node-token (fetched, never pre-stored) |
| `cilium.yml` / `cilium` | Gateway API CRDs, then Helm `cilium` with `kubeProxyReplacement`, `k8sServiceHost/Port`, `gatewayAPI`, `egressGateway`, `hubble`, `l2announcements` |
| `etcd-io-priority.yml` / `etcd_io_priority` | **prod only** (master switch defaults false; `group_vars/k3s_prod` sets it true). Switches the disk carrying `k3s_data_dir` to **BFQ**, drops `nr_requests` to 32, and gives `system.slice` + `k3s.service` `IOWeight=10000` so the embedded etcd outweighs `kubepods.slice` — then reads every value back out of the kernel and **fails** if it did not land. Mitigation for the 2026-09-07 k3s-prod-1 etcd WAL-fsync starvation. Restarts nothing. NVMe-backed nodes are skipped, not decorated. See `roles/etcd_io_priority/README.md` for the mechanisms in this space that are **inert** here and must not be "fixed" back in |
| `verify.yml` / `cluster_verify` | waits nodes Ready + Cilium/Hubble rollout; asserts kube-proxy absent |
| `flux-bootstrap.yml` / `flux_bootstrap` | calls `bootstrap/` (SOPS age key, then `flux bootstrap --path=clusters/dev`) |

## Cilium ownership / parity contract (docs/02)
Ansible installs Cilium; the Flux `platform/cilium` HelmRelease adopts config
afterward. The parity-critical values **must match** between this role's
`templates/cilium-values.yaml.j2` and that HelmRelease:
- `kubeProxyReplacement: true`
- `k8sServiceHost: {{ k3s_apiserver_endpoint }}` — **the kube-vip CONTROL-PLANE VIP**
  (`= kube_vip_address`: dev `192.168.1.49`, prod `192.168.2.9`), **not** `k3s_api_host`
- `k8sServicePort: {{ k3s_api_port }}` (6443 in both environments)

### Two API addresses, split 2026-09-06 — do not re-merge them
`k3s_api_host` and `k3s_apiserver_endpoint` were one variable, and the conflation put every
prod `cilium-agent` on `192.168.2.10` (`k3s-prod-1`'s own address) on a cluster whose point is
HA — verified live. Lose that node and the datapath keeps serving from Cilium's cache while no
service or policy update ever lands, which is a silent failure, not an outage.

| variable | value | consumed by | why it must be that address |
|---|---|---|---|
| `k3s_api_host` | the founder's own node IP (dev `192.168.1.50`, prod `192.168.2.10`) | `roles/k3s_server`: the secondary `--server https://…:6443` join URL, and one `k3s_tls_sans` entry | on a **cold** build no VIP exists yet (kube-vip is a k3s auto-deploy manifest, so it needs a cluster first). The founder just ran `--cluster-init` and is up |
| `k3s_apiserver_endpoint` | `{{ kube_vip_address }}` (dev `192.168.1.49`, prod `192.168.2.9`) | `roles/cilium` → `k8sServiceHost`; any steady-state client | it must survive losing any single server |

Consequences already handled: `playbooks/site.yml` imports `kube-vip.yml` **between** `k3s.yml`
and `cilium.yml` (the VIP has to be floating before the CNI dials it), and `roles/kube_vip`'s
last task blocks until the VIP answers on 6443. `k3s_tls_sans` already carried both addresses,
so no cert reissue and no k3s restart are involved.

The Flux side of the same contract is `${K3S_API_HOST}` in `platform/cilium/helmrelease.yaml`,
set per overlay in `clusters/<env>/kustomization.yaml`; `[parity]` means it must equal
`k3s_apiserver_endpoint` for the SAME environment. Asserted by
`conformance/tests/l0-static/ha-control-plane-as-code`.

## Interface assumed of `bootstrap/` (authored by other workers, NOT here)
`roles/flux_bootstrap` invokes two entrypoints via overridable command vars:
- `flux_bootstrap_sops_cmd` → `bootstrap/sops/install-age-key.sh` (Security
  Engineer, T-1.4): installs the age **private** key as the `sops-age` Secret in
  `flux-system`. Key never committed (`.gitignore` guards `age.key`/`*.key`).
- `flux_bootstrap_flux_cmd` → `bootstrap/flux/bootstrap.sh` (GitOps Engineer,
  T-1.4): runs `flux bootstrap ... --path=clusters/dev`.

**These script names/args are assumptions** — if the bootstrap deliverables
expose different entrypoints, override the two vars in
`roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged
for PM reconciliation.

## Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)

`inventory/hosts.ini` carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
`[k3s_prod:children]` section for the three bare-metal nodes — `k3s-prod-1` `192.168.2.10`,
`k3s-prod-2` `.11`, `k3s-prod-3` `.12` — with its own `group_vars/k3s_prod/` (vaulted
`ansible_become_password` + non-secret overrides).

**All three are live in the inventory.** This paragraph used to say `k3s-prod-2`/`-3` "are
addressed but not yet racked, so their inventory lines stay commented out"; that stopped being
true on 2026-09-03 when node 3 came online, and the group has been complete since. Corrected
2026-09-07 because the stale sentence is the kind that gets used to reason about blast radius:
`-e k3s_hosts=k3s_prod` targets **three** hosts, not one. Confirm it, every time, before running
anything:

```bash
ansible-playbook playbooks/<play>.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3 --list-hosts
```

**Nothing here is forked for prod.** Every node-host play takes its target group from a variable
that defaults to the dev group, so the dev commands above are unchanged, and prod is the same
command plus an override. Two separate mechanisms, for two separate reasons:

| Layer | Knob | Set via | Why |
|---|---|---|---|
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts`, `vpn_egress_gateway_hosts` | `-e` **extra vars only** | Ansible templates a play's `hosts:` *before* it has any host, so it cannot see `group_vars` at all |
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group` (+ `cluster_kubeconfig_name`) | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml` | Host context exists by then, so each environment answers for itself with **no per-run flag** — same mechanism `k3s_api_host` already used |

`playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
bring-up chain.
`playbooks/memattr-survey.yml` follows it (`-e memattr_survey_hosts=…`, defaulting to `k3s_prod`)
for the same reason, and adds a second interlock on top: because it **reboots** its targets, the
role also refuses any host that is not a member of `memattr_survey_required_group`. A host pattern
can be mistyped; inventory membership cannot.

```bash
# dev — completely unchanged, no extra vars
ansible-playbook playbooks/site.yml

# prod — one flag for the whole-cluster plays
ansible-playbook playbooks/node-baseline.yml   -e k3s_hosts=k3s_prod
ansible-playbook playbooks/cilium.yml          -e k3s_hosts=k3s_prod
ansible-playbook playbooks/kube-vip.yml        -e k3s_hosts=k3s_prod
ansible-playbook playbooks/verify.yml          -e k3s_hosts=k3s_prod
ansible-playbook playbooks/flux-bootstrap.yml  -e k3s_hosts=k3s_prod   # see caveat below

# prod — I/O arbitration for the shared etcd/Longhorn disk. ONE NODE AT A TIME, node 3 first
# (WD Blue, IO PSI 1.29 during the incident) before node 1 (the 2011 C400, 11.36).
ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3 --list-hosts
ansible-playbook playbooks/etcd-io-priority.yml -e k3s_hosts=k3s_prod --limit k3s-prod-3

# prod — k3s.yml needs both halves of the primary/secondary split
ansible-playbook playbooks/k3s.yml \
  -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary

# prod — the whole chain (extra vars are global, so they reach imported playbooks)
ansible-playbook playbooks/site.yml \
  -e k3s_hosts=k3s_prod \
  -e k3s_primary_hosts=k3s_prod_primary \
  -e k3s_secondary_hosts=k3s_prod_secondary
```

What the roles then resolve **by themselves** for a prod host, from `group_vars/k3s_prod/vars.yml`:
`k3s_primary_group=k3s_prod_primary` (the `--cluster-init` founder, the node-token `delegate_to`
target, and the node Cilium fetches the kubeconfig from), `k3s_servers_group=k3s_prod` (the tls-san
node-IP list), `k3s_api_host=192.168.2.10` (the founder, bootstrap-join only),
`kube_vip_address=192.168.2.9` — and therefore `k3s_apiserver_endpoint=192.168.2.9`, which is what
Cilium dials — and
`cluster_kubeconfig_name=prod.kubeconfig` — so a prod run can never overwrite the
`.cache/dev.kubeconfig` that `infra/devcontainer/dev.sh` exports as `$KUBECONFIG`.

### Packages-only baseline (`--tags packages`)
`roles/node_baseline` is tagged `packages` / `system` / `disks`. An untagged run is unchanged (all
three, in order). `--tags packages` runs **only** the apt task:

```bash
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
```

This exists to break a circular dependency: prod's disk layout (`infra/PROD-PROVISIONING.md` §2) is
blocked on a SMART read, SMART needs `smartmontools`, `smartmontools` ships in
`node_baseline_packages` — but the role's disk tasks need block devices that decision hasn't chosen
yet. With `--tags packages` no `disks`-tagged task is selected, so the deliberate
`/dev/BLOCKED-pending-disk-decision-…` placeholders in `group_vars/k3s_prod/vars.yml` are never
even templated, let alone handed to `mkfs`. A **full** run against a node whose devices are still
placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never
hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).

### Still not runnable end to end
- The **disk-layout** (§2) and the `node_baseline_*_device` overrides remain BLOCKED placeholders.
- `group_vars/k3s_prod/vault.yml` (the become password) does not exist yet, so **no** prod play can
  escalate — and `--check` against prod fails for the same reason.
- `flux-bootstrap.yml` retargets, but `roles/flux_bootstrap` shells out to
  `bootstrap/flux/bootstrap.sh`, which bootstraps `--path=clusters/dev`. Prod needs
  `--path=clusters/prod`: a `bootstrap/` change, not a target-group change (§7).
- `vpn.egress.gateway.yml` is parameterised (`-e vpn_egress_gateway_hosts=…`) but prod has **no**
  egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own
  vaulted PIA credentials.
- The operator has ruled the prod cluster stays **uninitialized** until all three nodes are present
  (§8). `192.168.2.20` is prod's *ingress* Gateway VIP — a Flux/`cluster-vars` concern, not an
  Ansible one; the kube-vip **control-plane** VIP is `192.168.2.9` (`kube_vip_address`).

Everything above this section describes the **dev** runbook; the values in it (`192.168.1.x`,
`.cache/dev.kubeconfig`, `clusters/dev`) are dev's, supplied by `group_vars`, not baked into the
roles.

## Static validation (PM, in dev container — NOT run here)
```
yamllint -c ../../ci/yamllint.yaml infra/ansible
ansible-playbook -i inventory/hosts.ini --syntax-check playbooks/site.yml
ansible-lint playbooks/ roles/          # if the profile is enforced for infra/
```