feat(ansible): parameterise target groups so prod reuses the roles unchanged (R-ENV-02)

roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and blocked PROD-PROVISIONING.md §6 outright. TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host context exists, so group_vars is unreadable there, while roles run WITH host context: * which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts (k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern. * which group a ROLE means by "my primary"/"my servers" — k3s_primary_group, k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod. Follows the k3s_api_host convention already in this repo. No per-run flag. DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks differs in exactly two files, only by added TAGS annotations and one new assert; no task reordered, renamed or removed. Var resolution verified per environment with -c local: dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod -> k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12. BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged packages/system/disks, so the package install can run alone: ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages Skipping is the guard — no disks-tagged task is selected, so the BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs. Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2 explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used (it would change the default run). FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded .cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG. Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it, leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name (dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical. KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7). vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need its own vaulted PIA credentials. --check against prod is still impossible until the prod vault exists — even the packages-only run needs it. Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus debug/assert with -c local, only. Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-31 23:53:36 -05:00
1 parent 1157c1b071
commit e623d27e2d
@@ -35,9 +35,10 @@ ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOP
 > **`site.yml` is NOT everything.** It covers the core bring-up only. Two node-host
 > layers live OUTSIDE `site.yml` and must be run separately:
 > ```
-> ansible-playbook playbooks/kube-vip.yml            # control-plane VIP (192.168.1.49); targets k3s_servers
-> ansible-playbook playbooks/vpn.egress.gateway.yml  # node-host OpenVPN split-tunnel + kill-switch; targets k3s-dev-3
+> ansible-playbook playbooks/kube-vip.yml            # control-plane VIP (192.168.1.49); defaults to k3s_servers
+> ansible-playbook playbooks/vpn.egress.gateway.yml  # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3
 > ```
+> (Both take the same target-group override as the rest — see **Targeting prod** below.)

 ## Re-provisioning a node (tofu -replace / fresh OS)
 A node rebuild wipes **everything on the host filesystem** — including host-level config
@@ -88,24 +89,90 @@ expose different entrypoints, override the two vars in
 `roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged
 for PM reconciliation.

-## Prod inventory — scaffolding only, Phase 7 PREP (nothing above is run against it)
+## Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)

-`inventory/hosts.ini` also carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
+`inventory/hosts.ini` carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
 `[k3s_prod:children]` section for the three planned bare-metal nodes (`k3s-prod-1` at
 `192.168.2.10` exists and is reachable today; `k3s-prod-2`/`-3` at `.11`/`.12` are addressed but
 not yet racked, so their inventory lines stay commented out), with its own `group_vars/k3s_prod/`
-(vaulted `ansible_become_password` + non-secret overrides). Note `192.168.2.20` is prod's
-**ingress** Gateway VIP (a Flux/`cluster-vars` concern, not an Ansible one — it does not appear in
-`group_vars/k3s_prod/`); the kube-vip **control-plane** VIP (`kube_vip_address`) is a *separate*,
-still-undecided address, left as a loud BLOCKED placeholder alongside the `node_baseline` device
-overrides. See `group_vars/k3s_prod/README.md` and `infra/PROD-PROVISIONING.md` (T-7.1) §4/§2 for
-the open decisions that still block a real run.
+(vaulted `ansible_become_password` + non-secret overrides).

-Everything else in this README describes the **dev** runbook. It does not (yet) apply to prod
-as-is: the run order above and the roles it calls (`k3s_server`, `cluster_verify`, ...) target the
-literal `k3s_primary` / `k3s_secondary` / `k3s_servers` groups, not `k3s_prod_*` — see the GAP note
-at the top of `inventory/hosts.ini`'s prod section for what's still missing before that reuse (the
-R-ENV-02 promise) actually works end to end.
+**Nothing here is forked for prod.** Every node-host play takes its target group from a variable
+that defaults to the dev group, so the dev commands above are unchanged, and prod is the same
+command plus an override. Two separate mechanisms, for two separate reasons:
+
+| Layer | Knob | Set via | Why |
+|---|---|---|---|
+| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts`, `vpn_egress_gateway_hosts` | `-e` **extra vars only** | Ansible templates a play's `hosts:` *before* it has any host, so it cannot see `group_vars` at all |
+| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group` (+ `cluster_kubeconfig_name`) | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml` | Host context exists by then, so each environment answers for itself with **no per-run flag** — same mechanism `k3s_api_host` already used |
+
+`playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
+keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
+bring-up chain.
+
+```bash
+# dev — completely unchanged, no extra vars
+ansible-playbook playbooks/site.yml
+
+# prod — one flag for the whole-cluster plays
+ansible-playbook playbooks/node-baseline.yml   -e k3s_hosts=k3s_prod
+ansible-playbook playbooks/cilium.yml          -e k3s_hosts=k3s_prod
+ansible-playbook playbooks/kube-vip.yml        -e k3s_hosts=k3s_prod
+ansible-playbook playbooks/verify.yml          -e k3s_hosts=k3s_prod
+ansible-playbook playbooks/flux-bootstrap.yml  -e k3s_hosts=k3s_prod   # see caveat below
+
+# prod — k3s.yml needs both halves of the primary/secondary split
+ansible-playbook playbooks/k3s.yml \
+  -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
+
+# prod — the whole chain (extra vars are global, so they reach imported playbooks)
+ansible-playbook playbooks/site.yml \
+  -e k3s_hosts=k3s_prod \
+  -e k3s_primary_hosts=k3s_prod_primary \
+  -e k3s_secondary_hosts=k3s_prod_secondary
+```
+
+What the roles then resolve **by themselves** for a prod host, from `group_vars/k3s_prod/vars.yml`:
+`k3s_primary_group=k3s_prod_primary` (the `--cluster-init` founder, the node-token `delegate_to`
+target, and the node Cilium fetches the kubeconfig from), `k3s_servers_group=k3s_prod` (the tls-san
+node-IP list), `k3s_api_host=192.168.2.10`, `kube_vip_address=192.168.2.9`, and
+`cluster_kubeconfig_name=prod.kubeconfig` — so a prod run can never overwrite the
+`.cache/dev.kubeconfig` that `infra/devcontainer/dev.sh` exports as `$KUBECONFIG`.
+
+### Packages-only baseline (`--tags packages`)
+`roles/node_baseline` is tagged `packages` / `system` / `disks`. An untagged run is unchanged (all
+three, in order). `--tags packages` runs **only** the apt task:
+
+```bash
+ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
+```
+
+This exists to break a circular dependency: prod's disk layout (`infra/PROD-PROVISIONING.md` §2) is
+blocked on a SMART read, SMART needs `smartmontools`, `smartmontools` ships in
+`node_baseline_packages` — but the role's disk tasks need block devices that decision hasn't chosen
+yet. With `--tags packages` no `disks`-tagged task is selected, so the deliberate
+`/dev/BLOCKED-pending-disk-decision-…` placeholders in `group_vars/k3s_prod/vars.yml` are never
+even templated, let alone handed to `mkfs`. A **full** run against a node whose devices are still
+placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never
+hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).
+
+### Still not runnable end to end
+- The **disk-layout** (§2) and the `node_baseline_*_device` overrides remain BLOCKED placeholders.
+- `group_vars/k3s_prod/vault.yml` (the become password) does not exist yet, so **no** prod play can
+  escalate — and `--check` against prod fails for the same reason.
+- `flux-bootstrap.yml` retargets, but `roles/flux_bootstrap` shells out to
+  `bootstrap/flux/bootstrap.sh`, which bootstraps `--path=clusters/dev`. Prod needs
+  `--path=clusters/prod`: a `bootstrap/` change, not a target-group change (§7).
+- `vpn.egress.gateway.yml` is parameterised (`-e vpn_egress_gateway_hosts=…`) but prod has **no**
+  egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own
+  vaulted PIA credentials.
+- The operator has ruled the prod cluster stays **uninitialized** until all three nodes are present
+  (§8). `192.168.2.20` is prod's *ingress* Gateway VIP — a Flux/`cluster-vars` concern, not an
+  Ansible one; the kube-vip **control-plane** VIP is `192.168.2.9` (`kube_vip_address`).
+
+Everything above this section describes the **dev** runbook; the values in it (`192.168.1.x`,
+`.cache/dev.kubeconfig`, `clusters/dev`) are dev's, supplied by `group_vars`, not baked into the
+roles.

 ## Static validation (PM, in dev container — NOT run here)
 ```