From e623d27e2d1ea5dae22780f6e5f5f53dff1b94b2 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 31 Aug 2026 23:53:36 -0500 Subject: [PATCH] feat(ansible): parameterise target groups so prod reuses the roles unchanged (R-ENV-02) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and blocked PROD-PROVISIONING.md §6 outright. TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host context exists, so group_vars is unreadable there, while roles run WITH host context: * which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts (k3s.yml's split), vpn_egress_gateway_hosts; each `| default()`, set by -e only. Follows node-health.yml's established `_hosts | default(dev)` pattern. * which group a ROLE means by "my primary"/"my servers" — k3s_primary_group, k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod. Follows the k3s_api_host convention already in this repo. No per-run flag. DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks differs in exactly two files, only by added TAGS annotations and one new assert; no task reordered, renamed or removed. Var resolution verified per environment with -c local: dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod -> k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12. BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged packages/system/disks, so the package install can run alone: ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages Skipping is the guard — no disks-tagged task is selected, so the BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs. Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2 explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used (it would change the default run). FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded .cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG. Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it, leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name (dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical. KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7). vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need its own vaulted PIA credentials. --check against prod is still impossible until the prod vault exists — even the packages-only run needs it. Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus debug/assert with -c local, only. Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) --- infra/PROD-PROVISIONING.md | 89 ++++++++++++++--- infra/ansible/README.md | 97 ++++++++++++++++--- infra/ansible/inventory/group_vars/all.yml | 25 ++++- .../inventory/group_vars/k3s_prod/README.md | 16 ++- .../inventory/group_vars/k3s_prod/vars.yml | 23 ++++- infra/ansible/inventory/hosts.ini | 28 +++--- infra/ansible/playbooks/cilium.yml | 7 +- infra/ansible/playbooks/flux-bootstrap.yml | 9 +- infra/ansible/playbooks/k3s.yml | 15 ++- infra/ansible/playbooks/kube-vip.yml | 9 +- infra/ansible/playbooks/node-baseline.yml | 15 ++- infra/ansible/playbooks/node-health.yml | 16 +-- infra/ansible/playbooks/site.yml | 12 +++ infra/ansible/playbooks/verify.yml | 5 +- .../ansible/playbooks/vpn.egress.gateway.yml | 12 ++- infra/ansible/roles/cilium/defaults/main.yml | 6 +- infra/ansible/roles/cilium/tasks/main.yml | 5 +- .../roles/cluster_verify/defaults/main.yml | 4 +- .../roles/flux_bootstrap/defaults/main.yml | 3 +- .../roles/k3s_server/defaults/main.yml | 6 +- infra/ansible/roles/k3s_server/tasks/main.yml | 10 +- .../roles/node_baseline/tasks/main.yml | 48 +++++++++ 22 files changed, 384 insertions(+), 76 deletions(-) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 0371ff5..a2e8386 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -84,7 +84,14 @@ Consequences to settle before imaging nodes 2 and 3: - **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one physical failure domain. - [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry. -- [ ] Record the chosen device paths here and set them in `inventory/group_vars/k3s_prod.yml`. +- [ ] Record the chosen device paths here and set them in + `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately + `/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full + `node-baseline` run against a node still carrying those placeholders now stops on an explicit + assert naming this section, instead of failing obscurely at `mkfs` time. +- [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline` + is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk + tasks this decision blocks — see §5's drive-health box for the exact command. ## 3. OS install (per node) @@ -149,29 +156,81 @@ Consequences to settle before imaging nodes 2 and 3: - [ ] **Drive health baseline** (§2). **Do not hand-install the tooling** — package installation is the `node_baseline` role's job, never a manual `apt-get` on a node (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo). `smartmontools` is part - of `node_baseline_packages`, so it arrives with the baseline play. Reading the drive is then: + of `node_baseline_packages`, so it arrives with the baseline play — but a *full* baseline + would hit the disk tasks §2 has not decided yet. Run the **packages-only** baseline instead + (the `disks`-tagged tasks, including the BLOCKED-device assert, are not selected at all, so + the placeholder device paths are never even templated): + ```bash + cd infra/ansible + ansible-playbook playbooks/node-baseline.yml \ + -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1 + ``` + (Needs `group_vars/k3s_prod/vault.yml` to exist first — the become password above.) Reading + the drive is then: ```bash sudo smartctl -a /dev/sda # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors ``` + Record the result in §2 and close the layout decision in the same change that replaces the + two `node_baseline_*_device` placeholders. ## 6. Ansible baseline & cluster join -Reuses the dev roles unchanged — that reuse **is** R-ENV-02. Only inventory and group_vars differ. +Reuses the dev roles **unchanged** — that reuse **is** R-ENV-02. Only inventory and group_vars +differ, plus a target-group override on the command line. -- [ ] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached +### How prod is targeted (done, 2026-08-31) + +Every node-host play used to hardcode the literal dev groups `k3s_servers` / `k3s_primary` / +`k3s_secondary`, so nothing here could select prod at all. Two knobs now close that, deliberately +kept separate because Ansible resolves them at different times: + +| Layer | Knob | Set via | +|---|---|---| +| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts` (each defaults to the dev group) | `-e` **extra vars only** — a play's `hosts:` is templated before any host exists, so `group_vars` is not readable there | +| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group`, `cluster_kubeconfig_name` | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml`; **no per-run flag** | + +So `roles/k3s_server` finds the prod `--cluster-init` founder, builds the prod join target and +tls-san list, and `roles/cilium` fetches the prod kubeconfig, with **no role fork**. Running with +no extra vars is byte-for-byte the dev behaviour it always had (verified by a before/after +`--list-hosts` diff on every playbook). Full detail: `infra/ansible/README.md` → **Targeting prod**. + +- [x] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached **directly** on the LAN — no NAT port-forward hop, unlike dev's `192.168.2.102:225x` mapping. -- [ ] `group_vars/k3s_prod.yml`: `kube_vip_address`, `cluster_controller_api_server`, and the §2 - device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`. -- [ ] Run `playbooks/node-baseline.yml` → packages, kernel modules, sysctls, iscsid, data-disk - filesystems + mounts. (`qemu-guest-agent` is a no-op on bare metal — the role already only - *starts* it when the virtio channel exists, so it degrades cleanly.) -- [ ] Run `playbooks/k3s.yml` → HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. - **Only once all three nodes are present** — see §8. -- [ ] Run `playbooks/kube-vip.yml` → control-plane VIP on the prod LAN. -- [ ] Run `playbooks/cilium.yml` → CNI at bootstrap; Flux's HelmRelease adopts it afterwards. The - bootstrap values and the HelmRelease **must stay in parity** (`kubeProxyReplacement`, - `k8sServiceHost`/`Port`) or the two layers fight over CNI config (`docs/02`). + (Nodes 2/3 stay commented out until they are physically on the bench.) +- [x] `inventory/group_vars/k3s_prod/vars.yml`: `kube_vip_address` (`192.168.2.9`), + `cluster_controller_api_server`, `k3s_api_host`, the group-name overrides above — **and** the + §2 device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`, which + are still BLOCKED placeholders. +- [ ] `inventory/group_vars/k3s_prod/vault.yml` — the vaulted `ansible_become_password` (§5). Until + it exists **no** prod play can escalate, and `--check` against prod cannot work either. +- [ ] Run the **packages-only** baseline first, to unblock the §2 drive-health read: + ```bash + ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages + ``` +- [ ] Run the **full** `playbooks/node-baseline.yml -e k3s_hosts=k3s_prod` → packages, kernel + modules, sysctls, iscsid, data-disk filesystems + mounts. **Blocked until §2 is decided**: it + asserts on the placeholder device paths and stops. (`qemu-guest-agent` is a no-op on bare + metal — the role already only *starts* it when the virtio channel exists, so it degrades + cleanly.) +- [ ] Run `playbooks/k3s.yml -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary` + → HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. **Only once all three + nodes are present** — see §8. +- [ ] Run `playbooks/kube-vip.yml -e k3s_hosts=k3s_prod` → control-plane VIP (`192.168.2.9`) on the + prod LAN. +- [ ] Run `playbooks/cilium.yml -e k3s_hosts=k3s_prod` → CNI at bootstrap; Flux's HelmRelease adopts + it afterwards. The bootstrap values and the HelmRelease **must stay in parity** + (`kubeProxyReplacement`, `k8sServiceHost`/`Port`) or the two layers fight over CNI config + (`docs/02`). Note the prod kubeconfig is fetched to `.cache/prod.kubeconfig`, never over the + dev one the dev container exports as `$KUBECONFIG`. +- [ ] Run `playbooks/verify.yml -e k3s_hosts=k3s_prod`. +- [ ] `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod` **retargets but is not yet correct**: + `roles/flux_bootstrap` shells out to `bootstrap/flux/bootstrap.sh`, which bootstraps + `--path=clusters/dev`. Prod needs `--path=clusters/prod` — a `bootstrap/` change, not a + target-group change. Tracked with the other repo blockers in §7. - [ ] Decide whether prod runs a VPN egress gateway node (dev uses `k3s-dev-3`, ADR-0015). + `playbooks/vpn.egress.gateway.yml` takes `-e vpn_egress_gateway_hosts=`, but there is + deliberately no prod egress group in the inventory yet, and it would need its **own** vaulted + PIA credentials — the existing vault is scoped to the dev group. ## 7. Repo blockers — **prod cannot be reconciled until these land** diff --git a/infra/ansible/README.md b/infra/ansible/README.md index 4324ca0..b0abb08 100644 --- a/infra/ansible/README.md +++ b/infra/ansible/README.md @@ -35,9 +35,10 @@ ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOP > **`site.yml` is NOT everything.** It covers the core bring-up only. Two node-host > layers live OUTSIDE `site.yml` and must be run separately: > ``` -> ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); targets k3s_servers -> ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; targets k3s-dev-3 +> ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); defaults to k3s_servers +> ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3 > ``` +> (Both take the same target-group override as the rest — see **Targeting prod** below.) ## Re-provisioning a node (tofu -replace / fresh OS) A node rebuild wipes **everything on the host filesystem** — including host-level config @@ -88,24 +89,90 @@ expose different entrypoints, override the two vars in `roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged for PM reconciliation. -## Prod inventory — scaffolding only, Phase 7 PREP (nothing above is run against it) +## Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides) -`inventory/hosts.ini` also carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` / +`inventory/hosts.ini` carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` / `[k3s_prod:children]` section for the three planned bare-metal nodes (`k3s-prod-1` at `192.168.2.10` exists and is reachable today; `k3s-prod-2`/`-3` at `.11`/`.12` are addressed but not yet racked, so their inventory lines stay commented out), with its own `group_vars/k3s_prod/` -(vaulted `ansible_become_password` + non-secret overrides). Note `192.168.2.20` is prod's -**ingress** Gateway VIP (a Flux/`cluster-vars` concern, not an Ansible one — it does not appear in -`group_vars/k3s_prod/`); the kube-vip **control-plane** VIP (`kube_vip_address`) is a *separate*, -still-undecided address, left as a loud BLOCKED placeholder alongside the `node_baseline` device -overrides. See `group_vars/k3s_prod/README.md` and `infra/PROD-PROVISIONING.md` (T-7.1) §4/§2 for -the open decisions that still block a real run. +(vaulted `ansible_become_password` + non-secret overrides). -Everything else in this README describes the **dev** runbook. It does not (yet) apply to prod -as-is: the run order above and the roles it calls (`k3s_server`, `cluster_verify`, ...) target the -literal `k3s_primary` / `k3s_secondary` / `k3s_servers` groups, not `k3s_prod_*` — see the GAP note -at the top of `inventory/hosts.ini`'s prod section for what's still missing before that reuse (the -R-ENV-02 promise) actually works end to end. +**Nothing here is forked for prod.** Every node-host play takes its target group from a variable +that defaults to the dev group, so the dev commands above are unchanged, and prod is the same +command plus an override. Two separate mechanisms, for two separate reasons: + +| Layer | Knob | Set via | Why | +|---|---|---|---| +| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts`, `vpn_egress_gateway_hosts` | `-e` **extra vars only** | Ansible templates a play's `hosts:` *before* it has any host, so it cannot see `group_vars` at all | +| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group` (+ `cluster_kubeconfig_name`) | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml` | Host context exists by then, so each environment answers for itself with **no per-run flag** — same mechanism `k3s_api_host` already used | + +`playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it +keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the +bring-up chain. + +```bash +# dev — completely unchanged, no extra vars +ansible-playbook playbooks/site.yml + +# prod — one flag for the whole-cluster plays +ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod +ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod +ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod +ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod +ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod # see caveat below + +# prod — k3s.yml needs both halves of the primary/secondary split +ansible-playbook playbooks/k3s.yml \ + -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary + +# prod — the whole chain (extra vars are global, so they reach imported playbooks) +ansible-playbook playbooks/site.yml \ + -e k3s_hosts=k3s_prod \ + -e k3s_primary_hosts=k3s_prod_primary \ + -e k3s_secondary_hosts=k3s_prod_secondary +``` + +What the roles then resolve **by themselves** for a prod host, from `group_vars/k3s_prod/vars.yml`: +`k3s_primary_group=k3s_prod_primary` (the `--cluster-init` founder, the node-token `delegate_to` +target, and the node Cilium fetches the kubeconfig from), `k3s_servers_group=k3s_prod` (the tls-san +node-IP list), `k3s_api_host=192.168.2.10`, `kube_vip_address=192.168.2.9`, and +`cluster_kubeconfig_name=prod.kubeconfig` — so a prod run can never overwrite the +`.cache/dev.kubeconfig` that `infra/devcontainer/dev.sh` exports as `$KUBECONFIG`. + +### Packages-only baseline (`--tags packages`) +`roles/node_baseline` is tagged `packages` / `system` / `disks`. An untagged run is unchanged (all +three, in order). `--tags packages` runs **only** the apt task: + +```bash +ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1 +``` + +This exists to break a circular dependency: prod's disk layout (`infra/PROD-PROVISIONING.md` §2) is +blocked on a SMART read, SMART needs `smartmontools`, `smartmontools` ships in +`node_baseline_packages` — but the role's disk tasks need block devices that decision hasn't chosen +yet. With `--tags packages` no `disks`-tagged task is selected, so the deliberate +`/dev/BLOCKED-pending-disk-decision-…` placeholders in `group_vars/k3s_prod/vars.yml` are never +even templated, let alone handed to `mkfs`. A **full** run against a node whose devices are still +placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never +hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31). + +### Still not runnable end to end +- The **disk-layout** (§2) and the `node_baseline_*_device` overrides remain BLOCKED placeholders. +- `group_vars/k3s_prod/vault.yml` (the become password) does not exist yet, so **no** prod play can + escalate — and `--check` against prod fails for the same reason. +- `flux-bootstrap.yml` retargets, but `roles/flux_bootstrap` shells out to + `bootstrap/flux/bootstrap.sh`, which bootstraps `--path=clusters/dev`. Prod needs + `--path=clusters/prod`: a `bootstrap/` change, not a target-group change (§7). +- `vpn.egress.gateway.yml` is parameterised (`-e vpn_egress_gateway_hosts=…`) but prod has **no** + egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own + vaulted PIA credentials. +- The operator has ruled the prod cluster stays **uninitialized** until all three nodes are present + (§8). `192.168.2.20` is prod's *ingress* Gateway VIP — a Flux/`cluster-vars` concern, not an + Ansible one; the kube-vip **control-plane** VIP is `192.168.2.9` (`kube_vip_address`). + +Everything above this section describes the **dev** runbook; the values in it (`192.168.1.x`, +`.cache/dev.kubeconfig`, `clusters/dev`) are dev's, supplied by `group_vars`, not baked into the +roles. ## Static validation (PM, in dev container — NOT run here) ``` diff --git a/infra/ansible/inventory/group_vars/all.yml b/infra/ansible/inventory/group_vars/all.yml index 91d6307..bd3a241 100644 --- a/infra/ansible/inventory/group_vars/all.yml +++ b/infra/ansible/inventory/group_vars/all.yml @@ -4,12 +4,35 @@ # cloud-init created this login user (tofu var login_user); it has NOPASSWD sudo. ansible_user: k3s +# --- environment group NAMES (R-ENV-02: one set of roles, scoped overrides) --------- +# The GROUP NAMES a role uses to answer "who is MY environment's primary / MY +# environment's servers?". Roles must never hardcode `groups['k3s_primary']`, or they +# only ever work against dev. These two are the dev values; group_vars/k3s_prod/vars.yml +# overrides them with k3s_prod_primary / k3s_prod, so a host in the prod group resolves +# its own primary automatically — exactly the way k3s_api_host below is overridden there. +# +# These are GROUP NAMES (keys into `groups`), NOT host patterns. The play-level target +# pattern is a separate, extra-vars-only knob (`k3s_hosts` / `k3s_primary_hosts` / +# `k3s_secondary_hosts`, see playbooks/*.yml) because Ansible templates a play's `hosts:` +# before it has any host context and so cannot see group_vars at all. +k3s_primary_group: k3s_primary +k3s_servers_group: k3s_servers + +# Filename of the kubeconfig the controller-side roles (cilium, cluster_verify, +# flux_bootstrap) fetch from the primary into infra/ansible/.cache/. Scoped per +# environment so pointing those plays at prod can NEVER overwrite the dev kubeconfig +# that infra/devcontainer/dev.sh exports as $KUBECONFIG — an operator's `kubectl get +# nodes` silently answering from prod would be a genuinely dangerous mix-up. +# group_vars/k3s_prod/vars.yml overrides this to prod.kubeconfig. (.cache/ is gitignored; +# a kubeconfig is a credential.) +cluster_kubeconfig_name: dev.kubeconfig + # The kube-apiserver endpoint the whole stack agrees on: the primary's REAL node IP. # NOTE: ansible_host is now the pfSense NAT SSH target (192.168.2.102:225x), NOT the node # IP — so we read the real 192.168.1.x from k3s_node_ip instead. Used for k3s --tls-san / # secondary --server AND for Cilium k8sServiceHost so the Ansible bootstrap and the Flux # HelmRelease stay in PARITY (docs/02 contract). -k3s_api_host: "{{ hostvars[groups['k3s_primary'][0]]['k3s_node_ip'] }}" +k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['k3s_node_ip'] }}" k3s_api_port: 6443 # Control-plane HA VIP floated by kube-vip (roles/kube_vip). A stable failover diff --git a/infra/ansible/inventory/group_vars/k3s_prod/README.md b/infra/ansible/inventory/group_vars/k3s_prod/README.md index c7a432c..5bffc4c 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/README.md +++ b/infra/ansible/inventory/group_vars/k3s_prod/README.md @@ -54,9 +54,15 @@ behind them instead of NOPASSWD. - This is the Ansible-bootstrap-layer secret (host credential), distinct from the platform's in-cluster **SOPS + age** secrets (ADR-0007) — different layer, different tool, same convention as `hypervisor`/`proxmox_hosts`. +- **Targeting prod is solved** (2026-08-31): plays take `-e k3s_hosts=k3s_prod` (and, for `k3s.yml`, + `-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`), while the + roles resolve "my environment's primary/servers" from `k3s_primary_group` / `k3s_servers_group` + in `vars.yml` here — no role fork, no per-run flag for that half. See + `infra/ansible/README.md` → "Targeting prod". - **Do not run any prod playbook yet.** Phase 7 is prep-only: the vault above just makes the plays - *able* to authenticate once the §7 repo blockers close and the disk-layout (§2) and - control-plane-VIP (§4) decisions in `infra/PROD-PROVISIONING.md` are made — and, separately, once - the group-name gap flagged in `inventory/hosts.ini`'s prod section is resolved (the existing - playbooks target `k3s_servers` / `k3s_primary` / `k3s_secondary` literally, not `k3s_prod_*`). - See that file for the full picture. + *able* to authenticate once the §7 repo blockers close and the disk-layout (§2) decision in + `infra/PROD-PROVISIONING.md` is made. The ONE exception the operator has sanctioned is the + packages-only baseline that unblocks the §2 drive-health read — + `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages` — which + cannot touch the BLOCKED device placeholders because every disk task is `disks`-tagged and so is + not selected. It still needs `vault.yml` (above) to exist first. diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index 3c901ec..6534690 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -29,13 +29,34 @@ ansible_become: true # The password itself is ansible_become_password — ansible-vault-encrypted, in # vault.yml (this directory), never here in plaintext. See README.md to create it. +# --- environment group NAMES (R-ENV-02) ------------------------------------------- +# group_vars/all.yml defines these as the DEV group names (k3s_primary / k3s_servers); +# here they name PROD's groups. Roles index `groups[k3s_primary_group]` / +# `groups[k3s_servers_group]` instead of a literal, so roles/k3s_server and roles/cilium +# resolve "my environment's primary" and "my environment's servers" correctly for prod +# hosts with NO role fork and no per-run flag — the reuse R-ENV-02 promises. +# +# NOTE these are GROUP NAMES, not the play-level target pattern. Selecting which hosts a +# play RUNS against is a separate extra-vars knob (`-e k3s_hosts=k3s_prod`, and for +# k3s.yml `-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`), +# because Ansible templates `hosts:` before any host — and therefore any group_vars — +# exists. See infra/ansible/README.md "Targeting prod". +k3s_primary_group: k3s_prod_primary +k3s_servers_group: k3s_prod + +# Keep the fetched prod kubeconfig in its OWN cache file. group_vars/all.yml defaults this +# to dev.kubeconfig, which is the file infra/devcontainer/dev.sh exports as $KUBECONFIG — +# a prod play must never clobber it (a `kubectl` that silently answers from prod is worse +# than no kubeconfig at all). .cache/ is gitignored; a kubeconfig is a credential. +cluster_kubeconfig_name: prod.kubeconfig + # --- k3s API host — safe to set now, the primary's real IP is already known ------- # Dev's k3s_api_host (group_vars/all.yml) reads a separate k3s_node_ip hostvar because # dev's ansible_host is the pfSense NAT SSH target, not the node's real address. Prod # has NO NAT hop — ansible_host in hosts.ini IS already the node's real, routable LAN # address (see the KEY DIFFERENCE note there) — so point straight at it instead of # needing a k3s_node_ip split. -k3s_api_host: "{{ hostvars[groups['k3s_prod_primary'][0]]['ansible_host'] }}" +k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}" # cluster_controller_tls_server_name (group_vars/all.yml) derives from kube_vip_address # by template reference, so it needs no separate override here — it picks up the real # value set below automatically for prod hosts too. diff --git a/infra/ansible/inventory/hosts.ini b/infra/ansible/inventory/hosts.ini index f4e2786..62b1c23 100644 --- a/infra/ansible/inventory/hosts.ini +++ b/infra/ansible/inventory/hosts.ini @@ -66,19 +66,21 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh # `ansible_port` is the node's own real sshd (22). See group_vars/k3s_prod/vars.yml for # the k3s_api_host override that reflects this directly-reachable topology. # -# GAP flagged for the PM, NOT fixed in this pass (inventory/group_vars only): the -# k3s_server role hardcodes the LITERAL group names `k3s_primary` / `k3s_secondary` / -# `k3s_servers` in roles/k3s_server/defaults/main.yml (tls-san SAN list) and -# roles/k3s_server/tasks/main.yml (node-token delegate_to), and every node-host -# playbook (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, verify.yml, -# flux-bootstrap.yml) declares `hosts: k3s_servers` / `k3s_primary` / `k3s_secondary` -# literally. Naming prod's groups k3s_prod_primary/k3s_prod_secondary — as this -# checklist and PROD-PROVISIONING.md §6 both call for — means those existing -# playbooks will NOT select prod hosts as they stand today. That's harmless right now -# (nothing can accidentally run against prod), but it means actually reusing the dev -# playbooks against prod later (the R-ENV-02 promise) needs either parametrizing the -# group name they target or some other mechanism — not yet designed, tracked here so -# it isn't lost before §6 is attempted for real. +# GROUP-NAME GAP: CLOSED 2026-08-31. The roles and playbooks used to hardcode the +# LITERAL dev groups (`hosts: k3s_servers` / `k3s_primary` / `k3s_secondary`, and +# `groups['k3s_primary']` inside roles/k3s_server + roles/cilium), so nothing could +# select the k3s_prod_* groups below. Both halves are now variables, deliberately kept +# separate because Ansible resolves them at different times: +# * WHICH HOSTS A PLAY RUNS ON — `k3s_hosts` (whole-cluster plays), `k3s_primary_hosts` +# / `k3s_secondary_hosts` (k3s.yml's split), each defaulting to the dev group. +# EXTRA VARS ONLY: a play's `hosts:` is templated before any host exists, so +# group_vars cannot be read there. Prod: `-e k3s_hosts=k3s_prod` etc. +# * WHICH GROUP A ROLE MEANS BY "MY PRIMARY"/"MY SERVERS" — `k3s_primary_group` / +# `k3s_servers_group`, set in group_vars/all.yml (dev) and group_vars/k3s_prod/vars.yml +# (prod), so roles resolve their own environment with NO per-run flag — the same +# mechanism k3s_api_host already used. +# Running with no extra vars is byte-for-byte the dev behaviour it always had. See +# infra/ansible/README.md "Targeting prod" and PROD-PROVISIONING.md §6. # ============================================================================= [k3s_prod_primary] diff --git a/infra/ansible/playbooks/cilium.yml b/infra/ansible/playbooks/cilium.yml index b49a8e1..4c52c7e 100644 --- a/infra/ansible/playbooks/cilium.yml +++ b/infra/ansible/playbooks/cilium.yml @@ -1,8 +1,13 @@ --- # Install Cilium as the CNI (ADR-0002). The cluster is NotReady until this runs. # Executed from the controller against a fetched kubeconfig (role handles delegation). +# +# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is +# ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod +# The role's "fetch the kubeconfig from the primary" delegation follows the +# k3s_primary_group group_var, so it picks the prod primary automatically. - name: Install Cilium CNI - hosts: k3s_servers + hosts: "{{ k3s_hosts | default('k3s_servers') }}" become: true gather_facts: false roles: diff --git a/infra/ansible/playbooks/flux-bootstrap.yml b/infra/ansible/playbooks/flux-bootstrap.yml index ba14c88..0c6a6da 100644 --- a/infra/ansible/playbooks/flux-bootstrap.yml +++ b/infra/ansible/playbooks/flux-bootstrap.yml @@ -1,8 +1,15 @@ --- # FINAL step: hand off to Flux + SOPS (T-1.4). Calls into bootstrap/ (authored by # the Security + GitOps Engineers) — this play does not author that subtree. +# +# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is +# ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod +# NOTE the Flux PATH is a separate concern: roles/flux_bootstrap shells out to +# bootstrap/flux/bootstrap.sh, which bootstraps --path=clusters/dev. Prod needs +# --path=clusters/prod, which is a bootstrap/ change, NOT a target-group change +# (infra/PROD-PROVISIONING.md §7 tracks it). - name: Flux + SOPS bootstrap hand-off - hosts: k3s_servers + hosts: "{{ k3s_hosts | default('k3s_servers') }}" become: false gather_facts: false roles: diff --git a/infra/ansible/playbooks/k3s.yml b/infra/ansible/playbooks/k3s.yml index 4363a60..8e45345 100644 --- a/infra/ansible/playbooks/k3s.yml +++ b/infra/ansible/playbooks/k3s.yml @@ -2,9 +2,20 @@ # Install the HA k3s control plane (embedded etcd, 3 servers) — ADR-0001. # Primary founds the cluster (--cluster-init); secondaries join it in a second # play so the primary's node-token fact is available to them. +# +# TARGET (R-ENV-02): this is the one playbook that needs TWO patterns, because the +# primary/secondary split is the whole point of the file. Both default to dev: +# ansible-playbook playbooks/k3s.yml # dev, unchanged +# ansible-playbook playbooks/k3s.yml \ +# -e k3s_primary_hosts=k3s_prod_primary \ +# -e k3s_secondary_hosts=k3s_prod_secondary # prod +# The role's OWN "who is the primary" lookups (join token delegate_to, tls-san list) +# do NOT use these — they follow the k3s_primary_group / k3s_servers_group group_vars, +# which prod sets once in group_vars/k3s_prod/vars.yml. These two vars only choose which +# hosts the plays RUN on, which Ansible resolves before any group_vars exist. - name: k3s primary (cluster-init) - hosts: k3s_primary + hosts: "{{ k3s_primary_hosts | default('k3s_primary') }}" become: true gather_facts: true roles: @@ -16,7 +27,7 @@ # an updated tls-san set) triggers the Restart k3s handler: restarting both # secondaries at once would drop etcd below quorum. One-at-a-time keeps 2/3 up. - name: k3s secondaries (join) - hosts: k3s_secondary + hosts: "{{ k3s_secondary_hosts | default('k3s_secondary') }}" become: true serial: 1 gather_facts: true diff --git a/infra/ansible/playbooks/kube-vip.yml b/infra/ansible/playbooks/kube-vip.yml index da47279..7dce62b 100644 --- a/infra/ansible/playbooks/kube-vip.yml +++ b/infra/ansible/playbooks/kube-vip.yml @@ -1,9 +1,14 @@ --- -# Deploy kube-vip as the control-plane HA VIP (192.168.1.49) for the kube-API. +# Deploy kube-vip as the control-plane HA VIP (192.168.1.49 on dev) for the kube-API. # Control-plane ONLY (svc_enable=false) — Cilium owns Service LoadBalancing. # Writes the manifest into each server's k3s auto-deploy dir; k3s applies it. +# +# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is +# ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod +# (the VIP itself comes from kube_vip_address, already scoped per environment in +# group_vars — dev 192.168.1.49, prod 192.168.2.9). - name: Deploy kube-vip control-plane VIP - hosts: k3s_servers + hosts: "{{ k3s_hosts | default('k3s_servers') }}" become: true gather_facts: true roles: diff --git a/infra/ansible/playbooks/node-baseline.yml b/infra/ansible/playbooks/node-baseline.yml index 0b45e17..fac0a72 100644 --- a/infra/ansible/playbooks/node-baseline.yml +++ b/infra/ansible/playbooks/node-baseline.yml @@ -2,8 +2,21 @@ # Node baseline — first-boot floor for k3s/Cilium/Longhorn, moved from cloud-init into # Ansible so Proxmox provisioning stays API-only (no SSH/SFTP to the hypervisor). Runs # FIRST in site.yml, before k3s. Idempotent; safe to re-run. +# +# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is +# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod +# See README.md "Targeting prod". The pattern is the one node-health.yml established. +# +# TAGS — this role is the only way `smartmontools` (and the rest of the package floor) +# reaches a node, but its data-disk tasks need block devices that prod has NOT decided +# yet (infra/PROD-PROVISIONING.md §2). Run JUST the package floor with: +# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages +# `--tags packages` runs the apt task ONLY; every disk task carries the `disks` tag and +# is therefore skipped, so the BLOCKED placeholder devices can never be touched. A full +# (untagged) run against a node whose device vars are still placeholders fails fast on an +# explicit assert instead of handing a bogus path to mkfs. See roles/node_baseline. - name: Baseline the k3s nodes (packages, kernel modules, Longhorn disk, swap-off) - hosts: k3s_servers + hosts: "{{ k3s_hosts | default('k3s_servers') }}" become: true gather_facts: true roles: diff --git a/infra/ansible/playbooks/node-health.yml b/infra/ansible/playbooks/node-health.yml index ca84853..0e7dff8 100644 --- a/infra/ansible/playbooks/node-health.yml +++ b/infra/ansible/playbooks/node-health.yml @@ -6,13 +6,15 @@ # ./dev.sh run bash -c 'cd /work/infra/ansible && \ # ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod' # -# TARGET GROUP IS A VARIABLE, unlike the other playbooks here. Every existing node -# playbook (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, verify.yml) hardcodes -# `hosts: k3s_servers` / `k3s_primary` / `k3s_secondary`, which is exactly why they cannot -# select the prod groups today (the gap flagged in inventory/hosts.ini's prod section, and -# the R-ENV-02 reuse promise it threatens). A health check is the one play you want to be -# able to point anywhere, so it takes the pattern as a var and defaults to dev. Run it -# against prod with `-e node_health_hosts=k3s_prod`, or a single box with `--limit`. +# TARGET GROUP IS A VARIABLE. This play established that pattern; the rest of the node +# playbooks (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, verify.yml, +# flux-bootstrap.yml) now follow it and take `k3s_hosts` / `k3s_primary_hosts` / +# `k3s_secondary_hosts` instead of the literal dev groups they used to hardcode — see +# infra/ansible/README.md "Targeting prod". This one keeps its OWN var name because it is +# a standalone diagnostic you point anywhere, not a step in the bring-up chain: it must +# stay independently targetable (`-e node_health_hosts=`) even while a bring-up run +# is scoped elsewhere. Run it against prod with `-e node_health_hosts=k3s_prod`, or a +# single box with `--limit`. # # BECOME: left at the play level exactly as the inventory sets it. The role itself pins # `ansible_become: false` on every unprivileged task and probes escalation once before diff --git a/infra/ansible/playbooks/site.yml b/infra/ansible/playbooks/site.yml index d471783..8c65f88 100644 --- a/infra/ansible/playbooks/site.yml +++ b/infra/ansible/playbooks/site.yml @@ -2,6 +2,18 @@ # Full dev/test cluster bring-up, in order (docs/02 sequencing; run AFTER # `tofu apply` has created and cloud-init has baselined the VMs): # k3s HA install -> Cilium CNI -> verify readiness -> Flux + SOPS hand-off. +# +# TARGET (R-ENV-02): every imported play takes its host pattern from a var that +# defaults to the dev group, so `ansible-playbook playbooks/site.yml` with no extra +# vars is byte-for-byte the dev run it always was. Extra vars are global and reach +# imported playbooks, so the whole chain retargets with three flags: +# ansible-playbook playbooks/site.yml \ +# -e k3s_hosts=k3s_prod \ +# -e k3s_primary_hosts=k3s_prod_primary \ +# -e k3s_secondary_hosts=k3s_prod_secondary +# DO NOT run that yet — prod is blocked on the disk-layout decision +# (infra/PROD-PROVISIONING.md §2) and the §7 repo blockers, and the operator has ruled +# the prod cluster stays uninitialized until all three nodes are present (§8). - name: Import - node baseline (packages, kernel modules, Longhorn disk, swap-off) ansible.builtin.import_playbook: node-baseline.yml diff --git a/infra/ansible/playbooks/verify.yml b/infra/ansible/playbooks/verify.yml index b5cbc68..08b2631 100644 --- a/infra/ansible/playbooks/verify.yml +++ b/infra/ansible/playbooks/verify.yml @@ -1,7 +1,10 @@ --- # Verify Phase-1 exit gate #1: 3 nodes Ready, Cilium/Hubble healthy, no kube-proxy. +# +# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is +# ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod - name: Verify cluster readiness - hosts: k3s_servers + hosts: "{{ k3s_hosts | default('k3s_servers') }}" become: false gather_facts: false roles: diff --git a/infra/ansible/playbooks/vpn.egress.gateway.yml b/infra/ansible/playbooks/vpn.egress.gateway.yml index 92d7a37..92d4a79 100644 --- a/infra/ansible/playbooks/vpn.egress.gateway.yml +++ b/infra/ansible/playbooks/vpn.egress.gateway.yml @@ -3,15 +3,23 @@ # (ADR-0015). Host infra — the split-tunnel OpenVPN service + policy routing + scoped # nftables kill-switch. The CiliumEgressGatewayPolicy that steers vpn-class pods to this # node's tun0 is authored separately as GitOps (platform/), against the same contract: -# * inventory group `vpn_egress_gateway` (this play's target) +# * inventory group `vpn_egress_gateway` (this play's default target) # * k8s node label homelab.keithjackson.dev/egress-gateway=true (set by the role) # * host tunnel iface tun0 (split-tunnel; never the node default route) # # Secrets come from Ansible vault (inventory/group_vars/vpn_egress_gateway/vault.yml): # vault_vpn_egress_gateway_pia_username, vault_vpn_egress_gateway_pia_password, vault_vpn_egress_gateway_pia_ovpn # Run: ansible-playbook playbooks/vpn.egress.gateway.yml +# +# TARGET (R-ENV-02): the group is a var, defaulting to dev's `vpn_egress_gateway`. Prod +# has NOT decided whether it runs an egress gateway at all, nor on which node +# (infra/PROD-PROVISIONING.md §6, last box) — so there is deliberately no prod egress +# group in the inventory yet. When there is, point this at it with +# -e vpn_egress_gateway_hosts= +# and it needs its OWN vaulted PIA credentials group_vars; this play's vault is scoped to +# the dev group. - name: Configure the node-level VPN egress gateway - hosts: vpn_egress_gateway + hosts: "{{ vpn_egress_gateway_hosts | default('vpn_egress_gateway') }}" become: true gather_facts: true # package module + service facts need os facts roles: diff --git a/infra/ansible/roles/cilium/defaults/main.yml b/infra/ansible/roles/cilium/defaults/main.yml index 02a7d1d..c22b05a 100644 --- a/infra/ansible/roles/cilium/defaults/main.yml +++ b/infra/ansible/roles/cilium/defaults/main.yml @@ -19,8 +19,10 @@ gateway_api_crd_urls: # TLSRoute (experimental) is required by Cilium's Gateway API support. - "https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/{{ gateway_api_version }}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml" -# Local kubeconfig fetched from the primary. *.kubeconfig is gitignored. -cilium_kubeconfig: "{{ playbook_dir }}/../.cache/dev.kubeconfig" +# Local kubeconfig fetched from the primary. *.kubeconfig is gitignored. The FILENAME +# comes from cluster_kubeconfig_name (group_vars) so a prod run cannot overwrite the dev +# kubeconfig the dev container exports as $KUBECONFIG. R-ENV-02. +cilium_kubeconfig: "{{ playbook_dir }}/../.cache/{{ cluster_kubeconfig_name | default('dev.kubeconfig') }}" cilium_values_path: "{{ playbook_dir }}/../.cache/cilium-values.yaml" # dev: single operator replica to save RAM (PHASE1-SCOPE RAM ceiling). NOT a diff --git a/infra/ansible/roles/cilium/tasks/main.yml b/infra/ansible/roles/cilium/tasks/main.yml index a84a8f9..e84b761 100644 --- a/infra/ansible/roles/cilium/tasks/main.yml +++ b/infra/ansible/roles/cilium/tasks/main.yml @@ -4,11 +4,14 @@ # The cluster is NotReady until this completes — expected (PHASE1-SCOPE risks). # run_once: this whole role acts on the cluster once, not per host. +# k3s_primary_group (group_vars) names THIS environment's primary group — `k3s_primary` +# on dev, `k3s_prod_primary` on prod — so the role fetches the right cluster's kubeconfig +# without being forked per environment (R-ENV-02). - name: Fetch the k3s kubeconfig from the primary ansible.builtin.slurp: src: /etc/rancher/k3s/k3s.yaml register: k3s_kubeconfig_raw - delegate_to: "{{ groups['k3s_primary'][0] }}" + delegate_to: "{{ groups[k3s_primary_group][0] }}" run_once: true - name: Ensure the local cache dir exists diff --git a/infra/ansible/roles/cluster_verify/defaults/main.yml b/infra/ansible/roles/cluster_verify/defaults/main.yml index 6c3f707..50344b9 100644 --- a/infra/ansible/roles/cluster_verify/defaults/main.yml +++ b/infra/ansible/roles/cluster_verify/defaults/main.yml @@ -1,4 +1,6 @@ --- # cluster_verify role defaults. -cluster_verify_kubeconfig: "{{ playbook_dir }}/../.cache/dev.kubeconfig" +# Filename comes from cluster_kubeconfig_name (group_vars) so dev and prod never share +# one cache file — dev.kubeconfig for dev, prod.kubeconfig for prod. R-ENV-02. +cluster_verify_kubeconfig: "{{ playbook_dir }}/../.cache/{{ cluster_kubeconfig_name | default('dev.kubeconfig') }}" cluster_verify_node_ready_timeout: "300s" diff --git a/infra/ansible/roles/flux_bootstrap/defaults/main.yml b/infra/ansible/roles/flux_bootstrap/defaults/main.yml index afe902a..c44e3dc 100644 --- a/infra/ansible/roles/flux_bootstrap/defaults/main.yml +++ b/infra/ansible/roles/flux_bootstrap/defaults/main.yml @@ -7,7 +7,8 @@ # args below are ASSUMPTIONS the PM must reconcile with those deliverables — see # roles/flux_bootstrap/tasks/main.yml and infra/ansible/README.md "Interface". -flux_bootstrap_kubeconfig: "{{ playbook_dir }}/../.cache/dev.kubeconfig" +# Filename from cluster_kubeconfig_name (group_vars): dev.kubeconfig / prod.kubeconfig. +flux_bootstrap_kubeconfig: "{{ playbook_dir }}/../.cache/{{ cluster_kubeconfig_name | default('dev.kubeconfig') }}" # Repo root, relative to playbooks/ (infra/ansible/playbooks -> ../../..). flux_bootstrap_repo_root: "{{ playbook_dir }}/../../.." diff --git a/infra/ansible/roles/k3s_server/defaults/main.yml b/infra/ansible/roles/k3s_server/defaults/main.yml index 462280c..b2a78ce 100644 --- a/infra/ansible/roles/k3s_server/defaults/main.yml +++ b/infra/ansible/roles/k3s_server/defaults/main.yml @@ -26,7 +26,11 @@ k3s_server_mode: "secondary" # the VIP), and 127.0.0.1 (local kubeconfig / health probes). k3s regenerates the # serving cert on restart when this set changes. Deduplicated (the primary IP is # also k3s_api_host). +# +# k3s_servers_group — NOT a literal group name — is what makes this role environment- +# agnostic (R-ENV-02): group_vars/all.yml sets it to `k3s_servers` (dev) and +# group_vars/k3s_prod/vars.yml to `k3s_prod`, so each environment SANs its own servers. k3s_tls_sans: >- {{ ([k3s_api_host, kube_vip_address, '127.0.0.1'] - + (groups['k3s_servers'] | map('extract', hostvars, 'k3s_node_ip') | list)) + + (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list)) | unique }} diff --git a/infra/ansible/roles/k3s_server/tasks/main.yml b/infra/ansible/roles/k3s_server/tasks/main.yml index c280031..eee9d6e 100644 --- a/infra/ansible/roles/k3s_server/tasks/main.yml +++ b/infra/ansible/roles/k3s_server/tasks/main.yml @@ -121,19 +121,23 @@ # contained — delegate_to still reaches the primary even when it isn't a play # target, so the join no longer depends on a fact an earlier play had to set. # T-1.5 finding F2 (see DECISIONS-LOG.md). +# +# "The primary" is resolved through k3s_primary_group (group_vars: `k3s_primary` for +# dev, `k3s_prod_primary` for prod), never a literal group name — so a prod secondary +# delegates to the PROD primary using this same role, unchanged (R-ENV-02). - name: Fetch the k3s node-token directly from the primary (secondary) ansible.builtin.slurp: src: /var/lib/rancher/k3s/server/token - delegate_to: "{{ groups['k3s_primary'][0] }}" + delegate_to: "{{ groups[k3s_primary_group][0] }}" register: k3s_primary_token_raw when: - k3s_server_mode == 'secondary' - - hostvars[groups['k3s_primary'][0]].k3s_node_token is not defined + - hostvars[groups[k3s_primary_group][0]].k3s_node_token is not defined - name: Resolve the join token (secondary) ansible.builtin.set_fact: k3s_join_token: >- - {{ hostvars[groups['k3s_primary'][0]].k3s_node_token + {{ hostvars[groups[k3s_primary_group][0]].k3s_node_token | default(k3s_primary_token_raw.content | b64decode | trim, true) }} when: k3s_server_mode == 'secondary' diff --git a/infra/ansible/roles/node_baseline/tasks/main.yml b/infra/ansible/roles/node_baseline/tasks/main.yml index c8cf0b1..15fd113 100644 --- a/infra/ansible/roles/node_baseline/tasks/main.yml +++ b/infra/ansible/roles/node_baseline/tasks/main.yml @@ -1,10 +1,23 @@ --- +# TAGS — every task here carries one of `packages`, `system` or `disks`. An untagged run +# is unchanged (all three run, in this order). The split exists because the PACKAGE floor +# and the DATA-DISK work have different readiness: packages are safe on any node from +# first contact, while the disk tasks need block devices that a new environment may not +# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked +# on a SMART read that needs `smartmontools` — which ships in node_baseline_packages). +# `--tags packages` breaks that circle: it installs the package floor and NOTHING else. +# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages +# Skipping is the guard: with `--tags packages` no `disks`-tagged task is selected, so a +# BLOCKED-pending-disk-decision device path is never even templated, let alone handed to +# mkfs. A FULL run against such a node stops at the assert below instead. + - name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs) ansible.builtin.apt: name: "{{ node_baseline_packages }}" state: present update_cache: true cache_valid_time: 3600 + tags: [packages] - name: Persist kernel modules for k3s/Cilium/Longhorn ansible.builtin.copy: @@ -13,12 +26,14 @@ group: root mode: "0644" content: "{{ node_baseline_kernel_modules | join('\n') }}\n" + tags: [system] - name: Load kernel modules now community.general.modprobe: name: "{{ item }}" state: present loop: "{{ node_baseline_kernel_modules }}" + tags: [system] - name: Install sysctl tuning for k3s/Cilium ansible.builtin.copy: @@ -34,12 +49,14 @@ fs.inotify.max_user_instances = 8192 fs.inotify.max_user_watches = 524288 notify: Reload sysctl + tags: [system] - name: Enable and start iscsid (Longhorn prerequisite) ansible.builtin.systemd: name: iscsid enabled: true state: started + tags: [system] # qemu-guest-agent binds to the virtio-serial channel /dev/virtio-ports/org.qemu.guest_agent.0, # which Proxmox attaches only when the VM's `agent` option is enabled. Our tofu keeps agent @@ -51,18 +68,44 @@ ansible.builtin.stat: path: /dev/virtio-ports/org.qemu.guest_agent.0 register: node_baseline_qga_channel + tags: [system] - name: Enable qemu-guest-agent (start only when its virtio channel is present) ansible.builtin.systemd: name: qemu-guest-agent enabled: true state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}" + tags: [system] + +# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided +# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries +# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see +# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc, +# which do not exist on bare metal either — a confusing failure that looks like a wiring +# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb, +# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work. +- name: Assert this environment's data-disk devices have actually been decided + ansible.builtin.assert: + that: + - "'BLOCKED' not in node_baseline_longhorn_device" + - "'BLOCKED' not in node_baseline_etcd_device" + fail_msg: >- + node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED + placeholders for this host's environment + (longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}). + The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 — + and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so + the SMART read that decision waits on can be taken) re-run this play with + `--tags packages`, which skips every disk task including this assert. + quiet: true + tags: [disks] - name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present) community.general.filesystem: fstype: ext4 dev: "{{ node_baseline_longhorn_device }}" opts: "-L {{ node_baseline_longhorn_fs_label }}" + tags: [disks] - name: Mount the Longhorn data disk ansible.posix.mount: @@ -71,6 +114,7 @@ fstype: ext4 opts: "defaults,nofail,x-systemd.device-timeout=5s" state: mounted + tags: [disks] # Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and # mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with @@ -84,6 +128,7 @@ fstype: ext4 dev: "{{ node_baseline_etcd_device }}" opts: "-L {{ node_baseline_etcd_fs_label }}" + tags: [disks] - name: Mount the dedicated etcd SSD disk at the k3s etcd data dir ansible.posix.mount: @@ -96,13 +141,16 @@ dump: "0" passno: "2" state: mounted + tags: [disks] - name: Disable swap now ansible.builtin.command: swapoff -a changed_when: false + tags: [system] - name: Neutralize swap entries in fstab (kubelet requires swap off) ansible.builtin.replace: path: /etc/fstab regexp: '^([^#].*\sswap\s.*)$' replace: '# \1 # disabled for k3s/kubelet' + tags: [system]