feat(ansible): parameterise target groups so prod reuses the roles unchanged (R-ENV-02)

roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.

TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
  * which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
    (k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
    only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
  * which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
    k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
    Follows the k3s_api_host convention already in this repo. No per-run flag.

DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.

BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
  ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).

FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.

KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.

Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.

Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

infra/ansible/roles/cilium/defaults/main.yml

@@ -19,8 +19,10 @@ gateway_api_crd_urls:
   # TLSRoute (experimental) is required by Cilium's Gateway API support.
   - "https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/{{ gateway_api_version }}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml"
 
-# Local kubeconfig fetched from the primary. *.kubeconfig is gitignored.
-cilium_kubeconfig: "{{ playbook_dir }}/../.cache/dev.kubeconfig"
+# Local kubeconfig fetched from the primary. *.kubeconfig is gitignored. The FILENAME
+# comes from cluster_kubeconfig_name (group_vars) so a prod run cannot overwrite the dev
+# kubeconfig the dev container exports as $KUBECONFIG. R-ENV-02.
+cilium_kubeconfig: "{{ playbook_dir }}/../.cache/{{ cluster_kubeconfig_name | default('dev.kubeconfig') }}"
 cilium_values_path: "{{ playbook_dir }}/../.cache/cilium-values.yaml"
 
 # dev: single operator replica to save RAM (PHASE1-SCOPE RAM ceiling). NOT a

infra/ansible/roles/cilium/tasks/main.yml

@@ -4,11 +4,14 @@
 # The cluster is NotReady until this completes — expected (PHASE1-SCOPE risks).
 # run_once: this whole role acts on the cluster once, not per host.
 
+# k3s_primary_group (group_vars) names THIS environment's primary group — `k3s_primary`
+# on dev, `k3s_prod_primary` on prod — so the role fetches the right cluster's kubeconfig
+# without being forked per environment (R-ENV-02).
 - name: Fetch the k3s kubeconfig from the primary
   ansible.builtin.slurp:
     src: /etc/rancher/k3s/k3s.yaml
   register: k3s_kubeconfig_raw
-  delegate_to: "{{ groups['k3s_primary'][0] }}"
+  delegate_to: "{{ groups[k3s_primary_group][0] }}"
   run_once: true
 
 - name: Ensure the local cache dir exists

infra/ansible/roles/cluster_verify/defaults/main.yml

@@ -1,4 +1,6 @@
 ---
 # cluster_verify role defaults.
-cluster_verify_kubeconfig: "{{ playbook_dir }}/../.cache/dev.kubeconfig"
+# Filename comes from cluster_kubeconfig_name (group_vars) so dev and prod never share
+# one cache file — dev.kubeconfig for dev, prod.kubeconfig for prod. R-ENV-02.
+cluster_verify_kubeconfig: "{{ playbook_dir }}/../.cache/{{ cluster_kubeconfig_name | default('dev.kubeconfig') }}"
 cluster_verify_node_ready_timeout: "300s"

infra/ansible/roles/flux_bootstrap/defaults/main.yml

@@ -7,7 +7,8 @@
 # args below are ASSUMPTIONS the PM must reconcile with those deliverables — see
 # roles/flux_bootstrap/tasks/main.yml and infra/ansible/README.md "Interface".
 
-flux_bootstrap_kubeconfig: "{{ playbook_dir }}/../.cache/dev.kubeconfig"
+# Filename from cluster_kubeconfig_name (group_vars): dev.kubeconfig / prod.kubeconfig.
+flux_bootstrap_kubeconfig: "{{ playbook_dir }}/../.cache/{{ cluster_kubeconfig_name | default('dev.kubeconfig') }}"
 
 # Repo root, relative to playbooks/ (infra/ansible/playbooks -> ../../..).
 flux_bootstrap_repo_root: "{{ playbook_dir }}/../../.."

infra/ansible/roles/k3s_server/defaults/main.yml

@@ -26,7 +26,11 @@ k3s_server_mode: "secondary"
 # the VIP), and 127.0.0.1 (local kubeconfig / health probes). k3s regenerates the
 # serving cert on restart when this set changes. Deduplicated (the primary IP is
 # also k3s_api_host).
+#
+# k3s_servers_group — NOT a literal group name — is what makes this role environment-
+# agnostic (R-ENV-02): group_vars/all.yml sets it to `k3s_servers` (dev) and
+# group_vars/k3s_prod/vars.yml to `k3s_prod`, so each environment SANs its own servers.
 k3s_tls_sans: >-
   {{ ([k3s_api_host, kube_vip_address, '127.0.0.1']
-      + (groups['k3s_servers'] | map('extract', hostvars, 'k3s_node_ip') | list))
+      + (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list))
      | unique }}

infra/ansible/roles/k3s_server/tasks/main.yml

@@ -121,19 +121,23 @@
 # contained — delegate_to still reaches the primary even when it isn't a play
 # target, so the join no longer depends on a fact an earlier play had to set.
 # T-1.5 finding F2 (see DECISIONS-LOG.md).
+#
+# "The primary" is resolved through k3s_primary_group (group_vars: `k3s_primary` for
+# dev, `k3s_prod_primary` for prod), never a literal group name — so a prod secondary
+# delegates to the PROD primary using this same role, unchanged (R-ENV-02).
 - name: Fetch the k3s node-token directly from the primary (secondary)
   ansible.builtin.slurp:
     src: /var/lib/rancher/k3s/server/token
-  delegate_to: "{{ groups['k3s_primary'][0] }}"
+  delegate_to: "{{ groups[k3s_primary_group][0] }}"
   register: k3s_primary_token_raw
   when:
     - k3s_server_mode == 'secondary'
-    - hostvars[groups['k3s_primary'][0]].k3s_node_token is not defined
+    - hostvars[groups[k3s_primary_group][0]].k3s_node_token is not defined
 
 - name: Resolve the join token (secondary)
   ansible.builtin.set_fact:
     k3s_join_token: >-
-      {{ hostvars[groups['k3s_primary'][0]].k3s_node_token
+      {{ hostvars[groups[k3s_primary_group][0]].k3s_node_token
          | default(k3s_primary_token_raw.content | b64decode | trim, true) }}
   when: k3s_server_mode == 'secondary'
 

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -1,10 +1,23 @@
 ---
+# TAGS — every task here carries one of `packages`, `system` or `disks`. An untagged run
+# is unchanged (all three run, in this order). The split exists because the PACKAGE floor
+# and the DATA-DISK work have different readiness: packages are safe on any node from
+# first contact, while the disk tasks need block devices that a new environment may not
+# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
+# on a SMART read that needs `smartmontools` — which ships in node_baseline_packages).
+# `--tags packages` breaks that circle: it installs the package floor and NOTHING else.
+#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
+# Skipping is the guard: with `--tags packages` no `disks`-tagged task is selected, so a
+# BLOCKED-pending-disk-decision device path is never even templated, let alone handed to
+# mkfs. A FULL run against such a node stops at the assert below instead.
+
 - name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
   ansible.builtin.apt:
     name: "{{ node_baseline_packages }}"
     state: present
     update_cache: true
     cache_valid_time: 3600
+  tags: [packages]
 
 - name: Persist kernel modules for k3s/Cilium/Longhorn
   ansible.builtin.copy:
@@ -13,12 +26,14 @@
     group: root
     mode: "0644"
     content: "{{ node_baseline_kernel_modules | join('\n') }}\n"
+  tags: [system]
 
 - name: Load kernel modules now
   community.general.modprobe:
     name: "{{ item }}"
     state: present
   loop: "{{ node_baseline_kernel_modules }}"
+  tags: [system]
 
 - name: Install sysctl tuning for k3s/Cilium
   ansible.builtin.copy:
@@ -34,12 +49,14 @@
       fs.inotify.max_user_instances       = 8192
       fs.inotify.max_user_watches         = 524288
   notify: Reload sysctl
+  tags: [system]
 
 - name: Enable and start iscsid (Longhorn prerequisite)
   ansible.builtin.systemd:
     name: iscsid
     enabled: true
     state: started
+  tags: [system]
 
 # qemu-guest-agent binds to the virtio-serial channel /dev/virtio-ports/org.qemu.guest_agent.0,
 # which Proxmox attaches only when the VM's `agent` option is enabled. Our tofu keeps agent
@@ -51,18 +68,44 @@
   ansible.builtin.stat:
     path: /dev/virtio-ports/org.qemu.guest_agent.0
   register: node_baseline_qga_channel
+  tags: [system]
 
 - name: Enable qemu-guest-agent (start only when its virtio channel is present)
   ansible.builtin.systemd:
     name: qemu-guest-agent
     enabled: true
     state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
+  tags: [system]
+
+# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
+# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
+# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
+# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
+# which do not exist on bare metal either — a confusing failure that looks like a wiring
+# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
+# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
+- name: Assert this environment's data-disk devices have actually been decided
+  ansible.builtin.assert:
+    that:
+      - "'BLOCKED' not in node_baseline_longhorn_device"
+      - "'BLOCKED' not in node_baseline_etcd_device"
+    fail_msg: >-
+      node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
+      placeholders for this host's environment
+      (longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
+      The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
+      and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
+      the SMART read that decision waits on can be taken) re-run this play with
+      `--tags packages`, which skips every disk task including this assert.
+    quiet: true
+  tags: [disks]
 
 - name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
   community.general.filesystem:
     fstype: ext4
     dev: "{{ node_baseline_longhorn_device }}"
     opts: "-L {{ node_baseline_longhorn_fs_label }}"
+  tags: [disks]
 
 - name: Mount the Longhorn data disk
   ansible.posix.mount:
@@ -71,6 +114,7 @@
     fstype: ext4
     opts: "defaults,nofail,x-systemd.device-timeout=5s"
     state: mounted
+  tags: [disks]
 
 # Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
 # mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
@@ -84,6 +128,7 @@
     fstype: ext4
     dev: "{{ node_baseline_etcd_device }}"
     opts: "-L {{ node_baseline_etcd_fs_label }}"
+  tags: [disks]
 
 - name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
   ansible.posix.mount:
@@ -96,13 +141,16 @@
     dump: "0"
     passno: "2"
     state: mounted
+  tags: [disks]
 
 - name: Disable swap now
   ansible.builtin.command: swapoff -a
   changed_when: false
+  tags: [system]
 
 - name: Neutralize swap entries in fstab (kubelet requires swap off)
   ansible.builtin.replace:
     path: /etc/fstab
     regexp: '^([^#].*\sswap\s.*)$'
     replace: '# \1  # disabled for k3s/kubelet'
+  tags: [system]