feat(ansible): parameterise target groups so prod reuses the roles unchanged (R-ENV-02)

roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.

TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
  * which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
    (k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
    only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
  * which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
    k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
    Follows the k3s_api_host convention already in this repo. No per-run flag.

DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.

BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
  ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).

FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.

KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.

Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.

Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -1,10 +1,23 @@
 ---
+# TAGS — every task here carries one of `packages`, `system` or `disks`. An untagged run
+# is unchanged (all three run, in this order). The split exists because the PACKAGE floor
+# and the DATA-DISK work have different readiness: packages are safe on any node from
+# first contact, while the disk tasks need block devices that a new environment may not
+# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
+# on a SMART read that needs `smartmontools` — which ships in node_baseline_packages).
+# `--tags packages` breaks that circle: it installs the package floor and NOTHING else.
+#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
+# Skipping is the guard: with `--tags packages` no `disks`-tagged task is selected, so a
+# BLOCKED-pending-disk-decision device path is never even templated, let alone handed to
+# mkfs. A FULL run against such a node stops at the assert below instead.
+
 - name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
   ansible.builtin.apt:
     name: "{{ node_baseline_packages }}"
     state: present
     update_cache: true
     cache_valid_time: 3600
+  tags: [packages]
 
 - name: Persist kernel modules for k3s/Cilium/Longhorn
   ansible.builtin.copy:
@@ -13,12 +26,14 @@
     group: root
     mode: "0644"
     content: "{{ node_baseline_kernel_modules | join('\n') }}\n"
+  tags: [system]
 
 - name: Load kernel modules now
   community.general.modprobe:
     name: "{{ item }}"
     state: present
   loop: "{{ node_baseline_kernel_modules }}"
+  tags: [system]
 
 - name: Install sysctl tuning for k3s/Cilium
   ansible.builtin.copy:
@@ -34,12 +49,14 @@
       fs.inotify.max_user_instances       = 8192
       fs.inotify.max_user_watches         = 524288
   notify: Reload sysctl
+  tags: [system]
 
 - name: Enable and start iscsid (Longhorn prerequisite)
   ansible.builtin.systemd:
     name: iscsid
     enabled: true
     state: started
+  tags: [system]
 
 # qemu-guest-agent binds to the virtio-serial channel /dev/virtio-ports/org.qemu.guest_agent.0,
 # which Proxmox attaches only when the VM's `agent` option is enabled. Our tofu keeps agent
@@ -51,18 +68,44 @@
   ansible.builtin.stat:
     path: /dev/virtio-ports/org.qemu.guest_agent.0
   register: node_baseline_qga_channel
+  tags: [system]
 
 - name: Enable qemu-guest-agent (start only when its virtio channel is present)
   ansible.builtin.systemd:
     name: qemu-guest-agent
     enabled: true
     state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
+  tags: [system]
+
+# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
+# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
+# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
+# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
+# which do not exist on bare metal either — a confusing failure that looks like a wiring
+# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
+# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
+- name: Assert this environment's data-disk devices have actually been decided
+  ansible.builtin.assert:
+    that:
+      - "'BLOCKED' not in node_baseline_longhorn_device"
+      - "'BLOCKED' not in node_baseline_etcd_device"
+    fail_msg: >-
+      node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
+      placeholders for this host's environment
+      (longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
+      The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
+      and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
+      the SMART read that decision waits on can be taken) re-run this play with
+      `--tags packages`, which skips every disk task including this assert.
+    quiet: true
+  tags: [disks]
 
 - name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
   community.general.filesystem:
     fstype: ext4
     dev: "{{ node_baseline_longhorn_device }}"
     opts: "-L {{ node_baseline_longhorn_fs_label }}"
+  tags: [disks]
 
 - name: Mount the Longhorn data disk
   ansible.posix.mount:
@@ -71,6 +114,7 @@
     fstype: ext4
     opts: "defaults,nofail,x-systemd.device-timeout=5s"
     state: mounted
+  tags: [disks]
 
 # Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
 # mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
@@ -84,6 +128,7 @@
     fstype: ext4
     dev: "{{ node_baseline_etcd_device }}"
     opts: "-L {{ node_baseline_etcd_fs_label }}"
+  tags: [disks]
 
 - name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
   ansible.posix.mount:
@@ -96,13 +141,16 @@
     dump: "0"
     passno: "2"
     state: mounted
+  tags: [disks]
 
 - name: Disable swap now
   ansible.builtin.command: swapoff -a
   changed_when: false
+  tags: [system]
 
 - name: Neutralize swap entries in fstab (kubelet requires swap off)
   ansible.builtin.replace:
     path: /etc/fstab
     regexp: '^([^#].*\sswap\s.*)$'
     replace: '# \1  # disabled for k3s/kubelet'
+  tags: [system]