roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.
TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
* which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
(k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
* which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
Follows the k3s_api_host convention already in this repo. No per-run flag.
DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.
BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).
FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.
KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.
Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.
Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).
Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.
- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
2026-08-26 LIVE STATE line
Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gateway API CRDs bumped v1.2.1->v1.3.0 (required by Cilium 1.18; applied server-side).
Land on current k8s 1.32 before the 1.33 hop.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Land the two pacing components on current k8s 1.31 before the 1.32 k3s hop.
Cilium 1.17 supports k8s 1.29-1.32; Longhorn 1.8 supports 1.25+; Gateway API CRDs
stay v1.2.1 (Cilium 1.17 targets v1.2.x). Also commits the full upgrade plan doc.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 5