roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.
TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
* which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
(k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
* which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
Follows the k3s_api_host convention already in this repo. No per-run flag.
DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.
BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).
FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.
KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.
Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.
Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PROD-PROVISIONING.md §4 now carries the full operator-decided address table (nodes
.10/.11/.12, control-plane VIP .9, ingress Gateway VIP .20, Cilium LB pool .20-.30) with
the two invariants that make it safe: the control-plane VIP sits outside the LB pool so
kube-vip and LB-IPAM cannot contend, and GATEWAY_VIP is the FIRST address of the pool
because the shared Gateway pins nothing and LB-IPAM hands it the first free address.
§5 no longer tells the operator to hand-install smartmontools — package installation is
the node_baseline role's job (operator ruling). §0 gains the node 2/3 addresses.
Task: T-7.1. Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two operator corrections:
1. PROD DOES NOT GET NOPASSWD SUDO. My initial checklist recommended mirroring dev's
passwordless sudo onto prod — rejected, correctly. Dev's NOPASSWD is a convenience
that must not propagate to production. Prod instead keeps a real sudo boundary and
drives Ansible `become` with an ansible-vault-encrypted `ansible_become_password` in
inventory/group_vars/k3s_prod/vault.yml, mirroring the EXISTING hypervisor and
proxmox_hosts groups. The decrypting passphrase stays outside the repo at
~/.config/home-k3s/ansible-vault-pass (ansible.cfg :: vault_password_file), so plays
still run unattended with no plaintext secret in git.
2. THE TASK BOARD IS NOT THE STATUS SURFACE — THE REPOSITORY IS. Operator ruling closes
the escalation raised in the previous commit. Requirement/spec status is to be DERIVED
from repo state (what is authored, reconciled, and covered by a green requirement-tagged
test), not from a hand-maintained board. The spec board is left as a historical seed;
no agent edits it and no SPEC_PIN churn is spent on it. ci/spec-immutability.sh keeps
its full strictness — no path exception was added, since weakening an R-TEST-06 gate
for a bookkeeping file would be a defect, not a fix. A repo-derived spec status report
becomes the artifact of record alongside the coverage report and this log.
Serves: R-ENV-03 (prod provisioning posture), R-SEC-01 (no plaintext secrets), R-TEST-06.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The operator added the first production node (k3s-prod-1, 192.168.2.10 — HP EliteDesk
800 G5 Mini, i7-9700T/31GiB, Debian 13, VT-x + /dev/kvm) and opened roadmap Phase 7 for
PREP WORK ONLY: no k3s install until the R-ENV-02 blockers close and the gate is green.
Records the Phase-7 entry-gate position honestly: the operator has declared hardware
ready, but "Phases 1-6 proven in dev/test" is NOT met (37/51 conformance; R-ENV-01/02/03
themselves uncovered). Proceeding is an explicit, accepted risk — not an oversight.
Adds infra/PROD-PROVISIONING.md (T-7.1, design-only): firmware/BIOS, disk layout, OS,
network, access, Ansible reuse, repo blockers, build sequence, Phase-7 exit criteria.
Key findings recorded in DECISIONS-LOG:
- R-ENV-02 is violated in the base — dev LAN addresses hardcoded into base manifests
(cilium k8sServiceHost, the Cilium LB pool, crafty/UniFi LB IPs, 3 S3 backup
endpoints). Prod is on 192.168.2.x, so prod would build WRONG today.
- clusters/prod never sets GATEWAY_VIP -> would inherit the base default 10.73.0.240,
an address on the RETIRED libvirt network.
- No NOPASSWD sudo on the prod node -> all Ansible fails at its first become.
- node_baseline device defaults are virtio (/dev/vdb,vdc) and absent on bare metal.
- ESCALATION: agents/task-board.md is structurally unmaintainable — AGENTS.md designates
it the one agent-editable surface, but ci/spec-immutability.sh fails the gate on ANY
spec-repo dirt with no path exception. Explains the all-todo board after six phases.
NOT worked around: exempting it would weaken an R-TEST-06 gate. Operator decision.
Operator decisions: Phase 7 open (prep only); wait for all 3 nodes before cluster-init;
check drive health before choosing the single-SSD disk layout.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03. Task: T-7.1.
Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85andClaude Opus 5