feat(ansible): prod node inventory + vaulted become; smartmontools into node_baseline

Prod inventory scaffolding for the three bare-metal nodes, and the become posture the
operator ruled for production.

BECOME: prod does NOT get dev's NOPASSWD sudo. Prod keeps a real sudo boundary and
supplies the password to Ansible as an ansible-vault-encrypted `ansible_become_password`
in inventory/group_vars/k3s_prod/vault.yml, mirroring the existing hypervisor and
proxmox_hosts groups. The decrypting passphrase stays outside the repo
(~/.config/home-k3s/ansible-vault-pass), so plays still run unattended with no plaintext
secret in git. Committed here: README.md + vault.yml.example only — the operator creates
the encrypted vault (`ansible-vault create`), as with every other vaulted group.

PACKAGES BELONG IN THE ROLE (operator ruling): smartmontools is now declared in
node_baseline_packages, not hand-installed on a node. Drive health tooling is a
permanent part of a bare-metal node's baseline — dev nodes are VMs and never needed it.
Same click-ops-never principle as the rest of the repo.

ADDRESSING: nodes .10/.11/.12, reached DIRECTLY on the LAN (no pfSense NAT hop, unlike
dev's 192.168.2.102:225x mapping), so ansible_host is the real node IP. kube-vip
control-plane VIP is 192.168.2.9 — deliberately BELOW the nodes and OUTSIDE the Cilium
LB pool (.20-.30) so kube-vip and LB-IPAM can never contend for an address. The ingress
Gateway VIP (.20) and the LB pool are Flux/cluster-vars concerns and deliberately do
not appear in Ansible.

BLOCKED-BY-DESIGN: node_baseline_longhorn_device / node_baseline_etcd_device are present
but set to an obviously invalid /dev/BLOCKED-... path pending the operator's disk-layout
decision (node 1 is a single SATA SSD; the role's /dev/vdb,/dev/vdc defaults are virtio
names that do not exist on bare metal). Leaving them unset would silently fall back to
those equally-nonexistent defaults — a confusing failure that looks like a wiring bug.
This fails just as hard and says exactly why.

KNOWN FOLLOW-UP: roles/k3s_server and every node-host playbook hardcode the literal
group names k3s_primary/k3s_secondary/k3s_servers, so the k3s_prod_* groups are not
selected by them as-is. Harmless now (nothing can accidentally run against prod) but the
"same roles, unchanged" reuse R-ENV-02 promises needs the target group parameterized
before §6 of PROD-PROVISIONING.md can execute. Documented in hosts.ini and both READMEs.

Serves: R-ENV-01, R-ENV-03, R-SEC-01. Task: T-7.1. Gate: MERGE GATE: PASS.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-08-31 22:44:10 -05:00
1 parent 3f980d2a17
commit 666cc0d809
6 files changed
+254

No files matched your search

+19
View File
@@ -88,6 +88,25 @@ expose different entrypoints, override the two vars in
`roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged
for PM reconciliation.
## Prod inventory — scaffolding only, Phase 7 PREP (nothing above is run against it)
`inventory/hosts.ini` also carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
`[k3s_prod:children]` section for the three planned bare-metal nodes (`k3s-prod-1` at
`192.168.2.10` exists and is reachable today; `k3s-prod-2`/`-3` at `.11`/`.12` are addressed but
not yet racked, so their inventory lines stay commented out), with its own `group_vars/k3s_prod/`
(vaulted `ansible_become_password` + non-secret overrides). Note `192.168.2.20` is prod's
**ingress** Gateway VIP (a Flux/`cluster-vars` concern, not an Ansible one — it does not appear in
`group_vars/k3s_prod/`); the kube-vip **control-plane** VIP (`kube_vip_address`) is a *separate*,
still-undecided address, left as a loud BLOCKED placeholder alongside the `node_baseline` device
overrides. See `group_vars/k3s_prod/README.md` and `infra/PROD-PROVISIONING.md` (T-7.1) §4/§2 for
the open decisions that still block a real run.
Everything else in this README describes the **dev** runbook. It does not (yet) apply to prod
as-is: the run order above and the roles it calls (`k3s_server`, `cluster_verify`, ...) target the
literal `k3s_primary` / `k3s_secondary` / `k3s_servers` groups, not `k3s_prod_*` — see the GAP note
at the top of `inventory/hosts.ini`'s prod section for what's still missing before that reuse (the
R-ENV-02 promise) actually works end to end.
## Static validation (PM, in dev container — NOT run here)
```
yamllint -c ../../ci/yamllint.yaml infra/ansible
@@ -0,0 +1,62 @@
# group_vars for the `k3s_prod` group (prod k3s nodes — Phase 7 PREP ONLY)
Every node-host play that touches a k3s node (`playbooks/node-baseline.yml`, `k3s.yml`,
`cilium.yml`, `kube-vip.yml`, ...) already runs with `become: true` at the play level. On **dev**
that succeeds without a password (cloud-init's `login_user` gets NOPASSWD sudo). **Prod
deliberately does NOT get that** — operator decision, 2026-08-31 (`DECISIONS-LOG.md`, "PHASE 7
OPENED", finding 3; `infra/PROD-PROVISIONING.md` §5). Supply the sudo password as an
**ansible-vault-encrypted** var here — it is encrypted, so it is safe to commit; the passphrase
that decrypts it stays out of the repo, exactly like the existing `hypervisor` and `proxmox_hosts`
groups.
## Two files (one committed-encrypted, one never committed)
1. **The vault passphrase** — a file **outside the repo**, referenced by `ansible.cfg`
(`vault_password_file = ~/.config/home-k3s/ansible-vault-pass`). Skip this if you already made
one for the `hypervisor` or `proxmox_hosts` vaults — it's the SAME passphrase file, shared by
every vaulted group in this repo:
```bash
mkdir -p ~/.config/home-k3s
install -m600 /dev/stdin ~/.config/home-k3s/ansible-vault-pass <<<'YOUR_VAULT_PASSPHRASE'
```
2. **The encrypted become password** — `vault.yml` in **this directory** (committed, encrypted).
`vault.yml.example` alongside it shows the exact shape (see it for the reasoning); copy its
content, fill in the real password, then encrypt it in place — do not commit the `.example`
content unencrypted as `vault.yml`:
```bash
cd infra/ansible
ansible-vault create inventory/group_vars/k3s_prod/vault.yml
# file content (single line):
# ansible_become_password: "THE k3s USER'S SUDO PASSWORD ON k3s-prod-1"
```
(`ansible-vault create` uses the passphrase file above automatically — no prompt.)
Once `vault.yml` exists, prod plays run unattended — same as dev, just with a real sudo boundary
behind them instead of NOPASSWD.
## Notes
- `vault.yml` is skipped by the yamllint gate (`ci/yamllint.yaml` ignores `**/vault.yml`) because a
vault blob is not parseable YAML. `vault.yml.example` is plain YAML and IS linted — it's a
copy-paste scaffold, not a secret (same convention as
`platform/backup/minio-credentials.sops.yaml.example`).
- The k3s **dev** nodes keep passwordless sudo unchanged — this vault is scoped to the `k3s_prod`
group only.
- `ansible_become: true` for this group is set explicitly in `vars.yml` (this directory) — belt
and braces alongside the playbooks' own `become: true`, so prod's posture is self-documenting
and doesn't rely on inheriting anything from `group_vars/all.yml` by accident.
- Non-secret prod vars live in `vars.yml`, not here — this README is scoped to the vault hand-off.
`k3s_api_host` is set to a real value. `kube_vip_address`, `cluster_controller_api_server`,
`node_baseline_longhorn_device` and `node_baseline_etcd_device` are all still-BLOCKED
placeholders pending operator decisions (control-plane VIP and disk layout, respectively) — see
`vars.yml`'s comments for what IS already decided (node addresses, the separate *ingress* VIP,
which is a Flux/`cluster-vars` concern and does not live here) versus what is not.
- This is the Ansible-bootstrap-layer secret (host credential), distinct from the platform's
in-cluster **SOPS + age** secrets (ADR-0007) — different layer, different tool, same convention
as `hypervisor`/`proxmox_hosts`.
- **Do not run any prod playbook yet.** Phase 7 is prep-only: the vault above just makes the plays
*able* to authenticate once the §7 repo blockers close and the disk-layout (§2) and
control-plane-VIP (§4) decisions in `infra/PROD-PROVISIONING.md` are made — and, separately, once
the group-name gap flagged in `inventory/hosts.ini`'s prod section is resolved (the existing
playbooks target `k3s_servers` / `k3s_primary` / `k3s_secondary` literally, not `k3s_prod_*`).
See that file for the full picture.
@@ -0,0 +1,85 @@
---
# group_vars/k3s_prod/vars.yml — NON-secret prod-scoped overrides for the [k3s_prod]
# umbrella group (inventory/hosts.ini: [k3s_prod_primary]/[k3s_prod_secondary] under
# [k3s_prod:children]). Ansible auto-loads group_vars/<groupname>/*.yml for every host
# that is a (transitive) member of that group. See infra/PROD-PROVISIONING.md (T-7.1)
# §5/§6 for the checklist this file implements, and DECISIONS-LOG.md "PHASE 7 OPENED"
# (2026-08-31) for the operator decisions behind it.
#
# Pairs with vault.yml in this same directory (ansible-vault-encrypted, operator-
# created — see README.md here) for the one secret prod needs: ansible_become_password.
#
# PHASE 7 IS PREP ONLY. Four vars below (kube_vip_address, cluster_controller_api_server,
# node_baseline_longhorn_device, node_baseline_etcd_device) are deliberately BROKEN
# PLACEHOLDERS, not real values — see their comments. Do not "fix" them with a guess;
# follow the referenced section of PROD-PROVISIONING.md, record the decision there,
# and update this file in the same change.
# --- become posture (§5) ----------------------------------------------------------
# Prod deliberately does NOT get dev's passwordless (NOPASSWD) sudo — operator
# decision, 2026-08-31 (DECISIONS-LOG.md "PHASE 7 OPENED", finding 3). group_vars/
# all.yml carries no explicit `ansible_become` of its own (dev's passwordless sudo
# means become succeeds without a password prompt, not that become is switched off),
# and every node-host playbook already sets `become: true` at the play level
# (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml) — for dev that silently
# succeeds because sudo needs no password there. Setting ansible_become explicitly
# here is belt-and-braces: this group's posture is self-documenting and does not rely
# on inheriting dev's assumption by accident.
ansible_become: true
# The password itself is ansible_become_password — ansible-vault-encrypted, in
# vault.yml (this directory), never here in plaintext. See README.md to create it.
# --- k3s API host — safe to set now, the primary's real IP is already known -------
# Dev's k3s_api_host (group_vars/all.yml) reads a separate k3s_node_ip hostvar because
# dev's ansible_host is the pfSense NAT SSH target, not the node's real address. Prod
# has NO NAT hop — ansible_host in hosts.ini IS already the node's real, routable LAN
# address (see the KEY DIFFERENCE note there) — so point straight at it instead of
# needing a k3s_node_ip split.
k3s_api_host: "{{ hostvars[groups['k3s_prod_primary'][0]]['ansible_host'] }}"
# cluster_controller_tls_server_name (group_vars/all.yml) derives from kube_vip_address
# by template reference, so it needs no separate override here — it picks up the real
# value set below automatically for prod hosts too.
# --- control-plane VIP + controller access — DECIDED (§4) -------------------------
# Prod addressing, all decided by the operator 2026-08-31:
# nodes k3s-prod-1/2/3 192.168.2.10 / .11 / .12 (hosts.ini)
# control-plane VIP 192.168.2.9 (THIS value — kube-vip, Ansible-side)
# ingress Gateway VIP 192.168.2.20 (Flux/cluster-vars — NOT here)
# ingress L2 pool 192.168.2.20 - .30 (Flux/cluster-vars — NOT here)
#
# The control-plane VIP sits BELOW the node addresses and deliberately OUTSIDE the
# Cilium LB pool (.20-.30), so kube-vip and Cilium LB-IPAM can never contend for the
# same address. That separation is the point: dev learned the same shape the other way
# round (control-plane .49, ingress pool .53-.60).
#
# This is the address kube-vip floats for the API and that k3s tls-san's onto its
# serving cert; secondaries use it as their --server join target. Dev's equivalent is
# kube_vip_address: 192.168.1.49 in group_vars/all.yml.
#
# HISTORY: an earlier pass here wrongly set this to 192.168.2.20 — that is the INGRESS
# VIP, not the control-plane VIP. Corrected. Keep the two distinct.
kube_vip_address: "192.168.2.9"
# Derived from kube_vip_address above (not repeated), so fixing the one line above is
# the only change needed once a real control-plane VIP is chosen. Prod has no NAT hop
# (unlike dev's cluster_controller_api_server, the dev-container's NAT path to its VIP
# — see group_vars/all.yml), so once real this collapses to a direct "https://<VIP>:6443".
cluster_controller_api_server: "https://{{ kube_vip_address }}:6443"
# --- node_baseline data-disk devices — BLOCKED, open decision (§2) ----------------
# roles/node_baseline defaults (node_baseline_longhorn_device: /dev/vdb,
# node_baseline_etcd_device: /dev/vdc) assume tofu-provisioned virtio disks that DO
# NOT EXIST on bare metal. Node 1 is a SINGLE SATA disk (/dev/sda) with no free block
# device at all today; PROD-PROVISIONING.md §2 lists three candidate fixes (add a
# second drive / repartition / reuse the /dev/sda5 "/srv" partition) that all wait on
# a SMART health read of a ~15-year-old consumer SSD (operator decision, 2026-08-31).
#
# The overrides below are deliberately present-but-broken: leaving them UNSET would
# let the role silently fall back to /dev/vdb//dev/vdc, which ALSO don't exist on this
# hardware — a confusing failure that looks like a wiring bug. This value fails just
# as hard, but says exactly why, and blocks a `find`/glob from accidentally matching
# a real device. DO NOT replace with a guess — replace only once §2 is decided, and
# record the chosen device paths in PROD-PROVISIONING.md §2 in the SAME change that
# updates these two lines. All three nodes must end up with an identical layout
# (Longhorn replica placement assumes symmetry) before any of this is used for real.
node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
@@ -0,0 +1,25 @@
# =============================================================================
# TEMPLATE — NOT ENCRYPTED, NOT LOADED (a plain `.example` file; Ansible only reads
# `vault.yml` in this directory, never `vault.yml.example`). Reference scaffold +
# OPERATOR HAND-OFF, same convention as platform/backup/minio-credentials.sops.yaml.example.
#
# WHAT THIS IS: the sudo (become) password for the `k3s` user on prod k3s nodes.
# Prod deliberately does NOT use passwordless (NOPASSWD) sudo the way dev does —
# operator decision, 2026-08-31 (DECISIONS-LOG.md "PHASE 7 OPENED", finding 3;
# infra/PROD-PROVISIONING.md §5). Every play that becomes root on a k3s_prod host
# (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, ...) needs this to escalate.
#
# HAND-OFF STEPS (operator):
# (a) Create the vault passphrase file ONCE, outside the repo, if you don't already
# have one from the `hypervisor`/`proxmox_hosts` vaults (it's the same file,
# shared by every vaulted group in this repo):
# mkdir -p ~/.config/home-k3s
# install -m600 /dev/stdin ~/.config/home-k3s/ansible-vault-pass <<<'YOUR_VAULT_PASSPHRASE'
# (b) Encrypt the REAL content into inventory/group_vars/k3s_prod/vault.yml — never
# commit this .example file's content unencrypted as vault.yml itself:
# cd infra/ansible
# ansible-vault create inventory/group_vars/k3s_prod/vault.yml
# # paste the single line below with the REAL password, save, exit
# (c) Verify: ansible-vault view inventory/group_vars/k3s_prod/vault.yml
# =============================================================================
ansible_become_password: "REPLACE-ME"
+52
View File
@@ -45,3 +45,55 @@ k3s-dev-3
# `hypervisor` group above (that is `localhost` for host-prereqs.yml, a different machine).
[proxmox_hosts]
proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh_private_key_file=~/.ssh/home_network
# =============================================================================
# PROD (k3s-prod-*) — Phase 7 PREP ONLY. See infra/PROD-PROVISIONING.md (T-7.1,
# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). Node 1
# (`k3s-prod-1`, HP EliteDesk 800 G5 Mini) is on the bench and SSH-reachable; nodes 2
# and 3 have not arrived. This section is inventory/group_vars scaffolding only —
# nothing has been run against node 1, and the operator has decided the CLUSTER stays
# uninitialized (no k3s install, no --cluster-init) until all three nodes are present
# (decision #2 in the DECISIONS-LOG entry above) — no single-server-etcd window, no
# later member-add dance.
#
# KEY DIFFERENCE FROM DEV: prod sits DIRECTLY on the 192.168.2.x LAN — no pfSense NAT
# hop in front of it. Dev's `ansible_host` above is a NAT target (192.168.2.102:225x)
# that port-forwards to the node's real 192.168.1.x address, which is why dev also
# carries a separate `k3s_node_ip` per host. Prod needs no such split: `ansible_host`
# below IS already the node's real, routable address, so `k3s_node_ip` is set to the
# SAME value (kept explicit, not omitted, only so the k3s_server role's hostvar lookup
# — see the GAP note below — has something correct to find once that gap is closed).
# `ansible_port` is the node's own real sshd (22). See group_vars/k3s_prod/vars.yml for
# the k3s_api_host override that reflects this directly-reachable topology.
#
# GAP flagged for the PM, NOT fixed in this pass (inventory/group_vars only): the
# k3s_server role hardcodes the LITERAL group names `k3s_primary` / `k3s_secondary` /
# `k3s_servers` in roles/k3s_server/defaults/main.yml (tls-san SAN list) and
# roles/k3s_server/tasks/main.yml (node-token delegate_to), and every node-host
# playbook (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, verify.yml,
# flux-bootstrap.yml) declares `hosts: k3s_servers` / `k3s_primary` / `k3s_secondary`
# literally. Naming prod's groups k3s_prod_primary/k3s_prod_secondary — as this
# checklist and PROD-PROVISIONING.md §6 both call for — means those existing
# playbooks will NOT select prod hosts as they stand today. That's harmless right now
# (nothing can accidentally run against prod), but it means actually reusing the dev
# playbooks against prod later (the R-ENV-02 promise) needs either parametrizing the
# group name they target or some other mechanism — not yet designed, tracked here so
# it isn't lost before §6 is attempted for real.
# =============================================================================
[k3s_prod_primary]
k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
[k3s_prod_secondary]
# Addresses for nodes 2/3 ARE decided (operator, 2026-08-31 — infra/PROD-PROVISIONING.md
# §0), but the nodes themselves are NOT yet racked/imaged/reachable. Left commented so
# no play can accidentally target a host that isn't there; uncomment each line only once
# that node is actually on the bench and answering SSH.
# k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
# k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
# Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
# pattern; named to match the group_vars directory so Ansible auto-loads it).
[k3s_prod:children]
k3s_prod_primary
k3s_prod_secondary
@@ -2,11 +2,22 @@
# node_baseline — first-boot floor for k3s/Cilium/Longhorn, moved out of cloud-init
# (Proxmox provisioning is API-only). Runs before k3s (site.yml). Idempotent.
node_baseline_packages:
# VM-only in practice (binds the virtio-serial channel Proxmox attaches) but harmless to install
# unconditionally: the task below only *starts* it when that channel exists, so on bare metal
# (prod) it installs and stays a dormant, disabled unit. Kept in the shared list rather than
# split per-platform so dev and prod install from the exact same package set (R-ENV-02 reuse).
- qemu-guest-agent
- open-iscsi
- nfs-common
- curl
- cryptsetup # Longhorn >=1.12 RequiredPackages check needs the cryptsetup userspace tool (dm_crypt module alone isn't enough)
# Drive health (smartctl) is a PERMANENT part of a bare-metal node's baseline, not a one-off
# manual install: prod runs on real disks that can fail/degrade (see infra/PROD-PROVISIONING.md
# §2/§5 — a ~15-year-old consumer SSD on node 1), where dev's virtio disks never needed it.
# Installed here (not hand-run with apt-get) so it's declared, idempotent, and present on every
# node from first baseline — package installation is the role's job, never a manual step
# (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo).
- smartmontools
node_baseline_kernel_modules:
- overlay
- br_netfilter