diff --git a/infra/ansible/README.md b/infra/ansible/README.md index 59a74b4..4324ca0 100644 --- a/infra/ansible/README.md +++ b/infra/ansible/README.md @@ -88,6 +88,25 @@ expose different entrypoints, override the two vars in `roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged for PM reconciliation. +## Prod inventory — scaffolding only, Phase 7 PREP (nothing above is run against it) + +`inventory/hosts.ini` also carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` / +`[k3s_prod:children]` section for the three planned bare-metal nodes (`k3s-prod-1` at +`192.168.2.10` exists and is reachable today; `k3s-prod-2`/`-3` at `.11`/`.12` are addressed but +not yet racked, so their inventory lines stay commented out), with its own `group_vars/k3s_prod/` +(vaulted `ansible_become_password` + non-secret overrides). Note `192.168.2.20` is prod's +**ingress** Gateway VIP (a Flux/`cluster-vars` concern, not an Ansible one — it does not appear in +`group_vars/k3s_prod/`); the kube-vip **control-plane** VIP (`kube_vip_address`) is a *separate*, +still-undecided address, left as a loud BLOCKED placeholder alongside the `node_baseline` device +overrides. See `group_vars/k3s_prod/README.md` and `infra/PROD-PROVISIONING.md` (T-7.1) §4/§2 for +the open decisions that still block a real run. + +Everything else in this README describes the **dev** runbook. It does not (yet) apply to prod +as-is: the run order above and the roles it calls (`k3s_server`, `cluster_verify`, ...) target the +literal `k3s_primary` / `k3s_secondary` / `k3s_servers` groups, not `k3s_prod_*` — see the GAP note +at the top of `inventory/hosts.ini`'s prod section for what's still missing before that reuse (the +R-ENV-02 promise) actually works end to end. + ## Static validation (PM, in dev container — NOT run here) ``` yamllint -c ../../ci/yamllint.yaml infra/ansible diff --git a/infra/ansible/inventory/group_vars/k3s_prod/README.md b/infra/ansible/inventory/group_vars/k3s_prod/README.md new file mode 100644 index 0000000..c7a432c --- /dev/null +++ b/infra/ansible/inventory/group_vars/k3s_prod/README.md @@ -0,0 +1,62 @@ +# group_vars for the `k3s_prod` group (prod k3s nodes — Phase 7 PREP ONLY) + +Every node-host play that touches a k3s node (`playbooks/node-baseline.yml`, `k3s.yml`, +`cilium.yml`, `kube-vip.yml`, ...) already runs with `become: true` at the play level. On **dev** +that succeeds without a password (cloud-init's `login_user` gets NOPASSWD sudo). **Prod +deliberately does NOT get that** — operator decision, 2026-08-31 (`DECISIONS-LOG.md`, "PHASE 7 +OPENED", finding 3; `infra/PROD-PROVISIONING.md` §5). Supply the sudo password as an +**ansible-vault-encrypted** var here — it is encrypted, so it is safe to commit; the passphrase +that decrypts it stays out of the repo, exactly like the existing `hypervisor` and `proxmox_hosts` +groups. + +## Two files (one committed-encrypted, one never committed) + +1. **The vault passphrase** — a file **outside the repo**, referenced by `ansible.cfg` + (`vault_password_file = ~/.config/home-k3s/ansible-vault-pass`). Skip this if you already made + one for the `hypervisor` or `proxmox_hosts` vaults — it's the SAME passphrase file, shared by + every vaulted group in this repo: + ```bash + mkdir -p ~/.config/home-k3s + install -m600 /dev/stdin ~/.config/home-k3s/ansible-vault-pass <<<'YOUR_VAULT_PASSPHRASE' + ``` + +2. **The encrypted become password** — `vault.yml` in **this directory** (committed, encrypted). + `vault.yml.example` alongside it shows the exact shape (see it for the reasoning); copy its + content, fill in the real password, then encrypt it in place — do not commit the `.example` + content unencrypted as `vault.yml`: + ```bash + cd infra/ansible + ansible-vault create inventory/group_vars/k3s_prod/vault.yml + # file content (single line): + # ansible_become_password: "THE k3s USER'S SUDO PASSWORD ON k3s-prod-1" + ``` + (`ansible-vault create` uses the passphrase file above automatically — no prompt.) + +Once `vault.yml` exists, prod plays run unattended — same as dev, just with a real sudo boundary +behind them instead of NOPASSWD. + +## Notes +- `vault.yml` is skipped by the yamllint gate (`ci/yamllint.yaml` ignores `**/vault.yml`) because a + vault blob is not parseable YAML. `vault.yml.example` is plain YAML and IS linted — it's a + copy-paste scaffold, not a secret (same convention as + `platform/backup/minio-credentials.sops.yaml.example`). +- The k3s **dev** nodes keep passwordless sudo unchanged — this vault is scoped to the `k3s_prod` + group only. +- `ansible_become: true` for this group is set explicitly in `vars.yml` (this directory) — belt + and braces alongside the playbooks' own `become: true`, so prod's posture is self-documenting + and doesn't rely on inheriting anything from `group_vars/all.yml` by accident. +- Non-secret prod vars live in `vars.yml`, not here — this README is scoped to the vault hand-off. + `k3s_api_host` is set to a real value. `kube_vip_address`, `cluster_controller_api_server`, + `node_baseline_longhorn_device` and `node_baseline_etcd_device` are all still-BLOCKED + placeholders pending operator decisions (control-plane VIP and disk layout, respectively) — see + `vars.yml`'s comments for what IS already decided (node addresses, the separate *ingress* VIP, + which is a Flux/`cluster-vars` concern and does not live here) versus what is not. +- This is the Ansible-bootstrap-layer secret (host credential), distinct from the platform's + in-cluster **SOPS + age** secrets (ADR-0007) — different layer, different tool, same convention + as `hypervisor`/`proxmox_hosts`. +- **Do not run any prod playbook yet.** Phase 7 is prep-only: the vault above just makes the plays + *able* to authenticate once the §7 repo blockers close and the disk-layout (§2) and + control-plane-VIP (§4) decisions in `infra/PROD-PROVISIONING.md` are made — and, separately, once + the group-name gap flagged in `inventory/hosts.ini`'s prod section is resolved (the existing + playbooks target `k3s_servers` / `k3s_primary` / `k3s_secondary` literally, not `k3s_prod_*`). + See that file for the full picture. diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml new file mode 100644 index 0000000..3c901ec --- /dev/null +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -0,0 +1,85 @@ +--- +# group_vars/k3s_prod/vars.yml — NON-secret prod-scoped overrides for the [k3s_prod] +# umbrella group (inventory/hosts.ini: [k3s_prod_primary]/[k3s_prod_secondary] under +# [k3s_prod:children]). Ansible auto-loads group_vars//*.yml for every host +# that is a (transitive) member of that group. See infra/PROD-PROVISIONING.md (T-7.1) +# §5/§6 for the checklist this file implements, and DECISIONS-LOG.md "PHASE 7 OPENED" +# (2026-08-31) for the operator decisions behind it. +# +# Pairs with vault.yml in this same directory (ansible-vault-encrypted, operator- +# created — see README.md here) for the one secret prod needs: ansible_become_password. +# +# PHASE 7 IS PREP ONLY. Four vars below (kube_vip_address, cluster_controller_api_server, +# node_baseline_longhorn_device, node_baseline_etcd_device) are deliberately BROKEN +# PLACEHOLDERS, not real values — see their comments. Do not "fix" them with a guess; +# follow the referenced section of PROD-PROVISIONING.md, record the decision there, +# and update this file in the same change. + +# --- become posture (§5) ---------------------------------------------------------- +# Prod deliberately does NOT get dev's passwordless (NOPASSWD) sudo — operator +# decision, 2026-08-31 (DECISIONS-LOG.md "PHASE 7 OPENED", finding 3). group_vars/ +# all.yml carries no explicit `ansible_become` of its own (dev's passwordless sudo +# means become succeeds without a password prompt, not that become is switched off), +# and every node-host playbook already sets `become: true` at the play level +# (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml) — for dev that silently +# succeeds because sudo needs no password there. Setting ansible_become explicitly +# here is belt-and-braces: this group's posture is self-documenting and does not rely +# on inheriting dev's assumption by accident. +ansible_become: true +# The password itself is ansible_become_password — ansible-vault-encrypted, in +# vault.yml (this directory), never here in plaintext. See README.md to create it. + +# --- k3s API host — safe to set now, the primary's real IP is already known ------- +# Dev's k3s_api_host (group_vars/all.yml) reads a separate k3s_node_ip hostvar because +# dev's ansible_host is the pfSense NAT SSH target, not the node's real address. Prod +# has NO NAT hop — ansible_host in hosts.ini IS already the node's real, routable LAN +# address (see the KEY DIFFERENCE note there) — so point straight at it instead of +# needing a k3s_node_ip split. +k3s_api_host: "{{ hostvars[groups['k3s_prod_primary'][0]]['ansible_host'] }}" +# cluster_controller_tls_server_name (group_vars/all.yml) derives from kube_vip_address +# by template reference, so it needs no separate override here — it picks up the real +# value set below automatically for prod hosts too. + +# --- control-plane VIP + controller access — DECIDED (§4) ------------------------- +# Prod addressing, all decided by the operator 2026-08-31: +# nodes k3s-prod-1/2/3 192.168.2.10 / .11 / .12 (hosts.ini) +# control-plane VIP 192.168.2.9 (THIS value — kube-vip, Ansible-side) +# ingress Gateway VIP 192.168.2.20 (Flux/cluster-vars — NOT here) +# ingress L2 pool 192.168.2.20 - .30 (Flux/cluster-vars — NOT here) +# +# The control-plane VIP sits BELOW the node addresses and deliberately OUTSIDE the +# Cilium LB pool (.20-.30), so kube-vip and Cilium LB-IPAM can never contend for the +# same address. That separation is the point: dev learned the same shape the other way +# round (control-plane .49, ingress pool .53-.60). +# +# This is the address kube-vip floats for the API and that k3s tls-san's onto its +# serving cert; secondaries use it as their --server join target. Dev's equivalent is +# kube_vip_address: 192.168.1.49 in group_vars/all.yml. +# +# HISTORY: an earlier pass here wrongly set this to 192.168.2.20 — that is the INGRESS +# VIP, not the control-plane VIP. Corrected. Keep the two distinct. +kube_vip_address: "192.168.2.9" +# Derived from kube_vip_address above (not repeated), so fixing the one line above is +# the only change needed once a real control-plane VIP is chosen. Prod has no NAT hop +# (unlike dev's cluster_controller_api_server, the dev-container's NAT path to its VIP +# — see group_vars/all.yml), so once real this collapses to a direct "https://:6443". +cluster_controller_api_server: "https://{{ kube_vip_address }}:6443" + +# --- node_baseline data-disk devices — BLOCKED, open decision (§2) ---------------- +# roles/node_baseline defaults (node_baseline_longhorn_device: /dev/vdb, +# node_baseline_etcd_device: /dev/vdc) assume tofu-provisioned virtio disks that DO +# NOT EXIST on bare metal. Node 1 is a SINGLE SATA disk (/dev/sda) with no free block +# device at all today; PROD-PROVISIONING.md §2 lists three candidate fixes (add a +# second drive / repartition / reuse the /dev/sda5 "/srv" partition) that all wait on +# a SMART health read of a ~15-year-old consumer SSD (operator decision, 2026-08-31). +# +# The overrides below are deliberately present-but-broken: leaving them UNSET would +# let the role silently fall back to /dev/vdb//dev/vdc, which ALSO don't exist on this +# hardware — a confusing failure that looks like a wiring bug. This value fails just +# as hard, but says exactly why, and blocks a `find`/glob from accidentally matching +# a real device. DO NOT replace with a guess — replace only once §2 is decided, and +# record the chosen device paths in PROD-PROVISIONING.md §2 in the SAME change that +# updates these two lines. All three nodes must end up with an identical layout +# (Longhorn replica placement assumes symmetry) before any of this is used for real. +node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2" +node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2" diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vault.yml.example b/infra/ansible/inventory/group_vars/k3s_prod/vault.yml.example new file mode 100644 index 0000000..d719807 --- /dev/null +++ b/infra/ansible/inventory/group_vars/k3s_prod/vault.yml.example @@ -0,0 +1,25 @@ +# ============================================================================= +# TEMPLATE — NOT ENCRYPTED, NOT LOADED (a plain `.example` file; Ansible only reads +# `vault.yml` in this directory, never `vault.yml.example`). Reference scaffold + +# OPERATOR HAND-OFF, same convention as platform/backup/minio-credentials.sops.yaml.example. +# +# WHAT THIS IS: the sudo (become) password for the `k3s` user on prod k3s nodes. +# Prod deliberately does NOT use passwordless (NOPASSWD) sudo the way dev does — +# operator decision, 2026-08-31 (DECISIONS-LOG.md "PHASE 7 OPENED", finding 3; +# infra/PROD-PROVISIONING.md §5). Every play that becomes root on a k3s_prod host +# (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, ...) needs this to escalate. +# +# HAND-OFF STEPS (operator): +# (a) Create the vault passphrase file ONCE, outside the repo, if you don't already +# have one from the `hypervisor`/`proxmox_hosts` vaults (it's the same file, +# shared by every vaulted group in this repo): +# mkdir -p ~/.config/home-k3s +# install -m600 /dev/stdin ~/.config/home-k3s/ansible-vault-pass <<<'YOUR_VAULT_PASSPHRASE' +# (b) Encrypt the REAL content into inventory/group_vars/k3s_prod/vault.yml — never +# commit this .example file's content unencrypted as vault.yml itself: +# cd infra/ansible +# ansible-vault create inventory/group_vars/k3s_prod/vault.yml +# # paste the single line below with the REAL password, save, exit +# (c) Verify: ansible-vault view inventory/group_vars/k3s_prod/vault.yml +# ============================================================================= +ansible_become_password: "REPLACE-ME" diff --git a/infra/ansible/inventory/hosts.ini b/infra/ansible/inventory/hosts.ini index 198accc..f4e2786 100644 --- a/infra/ansible/inventory/hosts.ini +++ b/infra/ansible/inventory/hosts.ini @@ -45,3 +45,55 @@ k3s-dev-3 # `hypervisor` group above (that is `localhost` for host-prereqs.yml, a different machine). [proxmox_hosts] proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh_private_key_file=~/.ssh/home_network + +# ============================================================================= +# PROD (k3s-prod-*) — Phase 7 PREP ONLY. See infra/PROD-PROVISIONING.md (T-7.1, +# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). Node 1 +# (`k3s-prod-1`, HP EliteDesk 800 G5 Mini) is on the bench and SSH-reachable; nodes 2 +# and 3 have not arrived. This section is inventory/group_vars scaffolding only — +# nothing has been run against node 1, and the operator has decided the CLUSTER stays +# uninitialized (no k3s install, no --cluster-init) until all three nodes are present +# (decision #2 in the DECISIONS-LOG entry above) — no single-server-etcd window, no +# later member-add dance. +# +# KEY DIFFERENCE FROM DEV: prod sits DIRECTLY on the 192.168.2.x LAN — no pfSense NAT +# hop in front of it. Dev's `ansible_host` above is a NAT target (192.168.2.102:225x) +# that port-forwards to the node's real 192.168.1.x address, which is why dev also +# carries a separate `k3s_node_ip` per host. Prod needs no such split: `ansible_host` +# below IS already the node's real, routable address, so `k3s_node_ip` is set to the +# SAME value (kept explicit, not omitted, only so the k3s_server role's hostvar lookup +# — see the GAP note below — has something correct to find once that gap is closed). +# `ansible_port` is the node's own real sshd (22). See group_vars/k3s_prod/vars.yml for +# the k3s_api_host override that reflects this directly-reachable topology. +# +# GAP flagged for the PM, NOT fixed in this pass (inventory/group_vars only): the +# k3s_server role hardcodes the LITERAL group names `k3s_primary` / `k3s_secondary` / +# `k3s_servers` in roles/k3s_server/defaults/main.yml (tls-san SAN list) and +# roles/k3s_server/tasks/main.yml (node-token delegate_to), and every node-host +# playbook (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, verify.yml, +# flux-bootstrap.yml) declares `hosts: k3s_servers` / `k3s_primary` / `k3s_secondary` +# literally. Naming prod's groups k3s_prod_primary/k3s_prod_secondary — as this +# checklist and PROD-PROVISIONING.md §6 both call for — means those existing +# playbooks will NOT select prod hosts as they stand today. That's harmless right now +# (nothing can accidentally run against prod), but it means actually reusing the dev +# playbooks against prod later (the R-ENV-02 promise) needs either parametrizing the +# group name they target or some other mechanism — not yet designed, tracked here so +# it isn't lost before §6 is attempted for real. +# ============================================================================= + +[k3s_prod_primary] +k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10 + +[k3s_prod_secondary] +# Addresses for nodes 2/3 ARE decided (operator, 2026-08-31 — infra/PROD-PROVISIONING.md +# §0), but the nodes themselves are NOT yet racked/imaged/reachable. Left commented so +# no play can accidentally target a host that isn't there; uncomment each line only once +# that node is actually on the bench and answering SSH. +# k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11 +# k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12 + +# Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children] +# pattern; named to match the group_vars directory so Ansible auto-loads it). +[k3s_prod:children] +k3s_prod_primary +k3s_prod_secondary diff --git a/infra/ansible/roles/node_baseline/defaults/main.yml b/infra/ansible/roles/node_baseline/defaults/main.yml index 5397b1d..1517d37 100644 --- a/infra/ansible/roles/node_baseline/defaults/main.yml +++ b/infra/ansible/roles/node_baseline/defaults/main.yml @@ -2,11 +2,22 @@ # node_baseline — first-boot floor for k3s/Cilium/Longhorn, moved out of cloud-init # (Proxmox provisioning is API-only). Runs before k3s (site.yml). Idempotent. node_baseline_packages: + # VM-only in practice (binds the virtio-serial channel Proxmox attaches) but harmless to install + # unconditionally: the task below only *starts* it when that channel exists, so on bare metal + # (prod) it installs and stays a dormant, disabled unit. Kept in the shared list rather than + # split per-platform so dev and prod install from the exact same package set (R-ENV-02 reuse). - qemu-guest-agent - open-iscsi - nfs-common - curl - cryptsetup # Longhorn >=1.12 RequiredPackages check needs the cryptsetup userspace tool (dm_crypt module alone isn't enough) + # Drive health (smartctl) is a PERMANENT part of a bare-metal node's baseline, not a one-off + # manual install: prod runs on real disks that can fail/degrade (see infra/PROD-PROVISIONING.md + # §2/§5 — a ~15-year-old consumer SSD on node 1), where dev's virtio disks never needed it. + # Installed here (not hand-run with apt-get) so it's declared, idempotent, and present on every + # node from first baseline — package installation is the role's job, never a manual step + # (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo). + - smartmontools node_baseline_kernel_modules: - overlay - br_netfilter