feat(ansible): parameterise target groups so prod reuses the roles unchanged (R-ENV-02)

roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.

TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
  * which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
    (k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
    only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
  * which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
    k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
    Follows the k3s_api_host convention already in this repo. No per-run flag.

DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.

BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
  ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).

FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.

KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.

Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.

Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-08-31 23:53:36 -05:00
1 parent 1157c1b071
commit e623d27e2d
22 files changed
+384 -76

No files matched your search

+74 -15
View File
@@ -84,7 +84,14 @@ Consequences to settle before imaging nodes 2 and 3:
- **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one
physical failure domain.
- [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry.
- [ ] Record the chosen device paths here and set them in `inventory/group_vars/k3s_prod.yml`.
- [ ] Record the chosen device paths here and set them in
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately
`/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full
`node-baseline` run against a node still carrying those placeholders now stops on an explicit
assert naming this section, instead of failing obscurely at `mkfs` time.
- [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline`
is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk
tasks this decision blocks — see §5's drive-health box for the exact command.
## 3. OS install (per node)
@@ -149,29 +156,81 @@ Consequences to settle before imaging nodes 2 and 3:
- [ ] **Drive health baseline** (§2). **Do not hand-install the tooling** — package installation is
the `node_baseline` role's job, never a manual `apt-get` on a node (operator ruling,
2026-08-31; same click-ops-never principle as the rest of the repo). `smartmontools` is part
of `node_baseline_packages`, so it arrives with the baseline play. Reading the drive is then:
of `node_baseline_packages`, so it arrives with the baseline play — but a *full* baseline
would hit the disk tasks §2 has not decided yet. Run the **packages-only** baseline instead
(the `disks`-tagged tasks, including the BLOCKED-device assert, are not selected at all, so
the placeholder device paths are never even templated):
```bash
cd infra/ansible
ansible-playbook playbooks/node-baseline.yml \
-e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
```
(Needs `group_vars/k3s_prod/vault.yml` to exist first — the become password above.) Reading
the drive is then:
```bash
sudo smartctl -a /dev/sda # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors
```
Record the result in §2 and close the layout decision in the same change that replaces the
two `node_baseline_*_device` placeholders.
## 6. Ansible baseline & cluster join
Reuses the dev roles unchanged — that reuse **is** R-ENV-02. Only inventory and group_vars differ.
Reuses the dev roles **unchanged** — that reuse **is** R-ENV-02. Only inventory and group_vars
differ, plus a target-group override on the command line.
- [ ] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached
### How prod is targeted (done, 2026-08-31)
Every node-host play used to hardcode the literal dev groups `k3s_servers` / `k3s_primary` /
`k3s_secondary`, so nothing here could select prod at all. Two knobs now close that, deliberately
kept separate because Ansible resolves them at different times:
| Layer | Knob | Set via |
|---|---|---|
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts` (each defaults to the dev group) | `-e` **extra vars only** — a play's `hosts:` is templated before any host exists, so `group_vars` is not readable there |
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group`, `cluster_kubeconfig_name` | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml`; **no per-run flag** |
So `roles/k3s_server` finds the prod `--cluster-init` founder, builds the prod join target and
tls-san list, and `roles/cilium` fetches the prod kubeconfig, with **no role fork**. Running with
no extra vars is byte-for-byte the dev behaviour it always had (verified by a before/after
`--list-hosts` diff on every playbook). Full detail: `infra/ansible/README.md` → **Targeting prod**.
- [x] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached
**directly** on the LAN — no NAT port-forward hop, unlike dev's `192.168.2.102:225x` mapping.
- [ ] `group_vars/k3s_prod.yml`: `kube_vip_address`, `cluster_controller_api_server`, and the §2
device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`.
- [ ] Run `playbooks/node-baseline.yml` → packages, kernel modules, sysctls, iscsid, data-disk
filesystems + mounts. (`qemu-guest-agent` is a no-op on bare metal — the role already only
*starts* it when the virtio channel exists, so it degrades cleanly.)
- [ ] Run `playbooks/k3s.yml` → HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled.
**Only once all three nodes are present** — see §8.
- [ ] Run `playbooks/kube-vip.yml` → control-plane VIP on the prod LAN.
- [ ] Run `playbooks/cilium.yml` → CNI at bootstrap; Flux's HelmRelease adopts it afterwards. The
bootstrap values and the HelmRelease **must stay in parity** (`kubeProxyReplacement`,
`k8sServiceHost`/`Port`) or the two layers fight over CNI config (`docs/02`).
(Nodes 2/3 stay commented out until they are physically on the bench.)
- [x] `inventory/group_vars/k3s_prod/vars.yml`: `kube_vip_address` (`192.168.2.9`),
`cluster_controller_api_server`, `k3s_api_host`, the group-name overrides above — **and** the
§2 device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`, which
are still BLOCKED placeholders.
- [ ] `inventory/group_vars/k3s_prod/vault.yml` — the vaulted `ansible_become_password` (§5). Until
it exists **no** prod play can escalate, and `--check` against prod cannot work either.
- [ ] Run the **packages-only** baseline first, to unblock the §2 drive-health read:
```bash
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
```
- [ ] Run the **full** `playbooks/node-baseline.yml -e k3s_hosts=k3s_prod` → packages, kernel
modules, sysctls, iscsid, data-disk filesystems + mounts. **Blocked until §2 is decided**: it
asserts on the placeholder device paths and stops. (`qemu-guest-agent` is a no-op on bare
metal — the role already only *starts* it when the virtio channel exists, so it degrades
cleanly.)
- [ ] Run `playbooks/k3s.yml -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`
→ HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. **Only once all three
nodes are present** — see §8.
- [ ] Run `playbooks/kube-vip.yml -e k3s_hosts=k3s_prod` → control-plane VIP (`192.168.2.9`) on the
prod LAN.
- [ ] Run `playbooks/cilium.yml -e k3s_hosts=k3s_prod` → CNI at bootstrap; Flux's HelmRelease adopts
it afterwards. The bootstrap values and the HelmRelease **must stay in parity**
(`kubeProxyReplacement`, `k8sServiceHost`/`Port`) or the two layers fight over CNI config
(`docs/02`). Note the prod kubeconfig is fetched to `.cache/prod.kubeconfig`, never over the
dev one the dev container exports as `$KUBECONFIG`.
- [ ] Run `playbooks/verify.yml -e k3s_hosts=k3s_prod`.
- [ ] `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod` **retargets but is not yet correct**:
`roles/flux_bootstrap` shells out to `bootstrap/flux/bootstrap.sh`, which bootstraps
`--path=clusters/dev`. Prod needs `--path=clusters/prod` — a `bootstrap/` change, not a
target-group change. Tracked with the other repo blockers in §7.
- [ ] Decide whether prod runs a VPN egress gateway node (dev uses `k3s-dev-3`, ADR-0015).
`playbooks/vpn.egress.gateway.yml` takes `-e vpn_egress_gateway_hosts=<group>`, but there is
deliberately no prod egress group in the inventory yet, and it would need its **own** vaulted
PIA credentials — the existing vault is scoped to the dev group.
## 7. Repo blockers — **prod cannot be reconciled until these land**
+82 -15
View File
@@ -35,9 +35,10 @@ ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOP
> **`site.yml` is NOT everything.** It covers the core bring-up only. Two node-host
> layers live OUTSIDE `site.yml` and must be run separately:
> ```
> ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); targets k3s_servers
> ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; targets k3s-dev-3
> ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); defaults to k3s_servers
> ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3
> ```
> (Both take the same target-group override as the rest — see **Targeting prod** below.)
## Re-provisioning a node (tofu -replace / fresh OS)
A node rebuild wipes **everything on the host filesystem** — including host-level config
@@ -88,24 +89,90 @@ expose different entrypoints, override the two vars in
`roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged
for PM reconciliation.
## Prod inventory — scaffolding only, Phase 7 PREP (nothing above is run against it)
## Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)
`inventory/hosts.ini` also carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
`inventory/hosts.ini` carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
`[k3s_prod:children]` section for the three planned bare-metal nodes (`k3s-prod-1` at
`192.168.2.10` exists and is reachable today; `k3s-prod-2`/`-3` at `.11`/`.12` are addressed but
not yet racked, so their inventory lines stay commented out), with its own `group_vars/k3s_prod/`
(vaulted `ansible_become_password` + non-secret overrides). Note `192.168.2.20` is prod's
**ingress** Gateway VIP (a Flux/`cluster-vars` concern, not an Ansible one — it does not appear in
`group_vars/k3s_prod/`); the kube-vip **control-plane** VIP (`kube_vip_address`) is a *separate*,
still-undecided address, left as a loud BLOCKED placeholder alongside the `node_baseline` device
overrides. See `group_vars/k3s_prod/README.md` and `infra/PROD-PROVISIONING.md` (T-7.1) §4/§2 for
the open decisions that still block a real run.
(vaulted `ansible_become_password` + non-secret overrides).
Everything else in this README describes the **dev** runbook. It does not (yet) apply to prod
as-is: the run order above and the roles it calls (`k3s_server`, `cluster_verify`, ...) target the
literal `k3s_primary` / `k3s_secondary` / `k3s_servers` groups, not `k3s_prod_*` — see the GAP note
at the top of `inventory/hosts.ini`'s prod section for what's still missing before that reuse (the
R-ENV-02 promise) actually works end to end.
**Nothing here is forked for prod.** Every node-host play takes its target group from a variable
that defaults to the dev group, so the dev commands above are unchanged, and prod is the same
command plus an override. Two separate mechanisms, for two separate reasons:
| Layer | Knob | Set via | Why |
|---|---|---|---|
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts`, `vpn_egress_gateway_hosts` | `-e` **extra vars only** | Ansible templates a play's `hosts:` *before* it has any host, so it cannot see `group_vars` at all |
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group` (+ `cluster_kubeconfig_name`) | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml` | Host context exists by then, so each environment answers for itself with **no per-run flag** — same mechanism `k3s_api_host` already used |
`playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
bring-up chain.
```bash
# dev — completely unchanged, no extra vars
ansible-playbook playbooks/site.yml
# prod — one flag for the whole-cluster plays
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod
ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod # see caveat below
# prod — k3s.yml needs both halves of the primary/secondary split
ansible-playbook playbooks/k3s.yml \
-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
# prod — the whole chain (extra vars are global, so they reach imported playbooks)
ansible-playbook playbooks/site.yml \
-e k3s_hosts=k3s_prod \
-e k3s_primary_hosts=k3s_prod_primary \
-e k3s_secondary_hosts=k3s_prod_secondary
```
What the roles then resolve **by themselves** for a prod host, from `group_vars/k3s_prod/vars.yml`:
`k3s_primary_group=k3s_prod_primary` (the `--cluster-init` founder, the node-token `delegate_to`
target, and the node Cilium fetches the kubeconfig from), `k3s_servers_group=k3s_prod` (the tls-san
node-IP list), `k3s_api_host=192.168.2.10`, `kube_vip_address=192.168.2.9`, and
`cluster_kubeconfig_name=prod.kubeconfig` — so a prod run can never overwrite the
`.cache/dev.kubeconfig` that `infra/devcontainer/dev.sh` exports as `$KUBECONFIG`.
### Packages-only baseline (`--tags packages`)
`roles/node_baseline` is tagged `packages` / `system` / `disks`. An untagged run is unchanged (all
three, in order). `--tags packages` runs **only** the apt task:
```bash
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
```
This exists to break a circular dependency: prod's disk layout (`infra/PROD-PROVISIONING.md` §2) is
blocked on a SMART read, SMART needs `smartmontools`, `smartmontools` ships in
`node_baseline_packages` — but the role's disk tasks need block devices that decision hasn't chosen
yet. With `--tags packages` no `disks`-tagged task is selected, so the deliberate
`/dev/BLOCKED-pending-disk-decision-…` placeholders in `group_vars/k3s_prod/vars.yml` are never
even templated, let alone handed to `mkfs`. A **full** run against a node whose devices are still
placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never
hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).
### Still not runnable end to end
- The **disk-layout** (§2) and the `node_baseline_*_device` overrides remain BLOCKED placeholders.
- `group_vars/k3s_prod/vault.yml` (the become password) does not exist yet, so **no** prod play can
escalate — and `--check` against prod fails for the same reason.
- `flux-bootstrap.yml` retargets, but `roles/flux_bootstrap` shells out to
`bootstrap/flux/bootstrap.sh`, which bootstraps `--path=clusters/dev`. Prod needs
`--path=clusters/prod`: a `bootstrap/` change, not a target-group change (§7).
- `vpn.egress.gateway.yml` is parameterised (`-e vpn_egress_gateway_hosts=…`) but prod has **no**
egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own
vaulted PIA credentials.
- The operator has ruled the prod cluster stays **uninitialized** until all three nodes are present
(§8). `192.168.2.20` is prod's *ingress* Gateway VIP — a Flux/`cluster-vars` concern, not an
Ansible one; the kube-vip **control-plane** VIP is `192.168.2.9` (`kube_vip_address`).
Everything above this section describes the **dev** runbook; the values in it (`192.168.1.x`,
`.cache/dev.kubeconfig`, `clusters/dev`) are dev's, supplied by `group_vars`, not baked into the
roles.
## Static validation (PM, in dev container — NOT run here)
```
+24 -1
View File
@@ -4,12 +4,35 @@
# cloud-init created this login user (tofu var login_user); it has NOPASSWD sudo.
ansible_user: k3s
# --- environment group NAMES (R-ENV-02: one set of roles, scoped overrides) ---------
# The GROUP NAMES a role uses to answer "who is MY environment's primary / MY
# environment's servers?". Roles must never hardcode `groups['k3s_primary']`, or they
# only ever work against dev. These two are the dev values; group_vars/k3s_prod/vars.yml
# overrides them with k3s_prod_primary / k3s_prod, so a host in the prod group resolves
# its own primary automatically — exactly the way k3s_api_host below is overridden there.
#
# These are GROUP NAMES (keys into `groups`), NOT host patterns. The play-level target
# pattern is a separate, extra-vars-only knob (`k3s_hosts` / `k3s_primary_hosts` /
# `k3s_secondary_hosts`, see playbooks/*.yml) because Ansible templates a play's `hosts:`
# before it has any host context and so cannot see group_vars at all.
k3s_primary_group: k3s_primary
k3s_servers_group: k3s_servers
# Filename of the kubeconfig the controller-side roles (cilium, cluster_verify,
# flux_bootstrap) fetch from the primary into infra/ansible/.cache/. Scoped per
# environment so pointing those plays at prod can NEVER overwrite the dev kubeconfig
# that infra/devcontainer/dev.sh exports as $KUBECONFIG — an operator's `kubectl get
# nodes` silently answering from prod would be a genuinely dangerous mix-up.
# group_vars/k3s_prod/vars.yml overrides this to prod.kubeconfig. (.cache/ is gitignored;
# a kubeconfig is a credential.)
cluster_kubeconfig_name: dev.kubeconfig
# The kube-apiserver endpoint the whole stack agrees on: the primary's REAL node IP.
# NOTE: ansible_host is now the pfSense NAT SSH target (192.168.2.102:225x), NOT the node
# IP — so we read the real 192.168.1.x from k3s_node_ip instead. Used for k3s --tls-san /
# secondary --server AND for Cilium k8sServiceHost so the Ansible bootstrap and the Flux
# HelmRelease stay in PARITY (docs/02 contract).
k3s_api_host: "{{ hostvars[groups['k3s_primary'][0]]['k3s_node_ip'] }}"
k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['k3s_node_ip'] }}"
k3s_api_port: 6443
# Control-plane HA VIP floated by kube-vip (roles/kube_vip). A stable failover
@@ -54,9 +54,15 @@ behind them instead of NOPASSWD.
- This is the Ansible-bootstrap-layer secret (host credential), distinct from the platform's
in-cluster **SOPS + age** secrets (ADR-0007) — different layer, different tool, same convention
as `hypervisor`/`proxmox_hosts`.
- **Targeting prod is solved** (2026-08-31): plays take `-e k3s_hosts=k3s_prod` (and, for `k3s.yml`,
`-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`), while the
roles resolve "my environment's primary/servers" from `k3s_primary_group` / `k3s_servers_group`
in `vars.yml` here — no role fork, no per-run flag for that half. See
`infra/ansible/README.md` → "Targeting prod".
- **Do not run any prod playbook yet.** Phase 7 is prep-only: the vault above just makes the plays
*able* to authenticate once the §7 repo blockers close and the disk-layout (§2) and
control-plane-VIP (§4) decisions in `infra/PROD-PROVISIONING.md` are made — and, separately, once
the group-name gap flagged in `inventory/hosts.ini`'s prod section is resolved (the existing
playbooks target `k3s_servers` / `k3s_primary` / `k3s_secondary` literally, not `k3s_prod_*`).
See that file for the full picture.
*able* to authenticate once the §7 repo blockers close and the disk-layout (§2) decision in
`infra/PROD-PROVISIONING.md` is made. The ONE exception the operator has sanctioned is the
packages-only baseline that unblocks the §2 drive-health read —
`ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages` — which
cannot touch the BLOCKED device placeholders because every disk task is `disks`-tagged and so is
not selected. It still needs `vault.yml` (above) to exist first.
@@ -29,13 +29,34 @@ ansible_become: true
# The password itself is ansible_become_password — ansible-vault-encrypted, in
# vault.yml (this directory), never here in plaintext. See README.md to create it.
# --- environment group NAMES (R-ENV-02) -------------------------------------------
# group_vars/all.yml defines these as the DEV group names (k3s_primary / k3s_servers);
# here they name PROD's groups. Roles index `groups[k3s_primary_group]` /
# `groups[k3s_servers_group]` instead of a literal, so roles/k3s_server and roles/cilium
# resolve "my environment's primary" and "my environment's servers" correctly for prod
# hosts with NO role fork and no per-run flag — the reuse R-ENV-02 promises.
#
# NOTE these are GROUP NAMES, not the play-level target pattern. Selecting which hosts a
# play RUNS against is a separate extra-vars knob (`-e k3s_hosts=k3s_prod`, and for
# k3s.yml `-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`),
# because Ansible templates `hosts:` before any host — and therefore any group_vars —
# exists. See infra/ansible/README.md "Targeting prod".
k3s_primary_group: k3s_prod_primary
k3s_servers_group: k3s_prod
# Keep the fetched prod kubeconfig in its OWN cache file. group_vars/all.yml defaults this
# to dev.kubeconfig, which is the file infra/devcontainer/dev.sh exports as $KUBECONFIG —
# a prod play must never clobber it (a `kubectl` that silently answers from prod is worse
# than no kubeconfig at all). .cache/ is gitignored; a kubeconfig is a credential.
cluster_kubeconfig_name: prod.kubeconfig
# --- k3s API host — safe to set now, the primary's real IP is already known -------
# Dev's k3s_api_host (group_vars/all.yml) reads a separate k3s_node_ip hostvar because
# dev's ansible_host is the pfSense NAT SSH target, not the node's real address. Prod
# has NO NAT hop — ansible_host in hosts.ini IS already the node's real, routable LAN
# address (see the KEY DIFFERENCE note there) — so point straight at it instead of
# needing a k3s_node_ip split.
k3s_api_host: "{{ hostvars[groups['k3s_prod_primary'][0]]['ansible_host'] }}"
k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}"
# cluster_controller_tls_server_name (group_vars/all.yml) derives from kube_vip_address
# by template reference, so it needs no separate override here — it picks up the real
# value set below automatically for prod hosts too.
+15 -13
View File
@@ -66,19 +66,21 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
# `ansible_port` is the node's own real sshd (22). See group_vars/k3s_prod/vars.yml for
# the k3s_api_host override that reflects this directly-reachable topology.
#
# GAP flagged for the PM, NOT fixed in this pass (inventory/group_vars only): the
# k3s_server role hardcodes the LITERAL group names `k3s_primary` / `k3s_secondary` /
# `k3s_servers` in roles/k3s_server/defaults/main.yml (tls-san SAN list) and
# roles/k3s_server/tasks/main.yml (node-token delegate_to), and every node-host
# playbook (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, verify.yml,
# flux-bootstrap.yml) declares `hosts: k3s_servers` / `k3s_primary` / `k3s_secondary`
# literally. Naming prod's groups k3s_prod_primary/k3s_prod_secondary — as this
# checklist and PROD-PROVISIONING.md §6 both call for — means those existing
# playbooks will NOT select prod hosts as they stand today. That's harmless right now
# (nothing can accidentally run against prod), but it means actually reusing the dev
# playbooks against prod later (the R-ENV-02 promise) needs either parametrizing the
# group name they target or some other mechanism — not yet designed, tracked here so
# it isn't lost before §6 is attempted for real.
# GROUP-NAME GAP: CLOSED 2026-08-31. The roles and playbooks used to hardcode the
# LITERAL dev groups (`hosts: k3s_servers` / `k3s_primary` / `k3s_secondary`, and
# `groups['k3s_primary']` inside roles/k3s_server + roles/cilium), so nothing could
# select the k3s_prod_* groups below. Both halves are now variables, deliberately kept
# separate because Ansible resolves them at different times:
# * WHICH HOSTS A PLAY RUNS ON — `k3s_hosts` (whole-cluster plays), `k3s_primary_hosts`
# / `k3s_secondary_hosts` (k3s.yml's split), each defaulting to the dev group.
# EXTRA VARS ONLY: a play's `hosts:` is templated before any host exists, so
# group_vars cannot be read there. Prod: `-e k3s_hosts=k3s_prod` etc.
# * WHICH GROUP A ROLE MEANS BY "MY PRIMARY"/"MY SERVERS" — `k3s_primary_group` /
# `k3s_servers_group`, set in group_vars/all.yml (dev) and group_vars/k3s_prod/vars.yml
# (prod), so roles resolve their own environment with NO per-run flag — the same
# mechanism k3s_api_host already used.
# Running with no extra vars is byte-for-byte the dev behaviour it always had. See
# infra/ansible/README.md "Targeting prod" and PROD-PROVISIONING.md §6.
# =============================================================================
[k3s_prod_primary]
+6 -1
View File
@@ -1,8 +1,13 @@
---
# Install Cilium as the CNI (ADR-0002). The cluster is NotReady until this runs.
# Executed from the controller against a fetched kubeconfig (role handles delegation).
#
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
# ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod
# The role's "fetch the kubeconfig from the primary" delegation follows the
# k3s_primary_group group_var, so it picks the prod primary automatically.
- name: Install Cilium CNI
hosts: k3s_servers
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
become: true
gather_facts: false
roles:
+8 -1
View File
@@ -1,8 +1,15 @@
---
# FINAL step: hand off to Flux + SOPS (T-1.4). Calls into bootstrap/ (authored by
# the Security + GitOps Engineers) — this play does not author that subtree.
#
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
# ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod
# NOTE the Flux PATH is a separate concern: roles/flux_bootstrap shells out to
# bootstrap/flux/bootstrap.sh, which bootstraps --path=clusters/dev. Prod needs
# --path=clusters/prod, which is a bootstrap/ change, NOT a target-group change
# (infra/PROD-PROVISIONING.md §7 tracks it).
- name: Flux + SOPS bootstrap hand-off
hosts: k3s_servers
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
become: false
gather_facts: false
roles:
+13 -2
View File
@@ -2,9 +2,20 @@
# Install the HA k3s control plane (embedded etcd, 3 servers) — ADR-0001.
# Primary founds the cluster (--cluster-init); secondaries join it in a second
# play so the primary's node-token fact is available to them.
#
# TARGET (R-ENV-02): this is the one playbook that needs TWO patterns, because the
# primary/secondary split is the whole point of the file. Both default to dev:
# ansible-playbook playbooks/k3s.yml # dev, unchanged
# ansible-playbook playbooks/k3s.yml \
# -e k3s_primary_hosts=k3s_prod_primary \
# -e k3s_secondary_hosts=k3s_prod_secondary # prod
# The role's OWN "who is the primary" lookups (join token delegate_to, tls-san list)
# do NOT use these — they follow the k3s_primary_group / k3s_servers_group group_vars,
# which prod sets once in group_vars/k3s_prod/vars.yml. These two vars only choose which
# hosts the plays RUN on, which Ansible resolves before any group_vars exist.
- name: k3s primary (cluster-init)
hosts: k3s_primary
hosts: "{{ k3s_primary_hosts | default('k3s_primary') }}"
become: true
gather_facts: true
roles:
@@ -16,7 +27,7 @@
# an updated tls-san set) triggers the Restart k3s handler: restarting both
# secondaries at once would drop etcd below quorum. One-at-a-time keeps 2/3 up.
- name: k3s secondaries (join)
hosts: k3s_secondary
hosts: "{{ k3s_secondary_hosts | default('k3s_secondary') }}"
become: true
serial: 1
gather_facts: true
+7 -2
View File
@@ -1,9 +1,14 @@
---
# Deploy kube-vip as the control-plane HA VIP (192.168.1.49) for the kube-API.
# Deploy kube-vip as the control-plane HA VIP (192.168.1.49 on dev) for the kube-API.
# Control-plane ONLY (svc_enable=false) — Cilium owns Service LoadBalancing.
# Writes the manifest into each server's k3s auto-deploy dir; k3s applies it.
#
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
# ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
# (the VIP itself comes from kube_vip_address, already scoped per environment in
# group_vars — dev 192.168.1.49, prod 192.168.2.9).
- name: Deploy kube-vip control-plane VIP
hosts: k3s_servers
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
become: true
gather_facts: true
roles:
+14 -1
View File
@@ -2,8 +2,21 @@
# Node baseline — first-boot floor for k3s/Cilium/Longhorn, moved from cloud-init into
# Ansible so Proxmox provisioning stays API-only (no SSH/SFTP to the hypervisor). Runs
# FIRST in site.yml, before k3s. Idempotent; safe to re-run.
#
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
# See README.md "Targeting prod". The pattern is the one node-health.yml established.
#
# TAGS — this role is the only way `smartmontools` (and the rest of the package floor)
# reaches a node, but its data-disk tasks need block devices that prod has NOT decided
# yet (infra/PROD-PROVISIONING.md §2). Run JUST the package floor with:
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
# `--tags packages` runs the apt task ONLY; every disk task carries the `disks` tag and
# is therefore skipped, so the BLOCKED placeholder devices can never be touched. A full
# (untagged) run against a node whose device vars are still placeholders fails fast on an
# explicit assert instead of handing a bogus path to mkfs. See roles/node_baseline.
- name: Baseline the k3s nodes (packages, kernel modules, Longhorn disk, swap-off)
hosts: k3s_servers
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
become: true
gather_facts: true
roles:
+9 -7
View File
@@ -6,13 +6,15 @@
# ./dev.sh run bash -c 'cd /work/infra/ansible && \
# ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod'
#
# TARGET GROUP IS A VARIABLE, unlike the other playbooks here. Every existing node
# playbook (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, verify.yml) hardcodes
# `hosts: k3s_servers` / `k3s_primary` / `k3s_secondary`, which is exactly why they cannot
# select the prod groups today (the gap flagged in inventory/hosts.ini's prod section, and
# the R-ENV-02 reuse promise it threatens). A health check is the one play you want to be
# able to point anywhere, so it takes the pattern as a var and defaults to dev. Run it
# against prod with `-e node_health_hosts=k3s_prod`, or a single box with `--limit`.
# TARGET GROUP IS A VARIABLE. This play established that pattern; the rest of the node
# playbooks (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, verify.yml,
# flux-bootstrap.yml) now follow it and take `k3s_hosts` / `k3s_primary_hosts` /
# `k3s_secondary_hosts` instead of the literal dev groups they used to hardcode — see
# infra/ansible/README.md "Targeting prod". This one keeps its OWN var name because it is
# a standalone diagnostic you point anywhere, not a step in the bring-up chain: it must
# stay independently targetable (`-e node_health_hosts=<group>`) even while a bring-up run
# is scoped elsewhere. Run it against prod with `-e node_health_hosts=k3s_prod`, or a
# single box with `--limit`.
#
# BECOME: left at the play level exactly as the inventory sets it. The role itself pins
# `ansible_become: false` on every unprivileged task and probes escalation once before
+12
View File
@@ -2,6 +2,18 @@
# Full dev/test cluster bring-up, in order (docs/02 sequencing; run AFTER
# `tofu apply` has created and cloud-init has baselined the VMs):
# k3s HA install -> Cilium CNI -> verify readiness -> Flux + SOPS hand-off.
#
# TARGET (R-ENV-02): every imported play takes its host pattern from a var that
# defaults to the dev group, so `ansible-playbook playbooks/site.yml` with no extra
# vars is byte-for-byte the dev run it always was. Extra vars are global and reach
# imported playbooks, so the whole chain retargets with three flags:
# ansible-playbook playbooks/site.yml \
# -e k3s_hosts=k3s_prod \
# -e k3s_primary_hosts=k3s_prod_primary \
# -e k3s_secondary_hosts=k3s_prod_secondary
# DO NOT run that yet — prod is blocked on the disk-layout decision
# (infra/PROD-PROVISIONING.md §2) and the §7 repo blockers, and the operator has ruled
# the prod cluster stays uninitialized until all three nodes are present (§8).
- name: Import - node baseline (packages, kernel modules, Longhorn disk, swap-off)
ansible.builtin.import_playbook: node-baseline.yml
+4 -1
View File
@@ -1,7 +1,10 @@
---
# Verify Phase-1 exit gate #1: 3 nodes Ready, Cilium/Hubble healthy, no kube-proxy.
#
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
# ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod
- name: Verify cluster readiness
hosts: k3s_servers
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
become: false
gather_facts: false
roles:
+10 -2
View File
@@ -3,15 +3,23 @@
# (ADR-0015). Host infra — the split-tunnel OpenVPN service + policy routing + scoped
# nftables kill-switch. The CiliumEgressGatewayPolicy that steers vpn-class pods to this
# node's tun0 is authored separately as GitOps (platform/), against the same contract:
# * inventory group `vpn_egress_gateway` (this play's target)
# * inventory group `vpn_egress_gateway` (this play's default target)
# * k8s node label homelab.keithjackson.dev/egress-gateway=true (set by the role)
# * host tunnel iface tun0 (split-tunnel; never the node default route)
#
# Secrets come from Ansible vault (inventory/group_vars/vpn_egress_gateway/vault.yml):
# vault_vpn_egress_gateway_pia_username, vault_vpn_egress_gateway_pia_password, vault_vpn_egress_gateway_pia_ovpn
# Run: ansible-playbook playbooks/vpn.egress.gateway.yml
#
# TARGET (R-ENV-02): the group is a var, defaulting to dev's `vpn_egress_gateway`. Prod
# has NOT decided whether it runs an egress gateway at all, nor on which node
# (infra/PROD-PROVISIONING.md §6, last box) — so there is deliberately no prod egress
# group in the inventory yet. When there is, point this at it with
# -e vpn_egress_gateway_hosts=<that group>
# and it needs its OWN vaulted PIA credentials group_vars; this play's vault is scoped to
# the dev group.
- name: Configure the node-level VPN egress gateway
hosts: vpn_egress_gateway
hosts: "{{ vpn_egress_gateway_hosts | default('vpn_egress_gateway') }}"
become: true
gather_facts: true # package module + service facts need os facts
roles:
+4 -2
View File
@@ -19,8 +19,10 @@ gateway_api_crd_urls:
# TLSRoute (experimental) is required by Cilium's Gateway API support.
- "https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/{{ gateway_api_version }}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml"
# Local kubeconfig fetched from the primary. *.kubeconfig is gitignored.
cilium_kubeconfig: "{{ playbook_dir }}/../.cache/dev.kubeconfig"
# Local kubeconfig fetched from the primary. *.kubeconfig is gitignored. The FILENAME
# comes from cluster_kubeconfig_name (group_vars) so a prod run cannot overwrite the dev
# kubeconfig the dev container exports as $KUBECONFIG. R-ENV-02.
cilium_kubeconfig: "{{ playbook_dir }}/../.cache/{{ cluster_kubeconfig_name | default('dev.kubeconfig') }}"
cilium_values_path: "{{ playbook_dir }}/../.cache/cilium-values.yaml"
# dev: single operator replica to save RAM (PHASE1-SCOPE RAM ceiling). NOT a
+4 -1
View File
@@ -4,11 +4,14 @@
# The cluster is NotReady until this completes — expected (PHASE1-SCOPE risks).
# run_once: this whole role acts on the cluster once, not per host.
# k3s_primary_group (group_vars) names THIS environment's primary group — `k3s_primary`
# on dev, `k3s_prod_primary` on prod — so the role fetches the right cluster's kubeconfig
# without being forked per environment (R-ENV-02).
- name: Fetch the k3s kubeconfig from the primary
ansible.builtin.slurp:
src: /etc/rancher/k3s/k3s.yaml
register: k3s_kubeconfig_raw
delegate_to: "{{ groups['k3s_primary'][0] }}"
delegate_to: "{{ groups[k3s_primary_group][0] }}"
run_once: true
- name: Ensure the local cache dir exists
@@ -1,4 +1,6 @@
---
# cluster_verify role defaults.
cluster_verify_kubeconfig: "{{ playbook_dir }}/../.cache/dev.kubeconfig"
# Filename comes from cluster_kubeconfig_name (group_vars) so dev and prod never share
# one cache file — dev.kubeconfig for dev, prod.kubeconfig for prod. R-ENV-02.
cluster_verify_kubeconfig: "{{ playbook_dir }}/../.cache/{{ cluster_kubeconfig_name | default('dev.kubeconfig') }}"
cluster_verify_node_ready_timeout: "300s"
@@ -7,7 +7,8 @@
# args below are ASSUMPTIONS the PM must reconcile with those deliverables — see
# roles/flux_bootstrap/tasks/main.yml and infra/ansible/README.md "Interface".
flux_bootstrap_kubeconfig: "{{ playbook_dir }}/../.cache/dev.kubeconfig"
# Filename from cluster_kubeconfig_name (group_vars): dev.kubeconfig / prod.kubeconfig.
flux_bootstrap_kubeconfig: "{{ playbook_dir }}/../.cache/{{ cluster_kubeconfig_name | default('dev.kubeconfig') }}"
# Repo root, relative to playbooks/ (infra/ansible/playbooks -> ../../..).
flux_bootstrap_repo_root: "{{ playbook_dir }}/../../.."
@@ -26,7 +26,11 @@ k3s_server_mode: "secondary"
# the VIP), and 127.0.0.1 (local kubeconfig / health probes). k3s regenerates the
# serving cert on restart when this set changes. Deduplicated (the primary IP is
# also k3s_api_host).
#
# k3s_servers_group — NOT a literal group name — is what makes this role environment-
# agnostic (R-ENV-02): group_vars/all.yml sets it to `k3s_servers` (dev) and
# group_vars/k3s_prod/vars.yml to `k3s_prod`, so each environment SANs its own servers.
k3s_tls_sans: >-
{{ ([k3s_api_host, kube_vip_address, '127.0.0.1']
+ (groups['k3s_servers'] | map('extract', hostvars, 'k3s_node_ip') | list))
+ (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list))
| unique }}
@@ -121,19 +121,23 @@
# contained — delegate_to still reaches the primary even when it isn't a play
# target, so the join no longer depends on a fact an earlier play had to set.
# T-1.5 finding F2 (see DECISIONS-LOG.md).
#
# "The primary" is resolved through k3s_primary_group (group_vars: `k3s_primary` for
# dev, `k3s_prod_primary` for prod), never a literal group name — so a prod secondary
# delegates to the PROD primary using this same role, unchanged (R-ENV-02).
- name: Fetch the k3s node-token directly from the primary (secondary)
ansible.builtin.slurp:
src: /var/lib/rancher/k3s/server/token
delegate_to: "{{ groups['k3s_primary'][0] }}"
delegate_to: "{{ groups[k3s_primary_group][0] }}"
register: k3s_primary_token_raw
when:
- k3s_server_mode == 'secondary'
- hostvars[groups['k3s_primary'][0]].k3s_node_token is not defined
- hostvars[groups[k3s_primary_group][0]].k3s_node_token is not defined
- name: Resolve the join token (secondary)
ansible.builtin.set_fact:
k3s_join_token: >-
{{ hostvars[groups['k3s_primary'][0]].k3s_node_token
{{ hostvars[groups[k3s_primary_group][0]].k3s_node_token
| default(k3s_primary_token_raw.content | b64decode | trim, true) }}
when: k3s_server_mode == 'secondary'
@@ -1,10 +1,23 @@
---
# TAGS — every task here carries one of `packages`, `system` or `disks`. An untagged run
# is unchanged (all three run, in this order). The split exists because the PACKAGE floor
# and the DATA-DISK work have different readiness: packages are safe on any node from
# first contact, while the disk tasks need block devices that a new environment may not
# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
# on a SMART read that needs `smartmontools` — which ships in node_baseline_packages).
# `--tags packages` breaks that circle: it installs the package floor and NOTHING else.
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
# Skipping is the guard: with `--tags packages` no `disks`-tagged task is selected, so a
# BLOCKED-pending-disk-decision device path is never even templated, let alone handed to
# mkfs. A FULL run against such a node stops at the assert below instead.
- name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
ansible.builtin.apt:
name: "{{ node_baseline_packages }}"
state: present
update_cache: true
cache_valid_time: 3600
tags: [packages]
- name: Persist kernel modules for k3s/Cilium/Longhorn
ansible.builtin.copy:
@@ -13,12 +26,14 @@
group: root
mode: "0644"
content: "{{ node_baseline_kernel_modules | join('\n') }}\n"
tags: [system]
- name: Load kernel modules now
community.general.modprobe:
name: "{{ item }}"
state: present
loop: "{{ node_baseline_kernel_modules }}"
tags: [system]
- name: Install sysctl tuning for k3s/Cilium
ansible.builtin.copy:
@@ -34,12 +49,14 @@
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 524288
notify: Reload sysctl
tags: [system]
- name: Enable and start iscsid (Longhorn prerequisite)
ansible.builtin.systemd:
name: iscsid
enabled: true
state: started
tags: [system]
# qemu-guest-agent binds to the virtio-serial channel /dev/virtio-ports/org.qemu.guest_agent.0,
# which Proxmox attaches only when the VM's `agent` option is enabled. Our tofu keeps agent
@@ -51,18 +68,44 @@
ansible.builtin.stat:
path: /dev/virtio-ports/org.qemu.guest_agent.0
register: node_baseline_qga_channel
tags: [system]
- name: Enable qemu-guest-agent (start only when its virtio channel is present)
ansible.builtin.systemd:
name: qemu-guest-agent
enabled: true
state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
tags: [system]
# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
# which do not exist on bare metal either — a confusing failure that looks like a wiring
# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
- name: Assert this environment's data-disk devices have actually been decided
ansible.builtin.assert:
that:
- "'BLOCKED' not in node_baseline_longhorn_device"
- "'BLOCKED' not in node_baseline_etcd_device"
fail_msg: >-
node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
placeholders for this host's environment
(longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
the SMART read that decision waits on can be taken) re-run this play with
`--tags packages`, which skips every disk task including this assert.
quiet: true
tags: [disks]
- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
community.general.filesystem:
fstype: ext4
dev: "{{ node_baseline_longhorn_device }}"
opts: "-L {{ node_baseline_longhorn_fs_label }}"
tags: [disks]
- name: Mount the Longhorn data disk
ansible.posix.mount:
@@ -71,6 +114,7 @@
fstype: ext4
opts: "defaults,nofail,x-systemd.device-timeout=5s"
state: mounted
tags: [disks]
# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
@@ -84,6 +128,7 @@
fstype: ext4
dev: "{{ node_baseline_etcd_device }}"
opts: "-L {{ node_baseline_etcd_fs_label }}"
tags: [disks]
- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
ansible.posix.mount:
@@ -96,13 +141,16 @@
dump: "0"
passno: "2"
state: mounted
tags: [disks]
- name: Disable swap now
ansible.builtin.command: swapoff -a
changed_when: false
tags: [system]
- name: Neutralize swap entries in fstab (kubelet requires swap off)
ansible.builtin.replace:
path: /etc/fstab
regexp: '^([^#].*\sswap\s.*)$'
replace: '# \1 # disabled for k3s/kubelet'
tags: [system]