feat(ansible): parameterise target groups so prod reuses the roles unchanged (R-ENV-02)
roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.
TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
* which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
(k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
* which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
Follows the k3s_api_host convention already in this repo. No per-run flag.
DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.
BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).
FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.
KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.
Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.
Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
1157c1b071
commit
e623d27e2d
22 files changed
+384
-76
No files matched your search
+74
-15
@@ -84,7 +84,14 @@ Consequences to settle before imaging nodes 2 and 3:
|
||||
- **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one
|
||||
physical failure domain.
|
||||
- [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry.
|
||||
- [ ] Record the chosen device paths here and set them in `inventory/group_vars/k3s_prod.yml`.
|
||||
- [ ] Record the chosen device paths here and set them in
|
||||
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately
|
||||
`/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full
|
||||
`node-baseline` run against a node still carrying those placeholders now stops on an explicit
|
||||
assert naming this section, instead of failing obscurely at `mkfs` time.
|
||||
- [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline`
|
||||
is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk
|
||||
tasks this decision blocks — see §5's drive-health box for the exact command.
|
||||
|
||||
## 3. OS install (per node)
|
||||
|
||||
@@ -149,29 +156,81 @@ Consequences to settle before imaging nodes 2 and 3:
|
||||
- [ ] **Drive health baseline** (§2). **Do not hand-install the tooling** — package installation is
|
||||
the `node_baseline` role's job, never a manual `apt-get` on a node (operator ruling,
|
||||
2026-08-31; same click-ops-never principle as the rest of the repo). `smartmontools` is part
|
||||
of `node_baseline_packages`, so it arrives with the baseline play. Reading the drive is then:
|
||||
of `node_baseline_packages`, so it arrives with the baseline play — but a *full* baseline
|
||||
would hit the disk tasks §2 has not decided yet. Run the **packages-only** baseline instead
|
||||
(the `disks`-tagged tasks, including the BLOCKED-device assert, are not selected at all, so
|
||||
the placeholder device paths are never even templated):
|
||||
```bash
|
||||
cd infra/ansible
|
||||
ansible-playbook playbooks/node-baseline.yml \
|
||||
-e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
|
||||
```
|
||||
(Needs `group_vars/k3s_prod/vault.yml` to exist first — the become password above.) Reading
|
||||
the drive is then:
|
||||
```bash
|
||||
sudo smartctl -a /dev/sda # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors
|
||||
```
|
||||
Record the result in §2 and close the layout decision in the same change that replaces the
|
||||
two `node_baseline_*_device` placeholders.
|
||||
|
||||
## 6. Ansible baseline & cluster join
|
||||
|
||||
Reuses the dev roles unchanged — that reuse **is** R-ENV-02. Only inventory and group_vars differ.
|
||||
Reuses the dev roles **unchanged** — that reuse **is** R-ENV-02. Only inventory and group_vars
|
||||
differ, plus a target-group override on the command line.
|
||||
|
||||
- [ ] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached
|
||||
### How prod is targeted (done, 2026-08-31)
|
||||
|
||||
Every node-host play used to hardcode the literal dev groups `k3s_servers` / `k3s_primary` /
|
||||
`k3s_secondary`, so nothing here could select prod at all. Two knobs now close that, deliberately
|
||||
kept separate because Ansible resolves them at different times:
|
||||
|
||||
| Layer | Knob | Set via |
|
||||
|---|---|---|
|
||||
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts` (each defaults to the dev group) | `-e` **extra vars only** — a play's `hosts:` is templated before any host exists, so `group_vars` is not readable there |
|
||||
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group`, `cluster_kubeconfig_name` | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml`; **no per-run flag** |
|
||||
|
||||
So `roles/k3s_server` finds the prod `--cluster-init` founder, builds the prod join target and
|
||||
tls-san list, and `roles/cilium` fetches the prod kubeconfig, with **no role fork**. Running with
|
||||
no extra vars is byte-for-byte the dev behaviour it always had (verified by a before/after
|
||||
`--list-hosts` diff on every playbook). Full detail: `infra/ansible/README.md` → **Targeting prod**.
|
||||
|
||||
- [x] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached
|
||||
**directly** on the LAN — no NAT port-forward hop, unlike dev's `192.168.2.102:225x` mapping.
|
||||
- [ ] `group_vars/k3s_prod.yml`: `kube_vip_address`, `cluster_controller_api_server`, and the §2
|
||||
device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`.
|
||||
- [ ] Run `playbooks/node-baseline.yml` → packages, kernel modules, sysctls, iscsid, data-disk
|
||||
filesystems + mounts. (`qemu-guest-agent` is a no-op on bare metal — the role already only
|
||||
*starts* it when the virtio channel exists, so it degrades cleanly.)
|
||||
- [ ] Run `playbooks/k3s.yml` → HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled.
|
||||
**Only once all three nodes are present** — see §8.
|
||||
- [ ] Run `playbooks/kube-vip.yml` → control-plane VIP on the prod LAN.
|
||||
- [ ] Run `playbooks/cilium.yml` → CNI at bootstrap; Flux's HelmRelease adopts it afterwards. The
|
||||
bootstrap values and the HelmRelease **must stay in parity** (`kubeProxyReplacement`,
|
||||
`k8sServiceHost`/`Port`) or the two layers fight over CNI config (`docs/02`).
|
||||
(Nodes 2/3 stay commented out until they are physically on the bench.)
|
||||
- [x] `inventory/group_vars/k3s_prod/vars.yml`: `kube_vip_address` (`192.168.2.9`),
|
||||
`cluster_controller_api_server`, `k3s_api_host`, the group-name overrides above — **and** the
|
||||
§2 device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`, which
|
||||
are still BLOCKED placeholders.
|
||||
- [ ] `inventory/group_vars/k3s_prod/vault.yml` — the vaulted `ansible_become_password` (§5). Until
|
||||
it exists **no** prod play can escalate, and `--check` against prod cannot work either.
|
||||
- [ ] Run the **packages-only** baseline first, to unblock the §2 drive-health read:
|
||||
```bash
|
||||
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
|
||||
```
|
||||
- [ ] Run the **full** `playbooks/node-baseline.yml -e k3s_hosts=k3s_prod` → packages, kernel
|
||||
modules, sysctls, iscsid, data-disk filesystems + mounts. **Blocked until §2 is decided**: it
|
||||
asserts on the placeholder device paths and stops. (`qemu-guest-agent` is a no-op on bare
|
||||
metal — the role already only *starts* it when the virtio channel exists, so it degrades
|
||||
cleanly.)
|
||||
- [ ] Run `playbooks/k3s.yml -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`
|
||||
→ HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. **Only once all three
|
||||
nodes are present** — see §8.
|
||||
- [ ] Run `playbooks/kube-vip.yml -e k3s_hosts=k3s_prod` → control-plane VIP (`192.168.2.9`) on the
|
||||
prod LAN.
|
||||
- [ ] Run `playbooks/cilium.yml -e k3s_hosts=k3s_prod` → CNI at bootstrap; Flux's HelmRelease adopts
|
||||
it afterwards. The bootstrap values and the HelmRelease **must stay in parity**
|
||||
(`kubeProxyReplacement`, `k8sServiceHost`/`Port`) or the two layers fight over CNI config
|
||||
(`docs/02`). Note the prod kubeconfig is fetched to `.cache/prod.kubeconfig`, never over the
|
||||
dev one the dev container exports as `$KUBECONFIG`.
|
||||
- [ ] Run `playbooks/verify.yml -e k3s_hosts=k3s_prod`.
|
||||
- [ ] `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod` **retargets but is not yet correct**:
|
||||
`roles/flux_bootstrap` shells out to `bootstrap/flux/bootstrap.sh`, which bootstraps
|
||||
`--path=clusters/dev`. Prod needs `--path=clusters/prod` — a `bootstrap/` change, not a
|
||||
target-group change. Tracked with the other repo blockers in §7.
|
||||
- [ ] Decide whether prod runs a VPN egress gateway node (dev uses `k3s-dev-3`, ADR-0015).
|
||||
`playbooks/vpn.egress.gateway.yml` takes `-e vpn_egress_gateway_hosts=<group>`, but there is
|
||||
deliberately no prod egress group in the inventory yet, and it would need its **own** vaulted
|
||||
PIA credentials — the existing vault is scoped to the dev group.
|
||||
|
||||
## 7. Repo blockers — **prod cannot be reconciled until these land**
|
||||
|
||||
|
||||
+82
-15
@@ -35,9 +35,10 @@ ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOP
|
||||
> **`site.yml` is NOT everything.** It covers the core bring-up only. Two node-host
|
||||
> layers live OUTSIDE `site.yml` and must be run separately:
|
||||
> ```
|
||||
> ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); targets k3s_servers
|
||||
> ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; targets k3s-dev-3
|
||||
> ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); defaults to k3s_servers
|
||||
> ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3
|
||||
> ```
|
||||
> (Both take the same target-group override as the rest — see **Targeting prod** below.)
|
||||
|
||||
## Re-provisioning a node (tofu -replace / fresh OS)
|
||||
A node rebuild wipes **everything on the host filesystem** — including host-level config
|
||||
@@ -88,24 +89,90 @@ expose different entrypoints, override the two vars in
|
||||
`roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged
|
||||
for PM reconciliation.
|
||||
|
||||
## Prod inventory — scaffolding only, Phase 7 PREP (nothing above is run against it)
|
||||
## Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)
|
||||
|
||||
`inventory/hosts.ini` also carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
|
||||
`inventory/hosts.ini` carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
|
||||
`[k3s_prod:children]` section for the three planned bare-metal nodes (`k3s-prod-1` at
|
||||
`192.168.2.10` exists and is reachable today; `k3s-prod-2`/`-3` at `.11`/`.12` are addressed but
|
||||
not yet racked, so their inventory lines stay commented out), with its own `group_vars/k3s_prod/`
|
||||
(vaulted `ansible_become_password` + non-secret overrides). Note `192.168.2.20` is prod's
|
||||
**ingress** Gateway VIP (a Flux/`cluster-vars` concern, not an Ansible one — it does not appear in
|
||||
`group_vars/k3s_prod/`); the kube-vip **control-plane** VIP (`kube_vip_address`) is a *separate*,
|
||||
still-undecided address, left as a loud BLOCKED placeholder alongside the `node_baseline` device
|
||||
overrides. See `group_vars/k3s_prod/README.md` and `infra/PROD-PROVISIONING.md` (T-7.1) §4/§2 for
|
||||
the open decisions that still block a real run.
|
||||
(vaulted `ansible_become_password` + non-secret overrides).
|
||||
|
||||
Everything else in this README describes the **dev** runbook. It does not (yet) apply to prod
|
||||
as-is: the run order above and the roles it calls (`k3s_server`, `cluster_verify`, ...) target the
|
||||
literal `k3s_primary` / `k3s_secondary` / `k3s_servers` groups, not `k3s_prod_*` — see the GAP note
|
||||
at the top of `inventory/hosts.ini`'s prod section for what's still missing before that reuse (the
|
||||
R-ENV-02 promise) actually works end to end.
|
||||
**Nothing here is forked for prod.** Every node-host play takes its target group from a variable
|
||||
that defaults to the dev group, so the dev commands above are unchanged, and prod is the same
|
||||
command plus an override. Two separate mechanisms, for two separate reasons:
|
||||
|
||||
| Layer | Knob | Set via | Why |
|
||||
|---|---|---|---|
|
||||
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts`, `vpn_egress_gateway_hosts` | `-e` **extra vars only** | Ansible templates a play's `hosts:` *before* it has any host, so it cannot see `group_vars` at all |
|
||||
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group` (+ `cluster_kubeconfig_name`) | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml` | Host context exists by then, so each environment answers for itself with **no per-run flag** — same mechanism `k3s_api_host` already used |
|
||||
|
||||
`playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
|
||||
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
|
||||
bring-up chain.
|
||||
|
||||
```bash
|
||||
# dev — completely unchanged, no extra vars
|
||||
ansible-playbook playbooks/site.yml
|
||||
|
||||
# prod — one flag for the whole-cluster plays
|
||||
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
|
||||
ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod
|
||||
ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
|
||||
ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod
|
||||
ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod # see caveat below
|
||||
|
||||
# prod — k3s.yml needs both halves of the primary/secondary split
|
||||
ansible-playbook playbooks/k3s.yml \
|
||||
-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
|
||||
|
||||
# prod — the whole chain (extra vars are global, so they reach imported playbooks)
|
||||
ansible-playbook playbooks/site.yml \
|
||||
-e k3s_hosts=k3s_prod \
|
||||
-e k3s_primary_hosts=k3s_prod_primary \
|
||||
-e k3s_secondary_hosts=k3s_prod_secondary
|
||||
```
|
||||
|
||||
What the roles then resolve **by themselves** for a prod host, from `group_vars/k3s_prod/vars.yml`:
|
||||
`k3s_primary_group=k3s_prod_primary` (the `--cluster-init` founder, the node-token `delegate_to`
|
||||
target, and the node Cilium fetches the kubeconfig from), `k3s_servers_group=k3s_prod` (the tls-san
|
||||
node-IP list), `k3s_api_host=192.168.2.10`, `kube_vip_address=192.168.2.9`, and
|
||||
`cluster_kubeconfig_name=prod.kubeconfig` — so a prod run can never overwrite the
|
||||
`.cache/dev.kubeconfig` that `infra/devcontainer/dev.sh` exports as `$KUBECONFIG`.
|
||||
|
||||
### Packages-only baseline (`--tags packages`)
|
||||
`roles/node_baseline` is tagged `packages` / `system` / `disks`. An untagged run is unchanged (all
|
||||
three, in order). `--tags packages` runs **only** the apt task:
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
|
||||
```
|
||||
|
||||
This exists to break a circular dependency: prod's disk layout (`infra/PROD-PROVISIONING.md` §2) is
|
||||
blocked on a SMART read, SMART needs `smartmontools`, `smartmontools` ships in
|
||||
`node_baseline_packages` — but the role's disk tasks need block devices that decision hasn't chosen
|
||||
yet. With `--tags packages` no `disks`-tagged task is selected, so the deliberate
|
||||
`/dev/BLOCKED-pending-disk-decision-…` placeholders in `group_vars/k3s_prod/vars.yml` are never
|
||||
even templated, let alone handed to `mkfs`. A **full** run against a node whose devices are still
|
||||
placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never
|
||||
hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).
|
||||
|
||||
### Still not runnable end to end
|
||||
- The **disk-layout** (§2) and the `node_baseline_*_device` overrides remain BLOCKED placeholders.
|
||||
- `group_vars/k3s_prod/vault.yml` (the become password) does not exist yet, so **no** prod play can
|
||||
escalate — and `--check` against prod fails for the same reason.
|
||||
- `flux-bootstrap.yml` retargets, but `roles/flux_bootstrap` shells out to
|
||||
`bootstrap/flux/bootstrap.sh`, which bootstraps `--path=clusters/dev`. Prod needs
|
||||
`--path=clusters/prod`: a `bootstrap/` change, not a target-group change (§7).
|
||||
- `vpn.egress.gateway.yml` is parameterised (`-e vpn_egress_gateway_hosts=…`) but prod has **no**
|
||||
egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own
|
||||
vaulted PIA credentials.
|
||||
- The operator has ruled the prod cluster stays **uninitialized** until all three nodes are present
|
||||
(§8). `192.168.2.20` is prod's *ingress* Gateway VIP — a Flux/`cluster-vars` concern, not an
|
||||
Ansible one; the kube-vip **control-plane** VIP is `192.168.2.9` (`kube_vip_address`).
|
||||
|
||||
Everything above this section describes the **dev** runbook; the values in it (`192.168.1.x`,
|
||||
`.cache/dev.kubeconfig`, `clusters/dev`) are dev's, supplied by `group_vars`, not baked into the
|
||||
roles.
|
||||
|
||||
## Static validation (PM, in dev container — NOT run here)
|
||||
```
|
||||
|
||||
@@ -4,12 +4,35 @@
|
||||
# cloud-init created this login user (tofu var login_user); it has NOPASSWD sudo.
|
||||
ansible_user: k3s
|
||||
|
||||
# --- environment group NAMES (R-ENV-02: one set of roles, scoped overrides) ---------
|
||||
# The GROUP NAMES a role uses to answer "who is MY environment's primary / MY
|
||||
# environment's servers?". Roles must never hardcode `groups['k3s_primary']`, or they
|
||||
# only ever work against dev. These two are the dev values; group_vars/k3s_prod/vars.yml
|
||||
# overrides them with k3s_prod_primary / k3s_prod, so a host in the prod group resolves
|
||||
# its own primary automatically — exactly the way k3s_api_host below is overridden there.
|
||||
#
|
||||
# These are GROUP NAMES (keys into `groups`), NOT host patterns. The play-level target
|
||||
# pattern is a separate, extra-vars-only knob (`k3s_hosts` / `k3s_primary_hosts` /
|
||||
# `k3s_secondary_hosts`, see playbooks/*.yml) because Ansible templates a play's `hosts:`
|
||||
# before it has any host context and so cannot see group_vars at all.
|
||||
k3s_primary_group: k3s_primary
|
||||
k3s_servers_group: k3s_servers
|
||||
|
||||
# Filename of the kubeconfig the controller-side roles (cilium, cluster_verify,
|
||||
# flux_bootstrap) fetch from the primary into infra/ansible/.cache/. Scoped per
|
||||
# environment so pointing those plays at prod can NEVER overwrite the dev kubeconfig
|
||||
# that infra/devcontainer/dev.sh exports as $KUBECONFIG — an operator's `kubectl get
|
||||
# nodes` silently answering from prod would be a genuinely dangerous mix-up.
|
||||
# group_vars/k3s_prod/vars.yml overrides this to prod.kubeconfig. (.cache/ is gitignored;
|
||||
# a kubeconfig is a credential.)
|
||||
cluster_kubeconfig_name: dev.kubeconfig
|
||||
|
||||
# The kube-apiserver endpoint the whole stack agrees on: the primary's REAL node IP.
|
||||
# NOTE: ansible_host is now the pfSense NAT SSH target (192.168.2.102:225x), NOT the node
|
||||
# IP — so we read the real 192.168.1.x from k3s_node_ip instead. Used for k3s --tls-san /
|
||||
# secondary --server AND for Cilium k8sServiceHost so the Ansible bootstrap and the Flux
|
||||
# HelmRelease stay in PARITY (docs/02 contract).
|
||||
k3s_api_host: "{{ hostvars[groups['k3s_primary'][0]]['k3s_node_ip'] }}"
|
||||
k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['k3s_node_ip'] }}"
|
||||
k3s_api_port: 6443
|
||||
|
||||
# Control-plane HA VIP floated by kube-vip (roles/kube_vip). A stable failover
|
||||
|
||||
@@ -54,9 +54,15 @@ behind them instead of NOPASSWD.
|
||||
- This is the Ansible-bootstrap-layer secret (host credential), distinct from the platform's
|
||||
in-cluster **SOPS + age** secrets (ADR-0007) — different layer, different tool, same convention
|
||||
as `hypervisor`/`proxmox_hosts`.
|
||||
- **Targeting prod is solved** (2026-08-31): plays take `-e k3s_hosts=k3s_prod` (and, for `k3s.yml`,
|
||||
`-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`), while the
|
||||
roles resolve "my environment's primary/servers" from `k3s_primary_group` / `k3s_servers_group`
|
||||
in `vars.yml` here — no role fork, no per-run flag for that half. See
|
||||
`infra/ansible/README.md` → "Targeting prod".
|
||||
- **Do not run any prod playbook yet.** Phase 7 is prep-only: the vault above just makes the plays
|
||||
*able* to authenticate once the §7 repo blockers close and the disk-layout (§2) and
|
||||
control-plane-VIP (§4) decisions in `infra/PROD-PROVISIONING.md` are made — and, separately, once
|
||||
the group-name gap flagged in `inventory/hosts.ini`'s prod section is resolved (the existing
|
||||
playbooks target `k3s_servers` / `k3s_primary` / `k3s_secondary` literally, not `k3s_prod_*`).
|
||||
See that file for the full picture.
|
||||
*able* to authenticate once the §7 repo blockers close and the disk-layout (§2) decision in
|
||||
`infra/PROD-PROVISIONING.md` is made. The ONE exception the operator has sanctioned is the
|
||||
packages-only baseline that unblocks the §2 drive-health read —
|
||||
`ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages` — which
|
||||
cannot touch the BLOCKED device placeholders because every disk task is `disks`-tagged and so is
|
||||
not selected. It still needs `vault.yml` (above) to exist first.
|
||||
@@ -29,13 +29,34 @@ ansible_become: true
|
||||
# The password itself is ansible_become_password — ansible-vault-encrypted, in
|
||||
# vault.yml (this directory), never here in plaintext. See README.md to create it.
|
||||
|
||||
# --- environment group NAMES (R-ENV-02) -------------------------------------------
|
||||
# group_vars/all.yml defines these as the DEV group names (k3s_primary / k3s_servers);
|
||||
# here they name PROD's groups. Roles index `groups[k3s_primary_group]` /
|
||||
# `groups[k3s_servers_group]` instead of a literal, so roles/k3s_server and roles/cilium
|
||||
# resolve "my environment's primary" and "my environment's servers" correctly for prod
|
||||
# hosts with NO role fork and no per-run flag — the reuse R-ENV-02 promises.
|
||||
#
|
||||
# NOTE these are GROUP NAMES, not the play-level target pattern. Selecting which hosts a
|
||||
# play RUNS against is a separate extra-vars knob (`-e k3s_hosts=k3s_prod`, and for
|
||||
# k3s.yml `-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`),
|
||||
# because Ansible templates `hosts:` before any host — and therefore any group_vars —
|
||||
# exists. See infra/ansible/README.md "Targeting prod".
|
||||
k3s_primary_group: k3s_prod_primary
|
||||
k3s_servers_group: k3s_prod
|
||||
|
||||
# Keep the fetched prod kubeconfig in its OWN cache file. group_vars/all.yml defaults this
|
||||
# to dev.kubeconfig, which is the file infra/devcontainer/dev.sh exports as $KUBECONFIG —
|
||||
# a prod play must never clobber it (a `kubectl` that silently answers from prod is worse
|
||||
# than no kubeconfig at all). .cache/ is gitignored; a kubeconfig is a credential.
|
||||
cluster_kubeconfig_name: prod.kubeconfig
|
||||
|
||||
# --- k3s API host — safe to set now, the primary's real IP is already known -------
|
||||
# Dev's k3s_api_host (group_vars/all.yml) reads a separate k3s_node_ip hostvar because
|
||||
# dev's ansible_host is the pfSense NAT SSH target, not the node's real address. Prod
|
||||
# has NO NAT hop — ansible_host in hosts.ini IS already the node's real, routable LAN
|
||||
# address (see the KEY DIFFERENCE note there) — so point straight at it instead of
|
||||
# needing a k3s_node_ip split.
|
||||
k3s_api_host: "{{ hostvars[groups['k3s_prod_primary'][0]]['ansible_host'] }}"
|
||||
k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}"
|
||||
# cluster_controller_tls_server_name (group_vars/all.yml) derives from kube_vip_address
|
||||
# by template reference, so it needs no separate override here — it picks up the real
|
||||
# value set below automatically for prod hosts too.
|
||||
|
||||
@@ -66,19 +66,21 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
|
||||
# `ansible_port` is the node's own real sshd (22). See group_vars/k3s_prod/vars.yml for
|
||||
# the k3s_api_host override that reflects this directly-reachable topology.
|
||||
#
|
||||
# GAP flagged for the PM, NOT fixed in this pass (inventory/group_vars only): the
|
||||
# k3s_server role hardcodes the LITERAL group names `k3s_primary` / `k3s_secondary` /
|
||||
# `k3s_servers` in roles/k3s_server/defaults/main.yml (tls-san SAN list) and
|
||||
# roles/k3s_server/tasks/main.yml (node-token delegate_to), and every node-host
|
||||
# playbook (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, verify.yml,
|
||||
# flux-bootstrap.yml) declares `hosts: k3s_servers` / `k3s_primary` / `k3s_secondary`
|
||||
# literally. Naming prod's groups k3s_prod_primary/k3s_prod_secondary — as this
|
||||
# checklist and PROD-PROVISIONING.md §6 both call for — means those existing
|
||||
# playbooks will NOT select prod hosts as they stand today. That's harmless right now
|
||||
# (nothing can accidentally run against prod), but it means actually reusing the dev
|
||||
# playbooks against prod later (the R-ENV-02 promise) needs either parametrizing the
|
||||
# group name they target or some other mechanism — not yet designed, tracked here so
|
||||
# it isn't lost before §6 is attempted for real.
|
||||
# GROUP-NAME GAP: CLOSED 2026-08-31. The roles and playbooks used to hardcode the
|
||||
# LITERAL dev groups (`hosts: k3s_servers` / `k3s_primary` / `k3s_secondary`, and
|
||||
# `groups['k3s_primary']` inside roles/k3s_server + roles/cilium), so nothing could
|
||||
# select the k3s_prod_* groups below. Both halves are now variables, deliberately kept
|
||||
# separate because Ansible resolves them at different times:
|
||||
# * WHICH HOSTS A PLAY RUNS ON — `k3s_hosts` (whole-cluster plays), `k3s_primary_hosts`
|
||||
# / `k3s_secondary_hosts` (k3s.yml's split), each defaulting to the dev group.
|
||||
# EXTRA VARS ONLY: a play's `hosts:` is templated before any host exists, so
|
||||
# group_vars cannot be read there. Prod: `-e k3s_hosts=k3s_prod` etc.
|
||||
# * WHICH GROUP A ROLE MEANS BY "MY PRIMARY"/"MY SERVERS" — `k3s_primary_group` /
|
||||
# `k3s_servers_group`, set in group_vars/all.yml (dev) and group_vars/k3s_prod/vars.yml
|
||||
# (prod), so roles resolve their own environment with NO per-run flag — the same
|
||||
# mechanism k3s_api_host already used.
|
||||
# Running with no extra vars is byte-for-byte the dev behaviour it always had. See
|
||||
# infra/ansible/README.md "Targeting prod" and PROD-PROVISIONING.md §6.
|
||||
# =============================================================================
|
||||
|
||||
[k3s_prod_primary]
|
||||
|
||||
@@ -1,8 +1,13 @@
|
||||
---
|
||||
# Install Cilium as the CNI (ADR-0002). The cluster is NotReady until this runs.
|
||||
# Executed from the controller against a fetched kubeconfig (role handles delegation).
|
||||
#
|
||||
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
|
||||
# ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod
|
||||
# The role's "fetch the kubeconfig from the primary" delegation follows the
|
||||
# k3s_primary_group group_var, so it picks the prod primary automatically.
|
||||
- name: Install Cilium CNI
|
||||
hosts: k3s_servers
|
||||
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
|
||||
become: true
|
||||
gather_facts: false
|
||||
roles:
|
||||
|
||||
@@ -1,8 +1,15 @@
|
||||
---
|
||||
# FINAL step: hand off to Flux + SOPS (T-1.4). Calls into bootstrap/ (authored by
|
||||
# the Security + GitOps Engineers) — this play does not author that subtree.
|
||||
#
|
||||
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
|
||||
# ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod
|
||||
# NOTE the Flux PATH is a separate concern: roles/flux_bootstrap shells out to
|
||||
# bootstrap/flux/bootstrap.sh, which bootstraps --path=clusters/dev. Prod needs
|
||||
# --path=clusters/prod, which is a bootstrap/ change, NOT a target-group change
|
||||
# (infra/PROD-PROVISIONING.md §7 tracks it).
|
||||
- name: Flux + SOPS bootstrap hand-off
|
||||
hosts: k3s_servers
|
||||
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
|
||||
become: false
|
||||
gather_facts: false
|
||||
roles:
|
||||
|
||||
@@ -2,9 +2,20 @@
|
||||
# Install the HA k3s control plane (embedded etcd, 3 servers) — ADR-0001.
|
||||
# Primary founds the cluster (--cluster-init); secondaries join it in a second
|
||||
# play so the primary's node-token fact is available to them.
|
||||
#
|
||||
# TARGET (R-ENV-02): this is the one playbook that needs TWO patterns, because the
|
||||
# primary/secondary split is the whole point of the file. Both default to dev:
|
||||
# ansible-playbook playbooks/k3s.yml # dev, unchanged
|
||||
# ansible-playbook playbooks/k3s.yml \
|
||||
# -e k3s_primary_hosts=k3s_prod_primary \
|
||||
# -e k3s_secondary_hosts=k3s_prod_secondary # prod
|
||||
# The role's OWN "who is the primary" lookups (join token delegate_to, tls-san list)
|
||||
# do NOT use these — they follow the k3s_primary_group / k3s_servers_group group_vars,
|
||||
# which prod sets once in group_vars/k3s_prod/vars.yml. These two vars only choose which
|
||||
# hosts the plays RUN on, which Ansible resolves before any group_vars exist.
|
||||
|
||||
- name: k3s primary (cluster-init)
|
||||
hosts: k3s_primary
|
||||
hosts: "{{ k3s_primary_hosts | default('k3s_primary') }}"
|
||||
become: true
|
||||
gather_facts: true
|
||||
roles:
|
||||
@@ -16,7 +27,7 @@
|
||||
# an updated tls-san set) triggers the Restart k3s handler: restarting both
|
||||
# secondaries at once would drop etcd below quorum. One-at-a-time keeps 2/3 up.
|
||||
- name: k3s secondaries (join)
|
||||
hosts: k3s_secondary
|
||||
hosts: "{{ k3s_secondary_hosts | default('k3s_secondary') }}"
|
||||
become: true
|
||||
serial: 1
|
||||
gather_facts: true
|
||||
|
||||
@@ -1,9 +1,14 @@
|
||||
---
|
||||
# Deploy kube-vip as the control-plane HA VIP (192.168.1.49) for the kube-API.
|
||||
# Deploy kube-vip as the control-plane HA VIP (192.168.1.49 on dev) for the kube-API.
|
||||
# Control-plane ONLY (svc_enable=false) — Cilium owns Service LoadBalancing.
|
||||
# Writes the manifest into each server's k3s auto-deploy dir; k3s applies it.
|
||||
#
|
||||
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
|
||||
# ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
|
||||
# (the VIP itself comes from kube_vip_address, already scoped per environment in
|
||||
# group_vars — dev 192.168.1.49, prod 192.168.2.9).
|
||||
- name: Deploy kube-vip control-plane VIP
|
||||
hosts: k3s_servers
|
||||
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
|
||||
become: true
|
||||
gather_facts: true
|
||||
roles:
|
||||
|
||||
@@ -2,8 +2,21 @@
|
||||
# Node baseline — first-boot floor for k3s/Cilium/Longhorn, moved from cloud-init into
|
||||
# Ansible so Proxmox provisioning stays API-only (no SSH/SFTP to the hypervisor). Runs
|
||||
# FIRST in site.yml, before k3s. Idempotent; safe to re-run.
|
||||
#
|
||||
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
|
||||
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
|
||||
# See README.md "Targeting prod". The pattern is the one node-health.yml established.
|
||||
#
|
||||
# TAGS — this role is the only way `smartmontools` (and the rest of the package floor)
|
||||
# reaches a node, but its data-disk tasks need block devices that prod has NOT decided
|
||||
# yet (infra/PROD-PROVISIONING.md §2). Run JUST the package floor with:
|
||||
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
|
||||
# `--tags packages` runs the apt task ONLY; every disk task carries the `disks` tag and
|
||||
# is therefore skipped, so the BLOCKED placeholder devices can never be touched. A full
|
||||
# (untagged) run against a node whose device vars are still placeholders fails fast on an
|
||||
# explicit assert instead of handing a bogus path to mkfs. See roles/node_baseline.
|
||||
- name: Baseline the k3s nodes (packages, kernel modules, Longhorn disk, swap-off)
|
||||
hosts: k3s_servers
|
||||
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
|
||||
become: true
|
||||
gather_facts: true
|
||||
roles:
|
||||
|
||||
@@ -6,13 +6,15 @@
|
||||
# ./dev.sh run bash -c 'cd /work/infra/ansible && \
|
||||
# ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod'
|
||||
#
|
||||
# TARGET GROUP IS A VARIABLE, unlike the other playbooks here. Every existing node
|
||||
# playbook (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, verify.yml) hardcodes
|
||||
# `hosts: k3s_servers` / `k3s_primary` / `k3s_secondary`, which is exactly why they cannot
|
||||
# select the prod groups today (the gap flagged in inventory/hosts.ini's prod section, and
|
||||
# the R-ENV-02 reuse promise it threatens). A health check is the one play you want to be
|
||||
# able to point anywhere, so it takes the pattern as a var and defaults to dev. Run it
|
||||
# against prod with `-e node_health_hosts=k3s_prod`, or a single box with `--limit`.
|
||||
# TARGET GROUP IS A VARIABLE. This play established that pattern; the rest of the node
|
||||
# playbooks (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, verify.yml,
|
||||
# flux-bootstrap.yml) now follow it and take `k3s_hosts` / `k3s_primary_hosts` /
|
||||
# `k3s_secondary_hosts` instead of the literal dev groups they used to hardcode — see
|
||||
# infra/ansible/README.md "Targeting prod". This one keeps its OWN var name because it is
|
||||
# a standalone diagnostic you point anywhere, not a step in the bring-up chain: it must
|
||||
# stay independently targetable (`-e node_health_hosts=<group>`) even while a bring-up run
|
||||
# is scoped elsewhere. Run it against prod with `-e node_health_hosts=k3s_prod`, or a
|
||||
# single box with `--limit`.
|
||||
#
|
||||
# BECOME: left at the play level exactly as the inventory sets it. The role itself pins
|
||||
# `ansible_become: false` on every unprivileged task and probes escalation once before
|
||||
|
||||
@@ -2,6 +2,18 @@
|
||||
# Full dev/test cluster bring-up, in order (docs/02 sequencing; run AFTER
|
||||
# `tofu apply` has created and cloud-init has baselined the VMs):
|
||||
# k3s HA install -> Cilium CNI -> verify readiness -> Flux + SOPS hand-off.
|
||||
#
|
||||
# TARGET (R-ENV-02): every imported play takes its host pattern from a var that
|
||||
# defaults to the dev group, so `ansible-playbook playbooks/site.yml` with no extra
|
||||
# vars is byte-for-byte the dev run it always was. Extra vars are global and reach
|
||||
# imported playbooks, so the whole chain retargets with three flags:
|
||||
# ansible-playbook playbooks/site.yml \
|
||||
# -e k3s_hosts=k3s_prod \
|
||||
# -e k3s_primary_hosts=k3s_prod_primary \
|
||||
# -e k3s_secondary_hosts=k3s_prod_secondary
|
||||
# DO NOT run that yet — prod is blocked on the disk-layout decision
|
||||
# (infra/PROD-PROVISIONING.md §2) and the §7 repo blockers, and the operator has ruled
|
||||
# the prod cluster stays uninitialized until all three nodes are present (§8).
|
||||
- name: Import - node baseline (packages, kernel modules, Longhorn disk, swap-off)
|
||||
ansible.builtin.import_playbook: node-baseline.yml
|
||||
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
---
|
||||
# Verify Phase-1 exit gate #1: 3 nodes Ready, Cilium/Hubble healthy, no kube-proxy.
|
||||
#
|
||||
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
|
||||
# ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod
|
||||
- name: Verify cluster readiness
|
||||
hosts: k3s_servers
|
||||
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
|
||||
become: false
|
||||
gather_facts: false
|
||||
roles:
|
||||
|
||||
@@ -3,15 +3,23 @@
|
||||
# (ADR-0015). Host infra — the split-tunnel OpenVPN service + policy routing + scoped
|
||||
# nftables kill-switch. The CiliumEgressGatewayPolicy that steers vpn-class pods to this
|
||||
# node's tun0 is authored separately as GitOps (platform/), against the same contract:
|
||||
# * inventory group `vpn_egress_gateway` (this play's target)
|
||||
# * inventory group `vpn_egress_gateway` (this play's default target)
|
||||
# * k8s node label homelab.keithjackson.dev/egress-gateway=true (set by the role)
|
||||
# * host tunnel iface tun0 (split-tunnel; never the node default route)
|
||||
#
|
||||
# Secrets come from Ansible vault (inventory/group_vars/vpn_egress_gateway/vault.yml):
|
||||
# vault_vpn_egress_gateway_pia_username, vault_vpn_egress_gateway_pia_password, vault_vpn_egress_gateway_pia_ovpn
|
||||
# Run: ansible-playbook playbooks/vpn.egress.gateway.yml
|
||||
#
|
||||
# TARGET (R-ENV-02): the group is a var, defaulting to dev's `vpn_egress_gateway`. Prod
|
||||
# has NOT decided whether it runs an egress gateway at all, nor on which node
|
||||
# (infra/PROD-PROVISIONING.md §6, last box) — so there is deliberately no prod egress
|
||||
# group in the inventory yet. When there is, point this at it with
|
||||
# -e vpn_egress_gateway_hosts=<that group>
|
||||
# and it needs its OWN vaulted PIA credentials group_vars; this play's vault is scoped to
|
||||
# the dev group.
|
||||
- name: Configure the node-level VPN egress gateway
|
||||
hosts: vpn_egress_gateway
|
||||
hosts: "{{ vpn_egress_gateway_hosts | default('vpn_egress_gateway') }}"
|
||||
become: true
|
||||
gather_facts: true # package module + service facts need os facts
|
||||
roles:
|
||||
|
||||
@@ -19,8 +19,10 @@ gateway_api_crd_urls:
|
||||
# TLSRoute (experimental) is required by Cilium's Gateway API support.
|
||||
- "https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/{{ gateway_api_version }}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml"
|
||||
|
||||
# Local kubeconfig fetched from the primary. *.kubeconfig is gitignored.
|
||||
cilium_kubeconfig: "{{ playbook_dir }}/../.cache/dev.kubeconfig"
|
||||
# Local kubeconfig fetched from the primary. *.kubeconfig is gitignored. The FILENAME
|
||||
# comes from cluster_kubeconfig_name (group_vars) so a prod run cannot overwrite the dev
|
||||
# kubeconfig the dev container exports as $KUBECONFIG. R-ENV-02.
|
||||
cilium_kubeconfig: "{{ playbook_dir }}/../.cache/{{ cluster_kubeconfig_name | default('dev.kubeconfig') }}"
|
||||
cilium_values_path: "{{ playbook_dir }}/../.cache/cilium-values.yaml"
|
||||
|
||||
# dev: single operator replica to save RAM (PHASE1-SCOPE RAM ceiling). NOT a
|
||||
|
||||
@@ -4,11 +4,14 @@
|
||||
# The cluster is NotReady until this completes — expected (PHASE1-SCOPE risks).
|
||||
# run_once: this whole role acts on the cluster once, not per host.
|
||||
|
||||
# k3s_primary_group (group_vars) names THIS environment's primary group — `k3s_primary`
|
||||
# on dev, `k3s_prod_primary` on prod — so the role fetches the right cluster's kubeconfig
|
||||
# without being forked per environment (R-ENV-02).
|
||||
- name: Fetch the k3s kubeconfig from the primary
|
||||
ansible.builtin.slurp:
|
||||
src: /etc/rancher/k3s/k3s.yaml
|
||||
register: k3s_kubeconfig_raw
|
||||
delegate_to: "{{ groups['k3s_primary'][0] }}"
|
||||
delegate_to: "{{ groups[k3s_primary_group][0] }}"
|
||||
run_once: true
|
||||
|
||||
- name: Ensure the local cache dir exists
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
---
|
||||
# cluster_verify role defaults.
|
||||
cluster_verify_kubeconfig: "{{ playbook_dir }}/../.cache/dev.kubeconfig"
|
||||
# Filename comes from cluster_kubeconfig_name (group_vars) so dev and prod never share
|
||||
# one cache file — dev.kubeconfig for dev, prod.kubeconfig for prod. R-ENV-02.
|
||||
cluster_verify_kubeconfig: "{{ playbook_dir }}/../.cache/{{ cluster_kubeconfig_name | default('dev.kubeconfig') }}"
|
||||
cluster_verify_node_ready_timeout: "300s"
|
||||
@@ -7,7 +7,8 @@
|
||||
# args below are ASSUMPTIONS the PM must reconcile with those deliverables — see
|
||||
# roles/flux_bootstrap/tasks/main.yml and infra/ansible/README.md "Interface".
|
||||
|
||||
flux_bootstrap_kubeconfig: "{{ playbook_dir }}/../.cache/dev.kubeconfig"
|
||||
# Filename from cluster_kubeconfig_name (group_vars): dev.kubeconfig / prod.kubeconfig.
|
||||
flux_bootstrap_kubeconfig: "{{ playbook_dir }}/../.cache/{{ cluster_kubeconfig_name | default('dev.kubeconfig') }}"
|
||||
|
||||
# Repo root, relative to playbooks/ (infra/ansible/playbooks -> ../../..).
|
||||
flux_bootstrap_repo_root: "{{ playbook_dir }}/../../.."
|
||||
|
||||
@@ -26,7 +26,11 @@ k3s_server_mode: "secondary"
|
||||
# the VIP), and 127.0.0.1 (local kubeconfig / health probes). k3s regenerates the
|
||||
# serving cert on restart when this set changes. Deduplicated (the primary IP is
|
||||
# also k3s_api_host).
|
||||
#
|
||||
# k3s_servers_group — NOT a literal group name — is what makes this role environment-
|
||||
# agnostic (R-ENV-02): group_vars/all.yml sets it to `k3s_servers` (dev) and
|
||||
# group_vars/k3s_prod/vars.yml to `k3s_prod`, so each environment SANs its own servers.
|
||||
k3s_tls_sans: >-
|
||||
{{ ([k3s_api_host, kube_vip_address, '127.0.0.1']
|
||||
+ (groups['k3s_servers'] | map('extract', hostvars, 'k3s_node_ip') | list))
|
||||
+ (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list))
|
||||
| unique }}
|
||||
@@ -121,19 +121,23 @@
|
||||
# contained — delegate_to still reaches the primary even when it isn't a play
|
||||
# target, so the join no longer depends on a fact an earlier play had to set.
|
||||
# T-1.5 finding F2 (see DECISIONS-LOG.md).
|
||||
#
|
||||
# "The primary" is resolved through k3s_primary_group (group_vars: `k3s_primary` for
|
||||
# dev, `k3s_prod_primary` for prod), never a literal group name — so a prod secondary
|
||||
# delegates to the PROD primary using this same role, unchanged (R-ENV-02).
|
||||
- name: Fetch the k3s node-token directly from the primary (secondary)
|
||||
ansible.builtin.slurp:
|
||||
src: /var/lib/rancher/k3s/server/token
|
||||
delegate_to: "{{ groups['k3s_primary'][0] }}"
|
||||
delegate_to: "{{ groups[k3s_primary_group][0] }}"
|
||||
register: k3s_primary_token_raw
|
||||
when:
|
||||
- k3s_server_mode == 'secondary'
|
||||
- hostvars[groups['k3s_primary'][0]].k3s_node_token is not defined
|
||||
- hostvars[groups[k3s_primary_group][0]].k3s_node_token is not defined
|
||||
|
||||
- name: Resolve the join token (secondary)
|
||||
ansible.builtin.set_fact:
|
||||
k3s_join_token: >-
|
||||
{{ hostvars[groups['k3s_primary'][0]].k3s_node_token
|
||||
{{ hostvars[groups[k3s_primary_group][0]].k3s_node_token
|
||||
| default(k3s_primary_token_raw.content | b64decode | trim, true) }}
|
||||
when: k3s_server_mode == 'secondary'
|
||||
|
||||
|
||||
@@ -1,10 +1,23 @@
|
||||
---
|
||||
# TAGS — every task here carries one of `packages`, `system` or `disks`. An untagged run
|
||||
# is unchanged (all three run, in this order). The split exists because the PACKAGE floor
|
||||
# and the DATA-DISK work have different readiness: packages are safe on any node from
|
||||
# first contact, while the disk tasks need block devices that a new environment may not
|
||||
# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
|
||||
# on a SMART read that needs `smartmontools` — which ships in node_baseline_packages).
|
||||
# `--tags packages` breaks that circle: it installs the package floor and NOTHING else.
|
||||
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
|
||||
# Skipping is the guard: with `--tags packages` no `disks`-tagged task is selected, so a
|
||||
# BLOCKED-pending-disk-decision device path is never even templated, let alone handed to
|
||||
# mkfs. A FULL run against such a node stops at the assert below instead.
|
||||
|
||||
- name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
|
||||
ansible.builtin.apt:
|
||||
name: "{{ node_baseline_packages }}"
|
||||
state: present
|
||||
update_cache: true
|
||||
cache_valid_time: 3600
|
||||
tags: [packages]
|
||||
|
||||
- name: Persist kernel modules for k3s/Cilium/Longhorn
|
||||
ansible.builtin.copy:
|
||||
@@ -13,12 +26,14 @@
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: "{{ node_baseline_kernel_modules | join('\n') }}\n"
|
||||
tags: [system]
|
||||
|
||||
- name: Load kernel modules now
|
||||
community.general.modprobe:
|
||||
name: "{{ item }}"
|
||||
state: present
|
||||
loop: "{{ node_baseline_kernel_modules }}"
|
||||
tags: [system]
|
||||
|
||||
- name: Install sysctl tuning for k3s/Cilium
|
||||
ansible.builtin.copy:
|
||||
@@ -34,12 +49,14 @@
|
||||
fs.inotify.max_user_instances = 8192
|
||||
fs.inotify.max_user_watches = 524288
|
||||
notify: Reload sysctl
|
||||
tags: [system]
|
||||
|
||||
- name: Enable and start iscsid (Longhorn prerequisite)
|
||||
ansible.builtin.systemd:
|
||||
name: iscsid
|
||||
enabled: true
|
||||
state: started
|
||||
tags: [system]
|
||||
|
||||
# qemu-guest-agent binds to the virtio-serial channel /dev/virtio-ports/org.qemu.guest_agent.0,
|
||||
# which Proxmox attaches only when the VM's `agent` option is enabled. Our tofu keeps agent
|
||||
@@ -51,18 +68,44 @@
|
||||
ansible.builtin.stat:
|
||||
path: /dev/virtio-ports/org.qemu.guest_agent.0
|
||||
register: node_baseline_qga_channel
|
||||
tags: [system]
|
||||
|
||||
- name: Enable qemu-guest-agent (start only when its virtio channel is present)
|
||||
ansible.builtin.systemd:
|
||||
name: qemu-guest-agent
|
||||
enabled: true
|
||||
state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
|
||||
tags: [system]
|
||||
|
||||
# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
|
||||
# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
|
||||
# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
|
||||
# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
|
||||
# which do not exist on bare metal either — a confusing failure that looks like a wiring
|
||||
# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
|
||||
# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
|
||||
- name: Assert this environment's data-disk devices have actually been decided
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- "'BLOCKED' not in node_baseline_longhorn_device"
|
||||
- "'BLOCKED' not in node_baseline_etcd_device"
|
||||
fail_msg: >-
|
||||
node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
|
||||
placeholders for this host's environment
|
||||
(longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
|
||||
The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
|
||||
and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
|
||||
the SMART read that decision waits on can be taken) re-run this play with
|
||||
`--tags packages`, which skips every disk task including this assert.
|
||||
quiet: true
|
||||
tags: [disks]
|
||||
|
||||
- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
|
||||
community.general.filesystem:
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_longhorn_device }}"
|
||||
opts: "-L {{ node_baseline_longhorn_fs_label }}"
|
||||
tags: [disks]
|
||||
|
||||
- name: Mount the Longhorn data disk
|
||||
ansible.posix.mount:
|
||||
@@ -71,6 +114,7 @@
|
||||
fstype: ext4
|
||||
opts: "defaults,nofail,x-systemd.device-timeout=5s"
|
||||
state: mounted
|
||||
tags: [disks]
|
||||
|
||||
# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
|
||||
# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
|
||||
@@ -84,6 +128,7 @@
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_etcd_device }}"
|
||||
opts: "-L {{ node_baseline_etcd_fs_label }}"
|
||||
tags: [disks]
|
||||
|
||||
- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
|
||||
ansible.posix.mount:
|
||||
@@ -96,13 +141,16 @@
|
||||
dump: "0"
|
||||
passno: "2"
|
||||
state: mounted
|
||||
tags: [disks]
|
||||
|
||||
- name: Disable swap now
|
||||
ansible.builtin.command: swapoff -a
|
||||
changed_when: false
|
||||
tags: [system]
|
||||
|
||||
- name: Neutralize swap entries in fstab (kubelet requires swap off)
|
||||
ansible.builtin.replace:
|
||||
path: /etc/fstab
|
||||
regexp: '^([^#].*\sswap\s.*)$'
|
||||
replace: '# \1 # disabled for k3s/kubelet'
|
||||
tags: [system]
|
||||
Reference in new issue
Block a user