feat(prod): kernel fix + disk layout — prod nodes ready for k3s
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
8e913fdd7b
commit
05baae90ec
16 files changed
+1782
-126
No files matched your search
@@ -0,0 +1,642 @@
|
||||
---
|
||||
# =====================================================================================
|
||||
# node_baseline — DATA DISKS. Every task here carries the `disks` tag (applied once by
|
||||
# the `include_tasks` in main.yml), so `--tags packages` still skips the whole file.
|
||||
# Split into its own file rather than living in main.yml because it is the one part of
|
||||
# the role that DESTRUCTIVELY claims storage, and it deserves to be read on its own.
|
||||
#
|
||||
# TWO ENVIRONMENTS, TWO SHAPES — and the shape is DECLARED, never inferred:
|
||||
#
|
||||
# dev (group_vars/all.yml, role defaults)
|
||||
# node_baseline_longhorn_source: device -> mkfs /dev/vdb, mount LABEL=longhorn
|
||||
# at /var/lib/longhorn (== dataPath)
|
||||
# node_baseline_etcd_source: device -> mkfs /dev/vdc, mount LABEL=k3s-etcd
|
||||
# k3s_data_dir: (k3s default, /var/lib/rancher/k3s)
|
||||
# THREE devices: OS, Longhorn, etcd. Two tofu-provisioned virtio disks exist for
|
||||
# exactly this purpose. NOTHING about dev changes in the 2026-09-04 revision.
|
||||
#
|
||||
# prod (group_vars/k3s_prod/vars.yml — operator ruling, PROD-PROVISIONING.md §2)
|
||||
# node_baseline_longhorn_source: claim-mountpoint (claim mountpoint /srv)
|
||||
# node_baseline_data_mount: /data (LABEL=k3s-data)
|
||||
# node_baseline_etcd_source: shared-data-partition
|
||||
# k3s_data_dir: /data/k3s
|
||||
# ONE physical disk per node, already fully partitioned by the Debian installer. The
|
||||
# ~424 GB `/srv` partition is claimed and mounted at a NEUTRAL mountpoint, then split
|
||||
# between TWO tenants: /data/longhorn (Longhorn replicas) + /data/k3s (the whole k3s
|
||||
# data dir: etcd AND the containerd image store).
|
||||
#
|
||||
# WHY THE PARTITION IS NO LONGER LONGHORN-ONLY (revised 2026-09-04, PROD-PROVISIONING.md
|
||||
# §2). Prod's `/var` is a separate partition of just 11.8 GiB. k3s's default data dir puts
|
||||
# BOTH etcd's WAL and the containerd image store on it. Measured on the fully-deployed dev
|
||||
# cluster, containerd alone is 21-25 GB per node and /var totals 58-70 GB — so prod would
|
||||
# have filled /var during its FIRST image pulls, and a full /var means etcd cannot fsync its
|
||||
# WAL. That is cluster death, not degradation. Moving k3s's data dir onto the 424 GB
|
||||
# partition is the fix; the partition therefore serves two tenants, and neither its
|
||||
# mountpoint nor its filesystem label may name only one of them.
|
||||
#
|
||||
# WHY /data AND LABEL k3s-data.
|
||||
# * `/data` — short, neutral, tenant-agnostic, and NOT under /var (the whole point).
|
||||
# Deliberately NOT `/srv`: /srv is the CLAIM mountpoint this role discovers the device
|
||||
# at, and the idempotency contract is that /srv is gone from /proc/mounts and
|
||||
# /etc/fstab once the claim has happened. Re-using it as the destination would make
|
||||
# "claimed" and "not yet claimed" indistinguishable.
|
||||
# * label `k3s-data`, deliberately NOT the bare `data`: guard #4 in the assert wall below
|
||||
# accepts a partition whose label is empty OR ALREADY OURS, and its own comment names
|
||||
# `data` as an example of somebody else's named volume. Labelling ours `data` would make
|
||||
# a stranger's `data` volume look already-claimed and weaken that guard. `k3s-data` is
|
||||
# cluster-scoped rather than tenant-scoped and matches the existing `k3s-etcd` label
|
||||
# convention in this repo.
|
||||
#
|
||||
# WHY A "SOURCE" ENUM INSTEAD OF A BLANK VARIABLE. "etcd has no dedicated disk" and
|
||||
# "somebody forgot to set node_baseline_etcd_device" look identical if the signal is an
|
||||
# empty string, and the second one must never silently succeed. `root-filesystem` and
|
||||
# `shared-data-partition` are explicit, greppable statements of intent that the assert
|
||||
# below accepts; an unset or placeholder device is still a hard failure. The two differ in
|
||||
# a way that matters: `root-filesystem` says etcd is on the OS filesystem, and
|
||||
# `shared-data-partition` says it is on the claimed data partition. Prod was the FORMER
|
||||
# until 2026-09-04 and is now the LATTER — and the role ASSERTS which one is true rather
|
||||
# than printing a claim it never checked (see the etcd section).
|
||||
#
|
||||
# WHY DISCOVERY INSTEAD OF A PER-HOST DEVICE PATH. The prod partition is /dev/sda5 on
|
||||
# nodes 1 and 3 (SATA) but /dev/nvme0n1p5 on node 2 (NVMe), so no single group value is
|
||||
# correct — and per-host `host_vars` would encode a *guess about hardware* into inventory
|
||||
# that goes stale the moment a node is re-imaged, a disk is replaced, or a fourth node
|
||||
# arrives with a different controller. Worse, a fleet-wide `/dev/sda5` would eventually
|
||||
# point at the wrong partition on some future node and mkfs it. So the role asks the NODE
|
||||
# which block device is mounted at the claim mountpoint, and then proves that answer is
|
||||
# safe (see the assert wall below) before touching it. The inventory declares INTENT
|
||||
# ("the partition currently mounted at /srv"); the node supplies the FACT.
|
||||
# =====================================================================================
|
||||
|
||||
# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet declared
|
||||
# its data-disk layout. Two failure shapes are caught:
|
||||
# * an undeclared/typo'd source enum, and
|
||||
# * source: device with a device path that is a PLACEHOLDER rather than a real device.
|
||||
# group_vars/k3s_prod/vars.yml deliberately carries `/dev/NOT-A-DEVICE-…` for both device
|
||||
# vars now that prod uses neither: if anyone ever flips a source back to `device` without
|
||||
# also supplying a real path, they get this assert instead of dev's /dev/vdb (which does
|
||||
# not exist on bare metal — a confusing failure that looks like a wiring bug). `BLOCKED`
|
||||
# stays in the token list because that was the pre-decision placeholder and may still be
|
||||
# carried by other environments/branches.
|
||||
- name: Assert this environment's data-disk layout has actually been declared
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- node_baseline_longhorn_source in ['device', 'claim-mountpoint']
|
||||
- node_baseline_etcd_source in ['device', 'root-filesystem', 'shared-data-partition']
|
||||
# `shared-data-partition` is only coherent when there IS a claimed data partition to
|
||||
# share, and only when the k3s data dir actually sits INSIDE it. Catch the
|
||||
# contradiction here, in inventory terms, rather than three sections later when etcd
|
||||
# has silently landed back on the OS filesystem.
|
||||
- >-
|
||||
node_baseline_etcd_source != 'shared-data-partition'
|
||||
or node_baseline_longhorn_source == 'claim-mountpoint'
|
||||
- >-
|
||||
node_baseline_etcd_source != 'shared-data-partition'
|
||||
or node_baseline_k3s_data_dir.startswith(node_baseline_data_mount ~ '/')
|
||||
# The transition knobs are a matched PAIR: a retired label with no retired mountpoint
|
||||
# (or vice versa) cannot describe a state to migrate from.
|
||||
- >-
|
||||
(node_baseline_data_retired_fs_label | length > 0)
|
||||
== (node_baseline_data_retired_mount | length > 0)
|
||||
- >-
|
||||
node_baseline_longhorn_source != 'device'
|
||||
or not (node_baseline_longhorn_device is search('BLOCKED|NOT-A-DEVICE'))
|
||||
- >-
|
||||
node_baseline_etcd_source != 'device'
|
||||
or not (node_baseline_etcd_device is search('BLOCKED|NOT-A-DEVICE'))
|
||||
- >-
|
||||
node_baseline_longhorn_source != 'claim-mountpoint'
|
||||
or (node_baseline_longhorn_claim_mountpoint | length) > 0
|
||||
fail_msg: >-
|
||||
This host's data-disk layout is not declared.
|
||||
longhorn_source={{ node_baseline_longhorn_source }}
|
||||
(device={{ node_baseline_longhorn_device }},
|
||||
claim_mountpoint={{ node_baseline_longhorn_claim_mountpoint | default('') }}),
|
||||
etcd_source={{ node_baseline_etcd_source }}
|
||||
(device={{ node_baseline_etcd_device }}).
|
||||
Valid Longhorn sources: `device` (a dedicated block device, dev's shape) or
|
||||
`claim-mountpoint` (re-purpose the partition already mounted at
|
||||
node_baseline_longhorn_claim_mountpoint, prod's shape). Valid etcd sources:
|
||||
`device`, `root-filesystem` (etcd on the OS filesystem, no claimed partition under
|
||||
it) or `shared-data-partition` (etcd inside node_baseline_k3s_data_dir on the CLAIMED
|
||||
data partition — prod's shape since 2026-09-04, which additionally requires
|
||||
longhorn_source `claim-mountpoint` and a k3s data dir under
|
||||
node_baseline_data_mount={{ node_baseline_data_mount }};
|
||||
k3s_data_dir={{ node_baseline_k3s_data_dir }}). Every one of these is an EXPLICIT
|
||||
operator decision recorded in infra/PROD-PROVISIONING.md §2, not an omission.
|
||||
Declare it in the environment's group_vars and record the reasoning in
|
||||
§2 in the SAME change. To install ONLY the package floor meanwhile, re-run with
|
||||
`--tags packages`, which skips every disk task including this assert.
|
||||
quiet: true
|
||||
|
||||
# =====================================================================================
|
||||
# THE DATA PARTITION (prod: Longhorn store + k3s data dir; dev: Longhorn store only)
|
||||
# =====================================================================================
|
||||
|
||||
# THE IDEMPOTENCY PIVOT for the claim path, and the reason a second run is a no-op
|
||||
# rather than a reformat. Once the claim has happened the partition carries the ext4
|
||||
# label node_baseline_data_fs_label and is mounted at node_baseline_data_mount — the claim
|
||||
# mountpoint (/srv) is gone from both /proc/mounts and /etc/fstab, so "discover what is
|
||||
# mounted at /srv" would find nothing on run 2. Asking `blkid -L <label>` instead answers
|
||||
# "has this node already been claimed?" from the durable, on-disk evidence: rc 0 + a device
|
||||
# path means converged, rc != 0 means the claim has not been made. Everything destructive
|
||||
# below is gated on rc != 0.
|
||||
- name: "Data partition: look for a filesystem already carrying the data-partition label"
|
||||
ansible.builtin.command: "blkid -L {{ node_baseline_data_fs_label }}"
|
||||
register: node_baseline_lh_labelled
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
|
||||
# --- The TRANSITION (2026-09-04 layout revision; one-time, then this whole block is dead) --
|
||||
# WHY THIS EXISTS. The idempotency pivot above is "does the CURRENT label exist?". When the
|
||||
# label ITSELF changes, an already-claimed node answers "no" and the greenfield claim path
|
||||
# takes over — which then fails closed at "the claim mountpoint must actually be a
|
||||
# mountpoint", because /srv was consumed by the previous claim. Fail-closed is correct
|
||||
# behaviour but it is not convergence, and the prod nodes are in exactly that state today:
|
||||
# claimed under LABEL=longhorn at /var/lib/longhorn by the previous version of this role.
|
||||
# So the role must handle the TRANSITION explicitly instead of pretending prod is greenfield.
|
||||
#
|
||||
# WHAT IT DOES: unmount the retired mountpoint -> drop its fstab entry -> re-stamp the
|
||||
# partition with the new label. It never touches the DATA and never reformats: e2label is a
|
||||
# metadata write, exactly as in the claim path below. The common mount task then mounts
|
||||
# LABEL=<new> at the new mountpoint, and the node is converged. On every subsequent run
|
||||
# `blkid -L <new label>` succeeds and none of this is reachable.
|
||||
#
|
||||
# THREE INDEPENDENT REASONS THE LIVE DEV CLUSTER CANNOT ENTER THIS PATH — it has 18 Longhorn
|
||||
# volumes whose replicas live on the partition labelled `longhorn` at /var/lib/longhorn, and
|
||||
# unmounting it would orphan every one of them:
|
||||
# 1. node_baseline_data_retired_fs_label defaults to "" and is set ONLY in
|
||||
# group_vars/k3s_prod/vars.yml. No retired label declared => no transition, ever.
|
||||
# 2. Gated on node_baseline_longhorn_source == 'claim-mountpoint'. Dev is `device`, so
|
||||
# the same structural gate that keeps dev out of the claim path keeps it out of this one.
|
||||
# 3. The emptiness assert below. Even if 1 and 2 were both defeated, a partition with
|
||||
# replica directories on it fails closed and nothing is unmounted.
|
||||
# Dev's label and mountpoint are also simply NOT CHANGING, so there is nothing to migrate.
|
||||
- name: "Data partition (transition): look for a filesystem still carrying the RETIRED label"
|
||||
ansible.builtin.command: "blkid -L {{ node_baseline_data_retired_fs_label }}"
|
||||
register: node_baseline_lh_retired_labelled
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
when:
|
||||
- node_baseline_longhorn_source == 'claim-mountpoint'
|
||||
- node_baseline_data_retired_fs_label | length > 0
|
||||
- node_baseline_lh_labelled.rc != 0
|
||||
|
||||
- name: "Data partition: decide between TRANSITION, CLAIM, and already-converged"
|
||||
ansible.builtin.set_fact:
|
||||
# A node claimed under the RETIRED label. Mutually exclusive with the claim below.
|
||||
node_baseline_lh_transition_needed: >-
|
||||
{{ node_baseline_longhorn_source == 'claim-mountpoint'
|
||||
and node_baseline_data_retired_fs_label | length > 0
|
||||
and node_baseline_lh_labelled.rc != 0
|
||||
and (node_baseline_lh_retired_labelled.rc | default(1)) == 0 }}
|
||||
# True only for a claim-mountpoint environment that has NOT been claimed under EITHER
|
||||
# label. A `device` environment (dev) can never enter the claim path at all — that is
|
||||
# the structural guarantee that no generalisation of prod's behaviour can reach the
|
||||
# LIVE dev nodes and their /dev/vdb Longhorn replicas.
|
||||
node_baseline_lh_claim_needed: >-
|
||||
{{ node_baseline_longhorn_source == 'claim-mountpoint'
|
||||
and node_baseline_lh_labelled.rc != 0
|
||||
and (node_baseline_lh_retired_labelled.rc | default(1)) != 0 }}
|
||||
|
||||
- name: "Data partition (transition): resolve what is mounted at the retired mountpoint"
|
||||
ansible.builtin.command: >-
|
||||
findmnt --noheadings --first-only --output SOURCE
|
||||
--mountpoint {{ node_baseline_data_retired_mount }}
|
||||
register: node_baseline_lh_retired_findmnt
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
# Read the retired mountpoint's top level, same contract as the claim path: `lost+found` is
|
||||
# not data, hidden files cannot hide. THIS is the assert that makes the transition safe —
|
||||
# an unmount of a populated Longhorn store would orphan every replica on the node.
|
||||
- name: "Data partition (transition): read the contents of the retired mountpoint"
|
||||
ansible.builtin.find:
|
||||
paths: "{{ node_baseline_data_retired_mount }}"
|
||||
file_type: any
|
||||
hidden: true
|
||||
recurse: false
|
||||
excludes: ['lost+found']
|
||||
register: node_baseline_lh_retired_contents
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
- name: "Data partition (transition): prove this node is safe to re-label"
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
# 1. The retired mountpoint is STILL A MOUNT, and it is backed by the very device the
|
||||
# retired label names. The only state this role knows how to transition is the one
|
||||
# the previous version of it produced; anything else is an operator decision.
|
||||
- (node_baseline_lh_retired_findmnt.rc | default(1)) == 0
|
||||
- >-
|
||||
(node_baseline_lh_retired_findmnt.stdout | default('') | trim)
|
||||
== (node_baseline_lh_retired_labelled.stdout | default('') | trim)
|
||||
# 2. It is EMPTY. Nothing is being orphaned. A populated store means either this is
|
||||
# not the node we think it is, or a tenant has already written data — in both cases
|
||||
# unmounting and re-labelling under it is destructive and the role must stop.
|
||||
- (node_baseline_lh_retired_contents.files | length) == 0
|
||||
fail_msg: >-
|
||||
REFUSING to transition this node's data partition from
|
||||
LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
|
||||
to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}.
|
||||
device by retired label='{{ node_baseline_lh_retired_labelled.stdout | default('') | trim }}',
|
||||
device mounted at the retired mountpoint='{{ node_baseline_lh_retired_findmnt.stdout | default('') | trim }}'
|
||||
(findmnt rc={{ node_baseline_lh_retired_findmnt.rc | default('n/a') }}),
|
||||
non-lost+found entries={{ node_baseline_lh_retired_contents.files | default([]) | length }}
|
||||
{{ node_baseline_lh_retired_contents.files | default([]) | map(attribute='path') | list }}.
|
||||
If the partition holds DATA, this transition would orphan it — Longhorn replicas are
|
||||
not recoverable from an unmounted, re-labelled filesystem. Migrate the data
|
||||
deliberately, or drop node_baseline_data_retired_fs_label and leave the node alone.
|
||||
Do NOT relax this assert.
|
||||
quiet: true
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
- name: "Data partition (transition): report exactly what is about to change"
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
Transitioning {{ node_baseline_lh_retired_labelled.stdout | trim }} (verified empty)
|
||||
from LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
|
||||
to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}:
|
||||
unmount -> drop its fstab entry -> relabel -> re-mount at the new path.
|
||||
No filesystem is created and no data is moved.
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
# e2label refuses to run on a mounted filesystem, and the fstab entry must go in the same
|
||||
# breath or the next boot re-mounts the partition at the RETIRED path underneath the new one.
|
||||
- name: "Data partition (transition): unmount the retired mountpoint"
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_data_retired_mount }}"
|
||||
state: unmounted
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
- name: "Data partition (transition): remove the retired mountpoint's /etc/fstab entry"
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_data_retired_mount }}"
|
||||
state: absent_from_fstab
|
||||
backup: true
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
- name: "Data partition (transition): re-stamp the partition with the new filesystem label"
|
||||
ansible.builtin.command: >-
|
||||
e2label {{ node_baseline_lh_retired_labelled.stdout | trim }}
|
||||
{{ node_baseline_data_fs_label }}
|
||||
changed_when: true
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
# --- The claim (prod, first run only) -------------------------------------------------
|
||||
# Ask the node, do not assume. `--mountpoint` (not `--target`) matches ONLY an exact
|
||||
# mountpoint: if /srv is not itself a mount, findmnt exits 1 rather than silently
|
||||
# returning the device behind `/` — which is precisely the mistake that would hand the
|
||||
# root partition to e2label.
|
||||
- name: "Longhorn (claim): resolve the block device mounted at the claim mountpoint"
|
||||
ansible.builtin.command: >-
|
||||
findmnt --noheadings --first-only --output SOURCE
|
||||
--mountpoint {{ node_baseline_longhorn_claim_mountpoint }}
|
||||
register: node_baseline_lh_claim_findmnt
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
- name: "Longhorn (claim): the claim mountpoint must actually be a mountpoint"
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- node_baseline_lh_claim_findmnt.rc == 0
|
||||
- (node_baseline_lh_claim_findmnt.stdout | trim) is match('^/dev/')
|
||||
fail_msg: >-
|
||||
node_baseline_longhorn_source is `claim-mountpoint` and no filesystem is claimed
|
||||
yet, but nothing is mounted at {{ node_baseline_longhorn_claim_mountpoint }} on
|
||||
this node (findmnt rc={{ node_baseline_lh_claim_findmnt.rc | default('n/a') }},
|
||||
stdout='{{ node_baseline_lh_claim_findmnt.stdout | default('') | trim }}').
|
||||
The role will NOT go looking for "a big partition" to claim instead. Either the
|
||||
node is partitioned differently from the fleet — in which case fix the node or give
|
||||
this host its own declaration — or the claim already happened and the label was
|
||||
removed by hand.
|
||||
quiet: true
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
- name: "Longhorn (claim): inspect the candidate partition"
|
||||
ansible.builtin.command: >-
|
||||
lsblk --json --nodeps --bytes
|
||||
--output PATH,TYPE,FSTYPE,LABEL,SIZE
|
||||
{{ node_baseline_lh_claim_findmnt.stdout | trim }}
|
||||
register: node_baseline_lh_claim_lsblk
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
# The devices this role must never, under any circumstances, hand to e2label. Built from
|
||||
# gathered facts (`gather_facts: true` in the play) plus the live swap list, so it is the
|
||||
# NODE's account of what is load-bearing, not a hardcoded list of partition names.
|
||||
- name: "Longhorn (claim): list the devices currently backing active swap"
|
||||
ansible.builtin.command: swapon --noheadings --show=NAME
|
||||
register: node_baseline_lh_swapdevs
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
# Read the mountpoint's top level. `excludes: lost+found` because every freshly-mkfs'd
|
||||
# ext4 has one and it is not data. hidden: true so a dotfile cannot hide from the check.
|
||||
- name: "Longhorn (claim): read the contents of the mountpoint to be claimed"
|
||||
ansible.builtin.find:
|
||||
paths: "{{ node_baseline_longhorn_claim_mountpoint }}"
|
||||
file_type: any
|
||||
hidden: true
|
||||
recurse: false
|
||||
excludes: ['lost+found']
|
||||
register: node_baseline_lh_claim_contents
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
# THE GUARD WALL. Five independent conditions, every one of which must hold before the
|
||||
# role modifies a filesystem it did not create. Any single failure stops the play with
|
||||
# the evidence in the message — nothing is unmounted, relabelled or mounted first.
|
||||
- name: "Longhorn (claim): prove the candidate partition is safe to claim"
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
# 1. It is a real block device — a partition or a whole disk. Catches a claim
|
||||
# mountpoint that is actually tmpfs, an NFS/CIFS export, or a bind mount, all of
|
||||
# which would have produced a non-/dev SOURCE or an lsblk row of another type.
|
||||
- node_baseline_lh_candidate.type in ['part', 'disk']
|
||||
# 2. It is not backing anything the OS needs. Derived from ansible_mounts + swapon,
|
||||
# so a differently-partitioned node is caught by its own facts.
|
||||
- node_baseline_lh_candidate.path not in node_baseline_lh_protected_devices
|
||||
# 3. It already holds ext4. The claim path RELABELS rather than reformats (see
|
||||
# below), which is only meaningful for the filesystem type we are about to
|
||||
# declare in fstab. Anything else is an operator decision, not a role's.
|
||||
- node_baseline_lh_candidate.fstype == 'ext4'
|
||||
# 4. It carries no filesystem label, or already carries ours. A partition labelled
|
||||
# `boot`, `data`, `backup`… is somebody's named volume; stamping `longhorn` over
|
||||
# it would make it unfindable by its own fstab entry.
|
||||
- (node_baseline_lh_candidate.label | default('', true)) in ['', node_baseline_data_fs_label]
|
||||
# 5. It is big enough to be the partition we meant. A data partition of a few GB is
|
||||
# not "small", it is the wrong partition — and on prod it must hold BOTH tenants.
|
||||
- (node_baseline_lh_candidate.size | int) >= (node_baseline_longhorn_min_size_gb | int) * 1024 * 1024 * 1024
|
||||
# 6. It is EMPTY. The operator's premise is that prod's /srv holds nothing but the
|
||||
# ext4 5% root reserve; this is where that premise is verified per node instead
|
||||
# of trusted. If any node has real data on it, the play stops here — the role
|
||||
# never wipes data it cannot account for.
|
||||
- (node_baseline_lh_claim_contents.files | length) == 0
|
||||
fail_msg: >-
|
||||
REFUSING to claim {{ node_baseline_lh_candidate.path }}
|
||||
(mounted at {{ node_baseline_longhorn_claim_mountpoint }}) for the Longhorn store.
|
||||
type={{ node_baseline_lh_candidate.type }},
|
||||
fstype={{ node_baseline_lh_candidate.fstype }},
|
||||
label='{{ node_baseline_lh_candidate.label | default('', true) }}',
|
||||
size={{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB
|
||||
(minimum {{ node_baseline_longhorn_min_size_gb }} GiB),
|
||||
non-lost+found entries={{ node_baseline_lh_claim_contents.files | length }}
|
||||
{{ node_baseline_lh_claim_contents.files | map(attribute='path') | list }},
|
||||
devices this node needs and that must never be claimed:
|
||||
{{ node_baseline_lh_protected_devices }}.
|
||||
This is the destructive step of the role and it fails CLOSED. If the partition
|
||||
holds data, back it up or pick a different one; do NOT relax this assert.
|
||||
quiet: true
|
||||
vars:
|
||||
node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
|
||||
# The node's OWN account of what it cannot afford to lose: the devices behind the
|
||||
# mounts in node_baseline_protected_mounts (from gathered facts, so a node
|
||||
# partitioned differently is judged on its own layout) plus every active swap device.
|
||||
# Built here rather than hardcoded so no partition-name list can go stale.
|
||||
node_baseline_lh_protected_devices: >-
|
||||
{{ (ansible_facts.mounts | default([])
|
||||
| selectattr('mount', 'in', node_baseline_protected_mounts)
|
||||
| map(attribute='device') | list)
|
||||
+ (node_baseline_lh_swapdevs.stdout_lines | default([]) | map('trim') | select | list) }}
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
- name: "Longhorn (claim): report exactly what is about to be claimed"
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
Claiming {{ node_baseline_lh_candidate.path }}
|
||||
({{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB ext4,
|
||||
currently mounted at {{ node_baseline_longhorn_claim_mountpoint }}, verified empty)
|
||||
as the data partition: unmount -> drop its fstab entry ->
|
||||
relabel LABEL={{ node_baseline_data_fs_label }} ->
|
||||
mount at {{ node_baseline_data_mount }}.
|
||||
vars:
|
||||
node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
# --- Why the role must unmount and relabel, not just point fstab at the device ---------
|
||||
# The mount task below mounts by `LABEL=`, deliberately: a label survives /dev/sda5 ->
|
||||
# /dev/nvme0n1p5 renaming across the fleet and across a controller change, which a device
|
||||
# path does not. The installer-created /srv filesystem has NO label (fstab pins it by
|
||||
# UUID), so it must be given one. e2label refuses to run on a mounted filesystem, hence
|
||||
# the unmount first — and the fstab entry must go in the same breath, or the next boot
|
||||
# re-mounts the same partition at /srv underneath Longhorn.
|
||||
#
|
||||
# RELABEL, NOT REFORMAT. The partition is already ext4 and already verified empty, so
|
||||
# mkfs would buy nothing and would destroy anything the emptiness check got wrong.
|
||||
# e2label is a metadata write: if some assert above was too permissive, the worst case is
|
||||
# a renamed filesystem, not a destroyed one. (This is also why `community.general.filesystem`
|
||||
# cannot do this job: it declines to touch a device that already has a filesystem, so it
|
||||
# would silently no-op and then the LABEL= mount would fail with no useful message.)
|
||||
- name: "Longhorn (claim): unmount the claim mountpoint"
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_longhorn_claim_mountpoint }}"
|
||||
state: unmounted
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
- name: "Longhorn (claim): remove the claim mountpoint's /etc/fstab entry"
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_longhorn_claim_mountpoint }}"
|
||||
state: absent_from_fstab
|
||||
backup: true
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
- name: "Longhorn (claim): stamp the data-partition filesystem label onto the partition"
|
||||
ansible.builtin.command: >-
|
||||
e2label {{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0].path }}
|
||||
{{ node_baseline_data_fs_label }}
|
||||
changed_when: true
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
# --- The dedicated-device path (dev) --------------------------------------------------
|
||||
# Unchanged behaviour for `source: device`: data-safe, because community.general.filesystem
|
||||
# never reformats a device that already carries a filesystem.
|
||||
- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
|
||||
community.general.filesystem:
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_longhorn_device }}"
|
||||
opts: "-L {{ node_baseline_data_fs_label }}"
|
||||
when: node_baseline_longhorn_source == 'device'
|
||||
|
||||
# --- Common to ALL paths (device, claim, transition) ----------------------------------
|
||||
# Mounted by LABEL for every shape, so dev and prod share ONE fstab convention and one
|
||||
# mount task. Idempotent: on a converged node this reports ok, which is what makes a
|
||||
# re-run a no-op rather than a reformat. It is also what completes the transition above:
|
||||
# the retired mountpoint is gone from fstab and the partition now answers to the new label,
|
||||
# so this task mounts it at the new path.
|
||||
- name: Mount the data partition
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_data_mount }}"
|
||||
src: "LABEL={{ node_baseline_data_fs_label }}"
|
||||
fstype: ext4
|
||||
opts: "defaults,nofail,x-systemd.device-timeout=5s"
|
||||
state: mounted
|
||||
|
||||
# --- The per-tenant subdirectories ---------------------------------------------------
|
||||
# WHY THIS IS CONDITIONAL RATHER THAN UNCONDITIONAL. On prod the partition serves two
|
||||
# tenants and each needs its own subdirectory carved out of it. On DEV neither path is a
|
||||
# subdirectory of the mount: Longhorn's dataPath IS the mount root, and k3s's data dir is
|
||||
# /var/lib/rancher/k3s on the OS filesystem, owned and created by a LIVE k3s. Creating
|
||||
# either there would at best be a no-op and at worst re-mode the running dev cluster's data
|
||||
# dir, so the role only creates paths that are STRICT DESCENDANTS of the partition it
|
||||
# mounted — i.e. only the directories it is actually responsible for. That test is derived
|
||||
# from the paths themselves, so no environment has to remember to switch it off.
|
||||
#
|
||||
# MODES. Longhorn's dataPath keeps 0755, matching what mkfs gives dev's mount root and what
|
||||
# the Longhorn manager expects to find. k3s's data dir is 0700 from the start: it holds the
|
||||
# etcd database, the node token and the containerd image store, and k3s would tighten it to
|
||||
# 0700 itself on first start — doing it here means it is never briefly world-readable.
|
||||
- name: Create the per-tenant data directories on the data partition
|
||||
ansible.builtin.file:
|
||||
path: "{{ item.path }}"
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "{{ item.mode }}"
|
||||
loop:
|
||||
- { path: "{{ node_baseline_longhorn_data_path }}", mode: "0755", tenant: "Longhorn replicas" }
|
||||
- { path: "{{ node_baseline_k3s_data_dir }}", mode: "0700", tenant: "k3s data dir (etcd + containerd)" }
|
||||
loop_control:
|
||||
label: "{{ item.path }} ({{ item.tenant }})"
|
||||
when: item.path.startswith(node_baseline_data_mount ~ '/')
|
||||
|
||||
# =====================================================================================
|
||||
# ETCD DATA DIR
|
||||
# =====================================================================================
|
||||
|
||||
# --- The dedicated-device path (dev) --------------------------------------------------
|
||||
# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
|
||||
# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
|
||||
# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
|
||||
# a disk that already carries a filesystem, so re-running against an already-migrated
|
||||
# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
|
||||
# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
|
||||
# role would mount over a populated directory.
|
||||
- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
|
||||
community.general.filesystem:
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_etcd_device }}"
|
||||
opts: "-L {{ node_baseline_etcd_fs_label }}"
|
||||
when: node_baseline_etcd_source == 'device'
|
||||
|
||||
- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_etcd_mount }}"
|
||||
src: "LABEL={{ node_baseline_etcd_fs_label }}"
|
||||
fstype: ext4
|
||||
# noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
|
||||
# missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
|
||||
opts: "defaults,noatime"
|
||||
dump: "0"
|
||||
passno: "2"
|
||||
state: mounted
|
||||
when: node_baseline_etcd_source == 'device'
|
||||
|
||||
# --- The no-dedicated-device paths (prod) ---------------------------------------------
|
||||
# Prod has ONE physical disk per node, already fully partitioned, with no free slot verified
|
||||
# populated — so etcd gets no device of its own. That is an operator decision
|
||||
# (PROD-PROVISIONING.md §2, 2026-09-03), and the shared-device I/O contention it accepts is
|
||||
# UNCHANGED by the 2026-09-04 layout revision: dev's etcd/Longhorn contention incident was
|
||||
# on a spinning HDD, and every prod node is SSD or NVMe, where concurrent rebuild I/O costs
|
||||
# latency rather than collapsing it. Accepted to be MEASURED, not assumed.
|
||||
#
|
||||
# WHAT DID CHANGE ON 2026-09-04 is WHICH FILESYSTEM etcd shares. It used to be the OS
|
||||
# filesystem (`root-filesystem`), which on these nodes means /var — a separate partition of
|
||||
# 11.8 GiB carrying the containerd image store too. Measured on dev, containerd alone is
|
||||
# 21-25 GB per node, so prod would have filled /var during its first image pulls and etcd
|
||||
# could not have fsynced its WAL. So k3s's whole data dir moved onto the claimed 424 GB
|
||||
# partition and etcd moved with it: `shared-data-partition`. Both enum values are still
|
||||
# supported, and the role now PROVES which one is true instead of printing a claim.
|
||||
#
|
||||
# There is nothing to CREATE here for either shape — the data-partition section above
|
||||
# already created node_baseline_k3s_data_dir, and k3s creates server/db inside it — so the
|
||||
# role's job is to verify the assumption rather than act on it.
|
||||
- name: "etcd (no dedicated device): check for anything already mounted at the etcd data dir"
|
||||
ansible.builtin.command: >-
|
||||
findmnt --noheadings --first-only --output SOURCE
|
||||
--mountpoint {{ node_baseline_etcd_mount }}
|
||||
register: node_baseline_etcd_existing_mount
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
|
||||
|
||||
- name: "etcd (no dedicated device): assert no dedicated device is already mounted there"
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- node_baseline_etcd_existing_mount.rc != 0
|
||||
fail_msg: >-
|
||||
node_baseline_etcd_source is `{{ node_baseline_etcd_source }}` (etcd deliberately has
|
||||
NO dedicated device — PROD-PROVISIONING.md §2) but {{ node_baseline_etcd_mount }} is
|
||||
ALREADY a separate mount, backed by
|
||||
{{ node_baseline_etcd_existing_mount.stdout | default('') | trim }}. That is a
|
||||
contradiction the role will not resolve on its own: either this node was previously
|
||||
built with a dedicated etcd device (in which case unmount it deliberately, after
|
||||
accounting for the etcd data on it) or the declaration is wrong. Refusing to leave
|
||||
the node in a state where the declared layout and the real one disagree.
|
||||
quiet: true
|
||||
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
|
||||
|
||||
# Evidence, not decoration: §2 records which filesystem etcd shares, and this establishes
|
||||
# WHICH one per node so the claim is auditable from a run log. `findmnt --target` needs an
|
||||
# existing path and k3s has not created server/db yet, so walk up to the deepest ancestor
|
||||
# that does exist — the same filesystem the data dir will be created on. TARGET is captured
|
||||
# separately from the human-readable line because the assert below judges on it.
|
||||
- name: "etcd (no dedicated device): resolve the filesystem etcd will actually live on"
|
||||
ansible.builtin.shell: |
|
||||
set -eu
|
||||
p={{ node_baseline_etcd_mount | quote }}
|
||||
while [ ! -d "$p" ]; do p=$(dirname "$p"); done
|
||||
findmnt --noheadings --first-only --output TARGET --target "$p"
|
||||
findmnt --noheadings --first-only --output SOURCE,TARGET,SIZE,AVAIL --target "$p"
|
||||
register: node_baseline_etcd_backing_fs
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
|
||||
|
||||
# THE ASSERT THAT MAKES THE WHOLE LAYOUT REVISION SELF-PROVING, and the reason
|
||||
# `shared-data-partition` is a distinct enum value rather than a comment. Declaring
|
||||
# k3s_data_dir: /data/k3s does NOT by itself put etcd on the data partition: if the mount
|
||||
# above had silently failed, /data would be a plain directory on `/` and etcd plus 21-25 GB
|
||||
# of container images would land back on the OS filesystem — the exact failure this change
|
||||
# exists to prevent, and completely invisible until the disk filled. So the role checks that
|
||||
# the filesystem backing the etcd data dir IS the data partition's mount, per node, on every
|
||||
# run. `root-filesystem` is deliberately NOT asserted this way: it makes no claim about
|
||||
# which filesystem it lands on beyond "not a dedicated one", and the debug line below is the
|
||||
# whole record for that shape.
|
||||
- name: "etcd (shared-data-partition): assert etcd really lands on the data partition"
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- >-
|
||||
(node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim)
|
||||
== node_baseline_data_mount
|
||||
fail_msg: >-
|
||||
node_baseline_etcd_source is `shared-data-partition`, which asserts that
|
||||
{{ node_baseline_etcd_mount }} sits on the claimed data partition mounted at
|
||||
{{ node_baseline_data_mount }} — but the filesystem actually backing it is mounted at
|
||||
'{{ node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim }}'
|
||||
(full: {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}).
|
||||
The data partition is therefore NOT mounted where this layout requires, and k3s would
|
||||
put etcd's WAL AND the containerd image store (21-25 GB measured on dev) on that
|
||||
filesystem instead. On prod that is /var at 11.8 GiB: it would fill during the first
|
||||
image pulls and etcd would stop being able to fsync its WAL. DO NOT install k3s on
|
||||
this node until the mount is correct.
|
||||
quiet: true
|
||||
when: node_baseline_etcd_source == 'shared-data-partition'
|
||||
|
||||
- name: "etcd (no dedicated device): show which filesystem etcd will live on"
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
etcd data dir {{ node_baseline_etcd_mount }} has NO dedicated device
|
||||
(source={{ node_baseline_etcd_source }}); it lands on
|
||||
{{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}
|
||||
(SOURCE TARGET SIZE AVAIL). Shared-device I/O contention is an accepted, to-be-
|
||||
measured risk — §2.
|
||||
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
# TAGS — every task here carries one of `packages`, `system`, `disks` or `dns`. An untagged
|
||||
# run is unchanged (all four run, in this order). The split exists because the PACKAGE floor
|
||||
# TAGS — every task here carries one of `packages`, `kernel`, `system`, `disks` or `dns`. An
|
||||
# untagged run is unchanged (all five run, in this order). The split exists because the PACKAGE floor
|
||||
# and the DATA-DISK work have different readiness: packages are safe on any node from
|
||||
# first contact, while the disk tasks need block devices that a new environment may not
|
||||
# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
|
||||
@@ -12,6 +12,10 @@
|
||||
# mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
|
||||
# at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
|
||||
# ansible-playbook playbooks/node-baseline.yml --tags dns
|
||||
# `kernel` (the backports-kernel block below) is separated for the same reason `packages`
|
||||
# is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
|
||||
# it must be runnable without touching a disk decision that is still open:
|
||||
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
|
||||
|
||||
- name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
|
||||
ansible.builtin.apt:
|
||||
@@ -21,6 +25,237 @@
|
||||
cache_valid_time: 3600
|
||||
tags: [packages]
|
||||
|
||||
# =====================================================================================
|
||||
# BACKPORTS KERNEL — tag `kernel` (EFI memattr; infra/PROD-PROVISIONING.md §0b and §3)
|
||||
# =====================================================================================
|
||||
# A STANDING DEVIATION from the Debian 13 baseline, encoded here so every current AND
|
||||
# FUTURE prod node gets it from provisioning instead of from someone remembering.
|
||||
#
|
||||
# WHAT IT FIXES. HP R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI
|
||||
# Memory Attributes Table. `efi_memattr_init()` in Linux < 6.13 computes
|
||||
# `tbl_size = sizeof(*tbl) + num_entries * desc_size` from two UNVALIDATED u32 fields into a
|
||||
# signed int and `memblock_reserve()`s the product; when bit 31 lands the value sign-extends
|
||||
# through phys_addr_t, `memblock_cap_size()` clamps it to PHYS_ADDR_MAX, and EVERY physical
|
||||
# address above the table is reserved. k3s-prod-2 booted with 2.6 GiB of its 16 GB, twice.
|
||||
# Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus"
|
||||
# (first released in v6.13) validates the size and rejects the table instead. It is absent
|
||||
# from v6.12 and from linux-6.12.y, has no `Cc: stable` and no `Fixes:` tag, so no automatic
|
||||
# stable feed will ever pick it up. There is NO kernel cmdline workaround —
|
||||
# `efi_memattr_init()` runs from `efi_config_parse_tables()` before any `efi=` bail-out could
|
||||
# take effect. Booting a >= 6.13 kernel is the only fix, and ALL THREE prod nodes are on the
|
||||
# same lottery — node 1 and node 3 have merely drawn safe tickets so far.
|
||||
#
|
||||
# WHY IT IS SCOPED OFF FOR DEV. `node_baseline_backports_kernel` defaults to FALSE in
|
||||
# defaults/main.yml and is set true in inventory/group_vars/k3s_prod/vars.yml only. Dev is
|
||||
# Proxmox VMs whose firmware publishes no MAT at all — genuinely unaffected — and pulling a
|
||||
# LIVE cluster onto a backports kernel as a side effect of sharing this role would be a real
|
||||
# change for no benefit. See defaults/main.yml for why this is a variable rather than a
|
||||
# `k3s-prod-*` hostname test.
|
||||
#
|
||||
# IT DOES NOT REBOOT. A new kernel is inert until the node reboots, and this role
|
||||
# deliberately will not take one: k3s-prod-2 hung on a warm reboot on 2026-09-03 — powered,
|
||||
# off the LAN for ~38 minutes, power button required (§0b "SECOND FAULT"). Rebooting is
|
||||
# therefore a SERIALIZED, SUPERVISED operator step — one node at a time, verify each returns
|
||||
# before touching the next, never node 2 first — and a role cannot enforce that. The last
|
||||
# task here reports the pending reboot loudly instead of letting a green play imply the node
|
||||
# is already fixed.
|
||||
- name: Backports kernel for the EFI memattr fix (>= 6.13, commit 8fbe4c49c0cc)
|
||||
when: node_baseline_backports_kernel | bool
|
||||
tags: [kernel]
|
||||
block:
|
||||
|
||||
# The suite name and the running release must agree, or apt resolves no candidate and
|
||||
# `state: latest` reports "ok" while leaving the node on the vulnerable 6.12 kernel —
|
||||
# a green play that fixed nothing. Fail on the mismatch instead (e.g. a node
|
||||
# dist-upgraded to forky while group_vars still says trixie-backports).
|
||||
- name: Assert the configured backports suite matches this node's Debian release
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_facts['distribution'] == 'Debian'
|
||||
- node_baseline_backports_suite == (ansible_facts['distribution_release'] ~ '-backports')
|
||||
fail_msg: >-
|
||||
node_baseline_backports_kernel is enabled but node_baseline_backports_suite
|
||||
({{ node_baseline_backports_suite }}) is not the backports suite for this node's
|
||||
release ({{ ansible_facts['distribution'] }}
|
||||
{{ ansible_facts['distribution_release'] | default('?') }}). apt would resolve no
|
||||
candidate and this play would report success while leaving the node on the
|
||||
vulnerable kernel. Fix the suite in group_vars — do not remove this assert.
|
||||
quiet: true
|
||||
|
||||
# deb822 `.sources`, not a legacy one-line `deb` entry: it is the format Debian 13
|
||||
# prefers, it carries `Signed-By` explicitly (binding this source to the archive keyring
|
||||
# rather than to whatever is in /etc/apt/trusted.gpg.d), and a `copy` of exact content is
|
||||
# idempotent in a way `apt_repository` — which rewrites and reformats — is not.
|
||||
#
|
||||
# THERE IS DELIBERATELY NO /etc/apt/preferences.d PIN, and that is not an oversight. The
|
||||
# backports Release file carries `NotAutomatic: yes` + `ButAutomaticUpgrades: yes`, which
|
||||
# apt already reads as: priority 100 so NOTHING is pulled from backports by accident, but
|
||||
# 500 for a package that was explicitly installed from there so it keeps getting updates.
|
||||
# Writing an explicit `Pin-Priority: 100` looks like belt-and-braces and is actively
|
||||
# HARMFUL — it overrides the ButAutomaticUpgrades half, and the kernel would then never
|
||||
# be upgraded again by `apt upgrade`. Enabling the source IS the whole configuration.
|
||||
- name: Enable the Debian backports apt source (deb822)
|
||||
ansible.builtin.copy:
|
||||
dest: "/etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
|
||||
# A DELIBERATE, STANDING deviation from the Debian 13 baseline, enabled ONLY so this
|
||||
# machine can boot a kernel >= 6.13 that validates the corrupt EFI Memory Attributes
|
||||
# Table its firmware publishes (upstream 8fbe4c49c0cc). See
|
||||
# infra/PROD-PROVISIONING.md §0b/§3, and {{ node_baseline_backports_kernel_stamp }}
|
||||
# on this node.
|
||||
#
|
||||
# Backports is NotAutomatic, so nothing is taken from here unless asked for by name
|
||||
# with `-t {{ node_baseline_backports_suite }}`. Do NOT add an apt pin for it:
|
||||
# ButAutomaticUpgrades is what keeps the installed kernel patched, and an explicit
|
||||
# Pin-Priority would switch that off.
|
||||
Types: deb
|
||||
URIs: {{ node_baseline_backports_mirror }}
|
||||
Suites: {{ node_baseline_backports_suite }}
|
||||
Components: {{ node_baseline_backports_components | join(' ') }}
|
||||
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
|
||||
register: node_baseline_backports_source
|
||||
|
||||
- name: Refresh the apt cache after enabling backports
|
||||
ansible.builtin.apt:
|
||||
update_cache: true
|
||||
when: node_baseline_backports_source is changed
|
||||
|
||||
# `state: latest`, NOT `state: present`. linux-image-amd64 is ALREADY installed on these
|
||||
# nodes from trixie main (6.12.107-1), so `present` would find the name satisfied and do
|
||||
# nothing at all — the single most likely way to write this task and have it silently fix
|
||||
# nothing forever. `latest` + `default_release` resolves the candidate inside backports
|
||||
# and upgrades the metapackage to it. Once the node is current a re-run is a genuine
|
||||
# no-op: no change, no churn, no spurious reboot flag. Scope is exactly
|
||||
# node_baseline_backports_kernel_packages — one metapackage — so nothing else on the node
|
||||
# is dragged out of the main archive.
|
||||
- name: Install the fixed kernel from backports (>= 6.13)
|
||||
ansible.builtin.apt:
|
||||
name: "{{ node_baseline_backports_kernel_packages }}"
|
||||
state: latest
|
||||
default_release: "{{ node_baseline_backports_suite }}"
|
||||
update_cache: true
|
||||
cache_valid_time: 3600
|
||||
register: node_baseline_backports_kernel_apt
|
||||
|
||||
# WHICH VERSION DID THIS NODE ACTUALLY LAND ON. Read back rather than assumed: the
|
||||
# metapackage tracks a moving target, so "we installed linux-image-amd64" is not an
|
||||
# answer anybody can roll back from six months later.
|
||||
- name: Read back the installed backports kernel metapackage version
|
||||
ansible.builtin.command: dpkg-query -W -f '${Version}' {{ node_baseline_backports_kernel_packages | first }}
|
||||
register: node_baseline_backports_kernel_version
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
|
||||
# `linux-version` ships in linux-base and knows Debian's kernel version ordering, which
|
||||
# neither `sort -V` nor a Jinja sort gets right across +deb13 / ABI suffixes. The full
|
||||
# list is the ROLLBACK INVENTORY: every entry is still installed and selectable from
|
||||
# GRUB's "Advanced options" without uninstalling anything.
|
||||
- name: List every kernel image installed on this node, newest first
|
||||
ansible.builtin.shell:
|
||||
cmd: set -o pipefail; linux-version list | linux-version sort --reverse
|
||||
executable: /bin/bash
|
||||
register: node_baseline_installed_kernels
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
|
||||
# The node must be able to explain itself to an operator standing in front of it with no
|
||||
# repo checkout. Content is deliberately time-INVARIANT (no timestamp) so re-running the
|
||||
# role is a real no-op rather than a permanent daily "changed".
|
||||
- name: Record the deviation, the exact version, and the rollback path on the node
|
||||
ansible.builtin.copy:
|
||||
dest: "{{ node_baseline_backports_kernel_stamp }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
|
||||
#
|
||||
# THIS NODE DELIBERATELY DOES NOT RUN THE DEBIAN 13 STOCK KERNEL.
|
||||
#
|
||||
# WHY
|
||||
# This machine's HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table.
|
||||
# Linux < 6.13 reserves memory from it without validating its size, and an unlucky
|
||||
# boot reserves everything above the table: 2.6 GiB of 16 GB, observed twice on
|
||||
# k3s-prod-2. Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size
|
||||
# is clearly bogus" (first released in v6.13) adds the validation; it is NOT in
|
||||
# v6.12 and NOT in linux-6.12.y. There is NO kernel cmdline workaround.
|
||||
# Full write-up: infra/PROD-PROVISIONING.md §0b in the home-network-k3s-impl repo.
|
||||
#
|
||||
# PROOF THE FIX IS ACTIVE (not merely a lucky boot)
|
||||
# BOOT-INDEPENDENT, and this is the one to trust — the string only exists in
|
||||
# >= 6.13, so its presence in the RUNNING image is proof regardless of the boot:
|
||||
# zstd -dc < <(tail -c +$(...)) ... | grep 'Corrupted EFI Memory Attributes'
|
||||
# (roles/node_health does the version check for you; see the health report's
|
||||
# firmware.efi_memattr finding.)
|
||||
#
|
||||
# FROM dmesg, WHICH IS A LOTTERY. efi_memattr_init() on >= 6.13 has THREE gates
|
||||
# and each prints a DIFFERENT line — all three are rejections:
|
||||
# tbl->version > 2 'Unexpected EFI Memory Attributes table version %d'
|
||||
# (NOTE: printed with %d, so garbage may read NEGATIVE)
|
||||
# bad desc_size 'Unexpected EFI Memory Attributes descriptor size ...'
|
||||
# num_entries > 64K '[Firmware Bug]: Corrupted EFI Memory Attributes
|
||||
# Table detected! (version == ..., num_entries == ...)'
|
||||
# Which one appears depends on WHICH field this boot's garbage corrupted, so do
|
||||
# NOT treat the absence of the FW_BUG line as the fix being absent — all three
|
||||
# prod nodes printed the *version* line on 2026-09-04.
|
||||
# The one line that means the fix is NOT active:
|
||||
# 'Failed to map EFI Memory Attributes table @ 0x...' — the kernel trusted the
|
||||
# corrupt table. On a >= 6.13 kernel this should be unreachable; if it appears,
|
||||
# escalate, do not shrug.
|
||||
#
|
||||
# INSTALLED
|
||||
# source /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
|
||||
# package {{ node_baseline_backports_kernel_packages | join(' ') }}
|
||||
# version {{ node_baseline_backports_kernel_version.stdout | default('(not read)') }}
|
||||
# kernels installed on this node, newest first:
|
||||
{% for kv in (node_baseline_installed_kernels.stdout_lines | default([])) %}
|
||||
# {{ kv }}
|
||||
{% endfor %}
|
||||
#
|
||||
# SECURITY COST
|
||||
# Packages from *-backports are provided AS IS. They are NOT covered by the Debian
|
||||
# Security Team's guarantees for the main archive: no security-tracker commitment,
|
||||
# no DSAs. Accepted deliberately, in exchange for the node keeping its RAM.
|
||||
#
|
||||
# ROLLBACK — no repo checkout needed
|
||||
# 1. NON-DESTRUCTIVE, try this first. Every kernel listed above is still installed
|
||||
# and bootable: pick the 6.12 one from GRUB -> "Advanced options for Debian
|
||||
# GNU/Linux". That answers "is the new kernel the problem?" without
|
||||
# uninstalling anything. Note it reinstates the memattr bug for that boot.
|
||||
# 2. PERMANENT. Set node_baseline_backports_kernel: false in
|
||||
# inventory/group_vars/k3s_prod/vars.yml FIRST — otherwise the next Ansible run
|
||||
# puts the backports kernel straight back — then, on this node:
|
||||
# apt-get install --reinstall linux-image-amd64/{{ ansible_facts['distribution_release'] }}
|
||||
# apt-get purge <the backports linux-image-*+deb13-amd64 listed above>
|
||||
# rm /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
|
||||
# apt-get update && update-grub && reboot
|
||||
#
|
||||
# EXIT CONDITION
|
||||
# If 8fbe4c49c0cc is ever picked up into linux-6.12.y (and so into Debian's 6.12
|
||||
# rebase), this deviation can be removed entirely: flip the variable to false,
|
||||
# roll back as above, and delete the `kernel` block from roles/node_baseline.
|
||||
|
||||
# Say the pending reboot out loud. A green play here does NOT mean the node is fixed —
|
||||
# it means the fixed kernel is on disk. See the block header for why the role will not
|
||||
# reboot on its own.
|
||||
- name: Report whether this node is running the kernel it now has installed
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
{{ ('REBOOT REQUIRED — running ' ~ ansible_facts['kernel'] ~ ', newest installed is '
|
||||
~ newest ~ '. The EFI memattr fix is NOT active until this node reboots. Reboot
|
||||
prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER
|
||||
node 2 first (infra/PROD-PROVISIONING.md §0b "SECOND FAULT"). Afterwards confirm
|
||||
with: dmesg | grep -i "Corrupted EFI Memory Attributes Table"')
|
||||
if newest != ansible_facts['kernel']
|
||||
else ('Running the newest installed kernel (' ~ ansible_facts['kernel']
|
||||
~ '); no reboot pending for the memattr fix.') }}
|
||||
vars:
|
||||
newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
|
||||
|
||||
- name: Persist kernel modules for k3s/Cilium/Longhorn
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
|
||||
@@ -79,70 +314,32 @@
|
||||
state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
|
||||
tags: [system]
|
||||
|
||||
# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
|
||||
# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
|
||||
# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
|
||||
# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
|
||||
# which do not exist on bare metal either — a confusing failure that looks like a wiring
|
||||
# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
|
||||
# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
|
||||
- name: Assert this environment's data-disk devices have actually been decided
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- "'BLOCKED' not in node_baseline_longhorn_device"
|
||||
- "'BLOCKED' not in node_baseline_etcd_device"
|
||||
fail_msg: >-
|
||||
node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
|
||||
placeholders for this host's environment
|
||||
(longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
|
||||
The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
|
||||
and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
|
||||
the SMART read that decision waits on can be taken) re-run this play with
|
||||
`--tags packages`, which skips every disk task including this assert.
|
||||
quiet: true
|
||||
tags: [disks]
|
||||
|
||||
- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
|
||||
community.general.filesystem:
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_longhorn_device }}"
|
||||
opts: "-L {{ node_baseline_longhorn_fs_label }}"
|
||||
tags: [disks]
|
||||
|
||||
- name: Mount the Longhorn data disk
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_longhorn_mount }}"
|
||||
src: "LABEL={{ node_baseline_longhorn_fs_label }}"
|
||||
fstype: ext4
|
||||
opts: "defaults,nofail,x-systemd.device-timeout=5s"
|
||||
state: mounted
|
||||
tags: [disks]
|
||||
|
||||
# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
|
||||
# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
|
||||
# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
|
||||
# a disk that already carries a filesystem, so re-running against an already-migrated
|
||||
# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
|
||||
# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
|
||||
# role would mount over a populated directory.
|
||||
- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
|
||||
community.general.filesystem:
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_etcd_device }}"
|
||||
opts: "-L {{ node_baseline_etcd_fs_label }}"
|
||||
tags: [disks]
|
||||
|
||||
- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_etcd_mount }}"
|
||||
src: "LABEL={{ node_baseline_etcd_fs_label }}"
|
||||
fstype: ext4
|
||||
# noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
|
||||
# missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
|
||||
opts: "defaults,noatime"
|
||||
dump: "0"
|
||||
passno: "2"
|
||||
state: mounted
|
||||
# =====================================================================================
|
||||
# DATA DISKS — tag `disks`, in tasks/disks.yml
|
||||
# =====================================================================================
|
||||
# Split into its own file for two reasons. It is the only DESTRUCTIVE part of the role
|
||||
# (prod re-purposes the partition already mounted at /srv as the Longhorn store), so it
|
||||
# should be readable end-to-end on its own; and both environments' storage shapes — dev's
|
||||
# two dedicated virtio disks, prod's single partitioned physical disk with etcd left on
|
||||
# the OS filesystem — are declared there as an explicit enum rather than inferred from
|
||||
# which variables happen to be set. See tasks/disks.yml and infra/PROD-PROVISIONING.md §2.
|
||||
#
|
||||
# The `disks` tag is applied ONCE here, in BOTH places it is needed:
|
||||
# * `tags:` on the include itself, so `--tags packages` never even reaches the include
|
||||
# and a placeholder device path is not so much as templated; and
|
||||
# * `apply: tags:`, so the tag reaches the tasks INSIDE the file. This half is not
|
||||
# optional and not decoration — a DYNAMIC include (`include_tasks`) does NOT propagate
|
||||
# its own tags to its children the way `import_tasks` does, so without `apply` the
|
||||
# included tasks are untagged and `--tags disks` selects the include and then runs
|
||||
# NOTHING inside it. That failure is silent and reports success, which is the worst
|
||||
# shape a guard can have.
|
||||
# A FULL run against a node whose layout is undeclared stops on the assert at the top of
|
||||
# that file instead of failing obscurely at mkfs time.
|
||||
- name: Provision the data disks (Longhorn store, etcd data dir)
|
||||
ansible.builtin.include_tasks:
|
||||
file: disks.yml
|
||||
apply:
|
||||
tags: [disks]
|
||||
tags: [disks]
|
||||
|
||||
- name: Disable swap now
|
||||
|
||||
Reference in new issue
Block a user