jakman85andClaude Opus 5
c1e36d9353
The claim guard builds node_baseline_lh_protected_devices from ansible_mounts plus `swapon --show`. With swap.target masked, a baselined node has no active swap, so that half of the list contributes nothing — which reads like a guard that quietly stopped guarding. It is not: the claim candidate comes from whatever is mounted at node_baseline_longhorn_claim_mountpoint, and a swap partition is neither mounted nor ext4, so it could never be the candidate. Say so where the next reader will ask. Co-Authored-By: Claude Opus 5 <[email protected]>
652 lines
38 KiB
YAML
652 lines
38 KiB
YAML
---
|
|
# =====================================================================================
|
|
# node_baseline — DATA DISKS. Every task here carries the `disks` tag (applied once by
|
|
# the `include_tasks` in main.yml), so `--tags packages` still skips the whole file.
|
|
# Split into its own file rather than living in main.yml because it is the one part of
|
|
# the role that DESTRUCTIVELY claims storage, and it deserves to be read on its own.
|
|
#
|
|
# TWO ENVIRONMENTS, TWO SHAPES — and the shape is DECLARED, never inferred:
|
|
#
|
|
# dev (group_vars/all.yml, role defaults)
|
|
# node_baseline_longhorn_source: device -> mkfs /dev/vdb, mount LABEL=longhorn
|
|
# at /var/lib/longhorn (== dataPath)
|
|
# node_baseline_etcd_source: device -> mkfs /dev/vdc, mount LABEL=k3s-etcd
|
|
# k3s_data_dir: (k3s default, /var/lib/rancher/k3s)
|
|
# THREE devices: OS, Longhorn, etcd. Two tofu-provisioned virtio disks exist for
|
|
# exactly this purpose. NOTHING about dev changes in the 2026-09-04 revision.
|
|
#
|
|
# prod (group_vars/k3s_prod/vars.yml — operator ruling, PROD-PROVISIONING.md §2)
|
|
# node_baseline_longhorn_source: claim-mountpoint (claim mountpoint /srv)
|
|
# node_baseline_data_mount: /data (LABEL=k3s-data)
|
|
# node_baseline_etcd_source: shared-data-partition
|
|
# k3s_data_dir: /data/k3s
|
|
# ONE physical disk per node, already fully partitioned by the Debian installer. The
|
|
# ~424 GB `/srv` partition is claimed and mounted at a NEUTRAL mountpoint, then split
|
|
# between TWO tenants: /data/longhorn (Longhorn replicas) + /data/k3s (the whole k3s
|
|
# data dir: etcd AND the containerd image store).
|
|
#
|
|
# WHY THE PARTITION IS NO LONGER LONGHORN-ONLY (revised 2026-09-04, PROD-PROVISIONING.md
|
|
# §2). Prod's `/var` is a separate partition of just 11.8 GiB. k3s's default data dir puts
|
|
# BOTH etcd's WAL and the containerd image store on it. Measured on the fully-deployed dev
|
|
# cluster, containerd alone is 21-25 GB per node and /var totals 58-70 GB — so prod would
|
|
# have filled /var during its FIRST image pulls, and a full /var means etcd cannot fsync its
|
|
# WAL. That is cluster death, not degradation. Moving k3s's data dir onto the 424 GB
|
|
# partition is the fix; the partition therefore serves two tenants, and neither its
|
|
# mountpoint nor its filesystem label may name only one of them.
|
|
#
|
|
# WHY /data AND LABEL k3s-data.
|
|
# * `/data` — short, neutral, tenant-agnostic, and NOT under /var (the whole point).
|
|
# Deliberately NOT `/srv`: /srv is the CLAIM mountpoint this role discovers the device
|
|
# at, and the idempotency contract is that /srv is gone from /proc/mounts and
|
|
# /etc/fstab once the claim has happened. Re-using it as the destination would make
|
|
# "claimed" and "not yet claimed" indistinguishable.
|
|
# * label `k3s-data`, deliberately NOT the bare `data`: guard #4 in the assert wall below
|
|
# accepts a partition whose label is empty OR ALREADY OURS, and its own comment names
|
|
# `data` as an example of somebody else's named volume. Labelling ours `data` would make
|
|
# a stranger's `data` volume look already-claimed and weaken that guard. `k3s-data` is
|
|
# cluster-scoped rather than tenant-scoped and matches the existing `k3s-etcd` label
|
|
# convention in this repo.
|
|
#
|
|
# WHY A "SOURCE" ENUM INSTEAD OF A BLANK VARIABLE. "etcd has no dedicated disk" and
|
|
# "somebody forgot to set node_baseline_etcd_device" look identical if the signal is an
|
|
# empty string, and the second one must never silently succeed. `root-filesystem` and
|
|
# `shared-data-partition` are explicit, greppable statements of intent that the assert
|
|
# below accepts; an unset or placeholder device is still a hard failure. The two differ in
|
|
# a way that matters: `root-filesystem` says etcd is on the OS filesystem, and
|
|
# `shared-data-partition` says it is on the claimed data partition. Prod was the FORMER
|
|
# until 2026-09-04 and is now the LATTER — and the role ASSERTS which one is true rather
|
|
# than printing a claim it never checked (see the etcd section).
|
|
#
|
|
# WHY DISCOVERY INSTEAD OF A PER-HOST DEVICE PATH. The prod partition is /dev/sda5 on
|
|
# nodes 1 and 3 (SATA) but /dev/nvme0n1p5 on node 2 (NVMe), so no single group value is
|
|
# correct — and per-host `host_vars` would encode a *guess about hardware* into inventory
|
|
# that goes stale the moment a node is re-imaged, a disk is replaced, or a fourth node
|
|
# arrives with a different controller. Worse, a fleet-wide `/dev/sda5` would eventually
|
|
# point at the wrong partition on some future node and mkfs it. So the role asks the NODE
|
|
# which block device is mounted at the claim mountpoint, and then proves that answer is
|
|
# safe (see the assert wall below) before touching it. The inventory declares INTENT
|
|
# ("the partition currently mounted at /srv"); the node supplies the FACT.
|
|
# =====================================================================================
|
|
|
|
# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet declared
|
|
# its data-disk layout. Two failure shapes are caught:
|
|
# * an undeclared/typo'd source enum, and
|
|
# * source: device with a device path that is a PLACEHOLDER rather than a real device.
|
|
# group_vars/k3s_prod/vars.yml deliberately carries `/dev/NOT-A-DEVICE-…` for both device
|
|
# vars now that prod uses neither: if anyone ever flips a source back to `device` without
|
|
# also supplying a real path, they get this assert instead of dev's /dev/vdb (which does
|
|
# not exist on bare metal — a confusing failure that looks like a wiring bug). `BLOCKED`
|
|
# stays in the token list because that was the pre-decision placeholder and may still be
|
|
# carried by other environments/branches.
|
|
- name: Assert this environment's data-disk layout has actually been declared
|
|
ansible.builtin.assert:
|
|
that:
|
|
- node_baseline_longhorn_source in ['device', 'claim-mountpoint']
|
|
- node_baseline_etcd_source in ['device', 'root-filesystem', 'shared-data-partition']
|
|
# `shared-data-partition` is only coherent when there IS a claimed data partition to
|
|
# share, and only when the k3s data dir actually sits INSIDE it. Catch the
|
|
# contradiction here, in inventory terms, rather than three sections later when etcd
|
|
# has silently landed back on the OS filesystem.
|
|
- >-
|
|
node_baseline_etcd_source != 'shared-data-partition'
|
|
or node_baseline_longhorn_source == 'claim-mountpoint'
|
|
- >-
|
|
node_baseline_etcd_source != 'shared-data-partition'
|
|
or node_baseline_k3s_data_dir.startswith(node_baseline_data_mount ~ '/')
|
|
# The transition knobs are a matched PAIR: a retired label with no retired mountpoint
|
|
# (or vice versa) cannot describe a state to migrate from.
|
|
- >-
|
|
(node_baseline_data_retired_fs_label | length > 0)
|
|
== (node_baseline_data_retired_mount | length > 0)
|
|
- >-
|
|
node_baseline_longhorn_source != 'device'
|
|
or not (node_baseline_longhorn_device is search('BLOCKED|NOT-A-DEVICE'))
|
|
- >-
|
|
node_baseline_etcd_source != 'device'
|
|
or not (node_baseline_etcd_device is search('BLOCKED|NOT-A-DEVICE'))
|
|
- >-
|
|
node_baseline_longhorn_source != 'claim-mountpoint'
|
|
or (node_baseline_longhorn_claim_mountpoint | length) > 0
|
|
fail_msg: >-
|
|
This host's data-disk layout is not declared.
|
|
longhorn_source={{ node_baseline_longhorn_source }}
|
|
(device={{ node_baseline_longhorn_device }},
|
|
claim_mountpoint={{ node_baseline_longhorn_claim_mountpoint | default('') }}),
|
|
etcd_source={{ node_baseline_etcd_source }}
|
|
(device={{ node_baseline_etcd_device }}).
|
|
Valid Longhorn sources: `device` (a dedicated block device, dev's shape) or
|
|
`claim-mountpoint` (re-purpose the partition already mounted at
|
|
node_baseline_longhorn_claim_mountpoint, prod's shape). Valid etcd sources:
|
|
`device`, `root-filesystem` (etcd on the OS filesystem, no claimed partition under
|
|
it) or `shared-data-partition` (etcd inside node_baseline_k3s_data_dir on the CLAIMED
|
|
data partition — prod's shape since 2026-09-04, which additionally requires
|
|
longhorn_source `claim-mountpoint` and a k3s data dir under
|
|
node_baseline_data_mount={{ node_baseline_data_mount }};
|
|
k3s_data_dir={{ node_baseline_k3s_data_dir }}). Every one of these is an EXPLICIT
|
|
operator decision recorded in infra/PROD-PROVISIONING.md §2, not an omission.
|
|
Declare it in the environment's group_vars and record the reasoning in
|
|
§2 in the SAME change. To install ONLY the package floor meanwhile, re-run with
|
|
`--tags packages`, which skips every disk task including this assert.
|
|
quiet: true
|
|
|
|
# =====================================================================================
|
|
# THE DATA PARTITION (prod: Longhorn store + k3s data dir; dev: Longhorn store only)
|
|
# =====================================================================================
|
|
|
|
# THE IDEMPOTENCY PIVOT for the claim path, and the reason a second run is a no-op
|
|
# rather than a reformat. Once the claim has happened the partition carries the ext4
|
|
# label node_baseline_data_fs_label and is mounted at node_baseline_data_mount — the claim
|
|
# mountpoint (/srv) is gone from both /proc/mounts and /etc/fstab, so "discover what is
|
|
# mounted at /srv" would find nothing on run 2. Asking `blkid -L <label>` instead answers
|
|
# "has this node already been claimed?" from the durable, on-disk evidence: rc 0 + a device
|
|
# path means converged, rc != 0 means the claim has not been made. Everything destructive
|
|
# below is gated on rc != 0.
|
|
- name: "Data partition: look for a filesystem already carrying the data-partition label"
|
|
ansible.builtin.command: "blkid -L {{ node_baseline_data_fs_label }}"
|
|
register: node_baseline_lh_labelled
|
|
changed_when: false
|
|
failed_when: false
|
|
check_mode: false
|
|
|
|
# --- The TRANSITION (2026-09-04 layout revision; one-time, then this whole block is dead) --
|
|
# WHY THIS EXISTS. The idempotency pivot above is "does the CURRENT label exist?". When the
|
|
# label ITSELF changes, an already-claimed node answers "no" and the greenfield claim path
|
|
# takes over — which then fails closed at "the claim mountpoint must actually be a
|
|
# mountpoint", because /srv was consumed by the previous claim. Fail-closed is correct
|
|
# behaviour but it is not convergence, and the prod nodes are in exactly that state today:
|
|
# claimed under LABEL=longhorn at /var/lib/longhorn by the previous version of this role.
|
|
# So the role must handle the TRANSITION explicitly instead of pretending prod is greenfield.
|
|
#
|
|
# WHAT IT DOES: unmount the retired mountpoint -> drop its fstab entry -> re-stamp the
|
|
# partition with the new label. It never touches the DATA and never reformats: e2label is a
|
|
# metadata write, exactly as in the claim path below. The common mount task then mounts
|
|
# LABEL=<new> at the new mountpoint, and the node is converged. On every subsequent run
|
|
# `blkid -L <new label>` succeeds and none of this is reachable.
|
|
#
|
|
# THREE INDEPENDENT REASONS THE LIVE DEV CLUSTER CANNOT ENTER THIS PATH — it has 18 Longhorn
|
|
# volumes whose replicas live on the partition labelled `longhorn` at /var/lib/longhorn, and
|
|
# unmounting it would orphan every one of them:
|
|
# 1. node_baseline_data_retired_fs_label defaults to "" and is set ONLY in
|
|
# group_vars/k3s_prod/vars.yml. No retired label declared => no transition, ever.
|
|
# 2. Gated on node_baseline_longhorn_source == 'claim-mountpoint'. Dev is `device`, so
|
|
# the same structural gate that keeps dev out of the claim path keeps it out of this one.
|
|
# 3. The emptiness assert below. Even if 1 and 2 were both defeated, a partition with
|
|
# replica directories on it fails closed and nothing is unmounted.
|
|
# Dev's label and mountpoint are also simply NOT CHANGING, so there is nothing to migrate.
|
|
- name: "Data partition (transition): look for a filesystem still carrying the RETIRED label"
|
|
ansible.builtin.command: "blkid -L {{ node_baseline_data_retired_fs_label }}"
|
|
register: node_baseline_lh_retired_labelled
|
|
changed_when: false
|
|
failed_when: false
|
|
check_mode: false
|
|
when:
|
|
- node_baseline_longhorn_source == 'claim-mountpoint'
|
|
- node_baseline_data_retired_fs_label | length > 0
|
|
- node_baseline_lh_labelled.rc != 0
|
|
|
|
- name: "Data partition: decide between TRANSITION, CLAIM, and already-converged"
|
|
ansible.builtin.set_fact:
|
|
# A node claimed under the RETIRED label. Mutually exclusive with the claim below.
|
|
node_baseline_lh_transition_needed: >-
|
|
{{ node_baseline_longhorn_source == 'claim-mountpoint'
|
|
and node_baseline_data_retired_fs_label | length > 0
|
|
and node_baseline_lh_labelled.rc != 0
|
|
and (node_baseline_lh_retired_labelled.rc | default(1)) == 0 }}
|
|
# True only for a claim-mountpoint environment that has NOT been claimed under EITHER
|
|
# label. A `device` environment (dev) can never enter the claim path at all — that is
|
|
# the structural guarantee that no generalisation of prod's behaviour can reach the
|
|
# LIVE dev nodes and their /dev/vdb Longhorn replicas.
|
|
node_baseline_lh_claim_needed: >-
|
|
{{ node_baseline_longhorn_source == 'claim-mountpoint'
|
|
and node_baseline_lh_labelled.rc != 0
|
|
and (node_baseline_lh_retired_labelled.rc | default(1)) != 0 }}
|
|
|
|
- name: "Data partition (transition): resolve what is mounted at the retired mountpoint"
|
|
ansible.builtin.command: >-
|
|
findmnt --noheadings --first-only --output SOURCE
|
|
--mountpoint {{ node_baseline_data_retired_mount }}
|
|
register: node_baseline_lh_retired_findmnt
|
|
changed_when: false
|
|
failed_when: false
|
|
check_mode: false
|
|
when: node_baseline_lh_transition_needed | bool
|
|
|
|
# Read the retired mountpoint's top level, same contract as the claim path: `lost+found` is
|
|
# not data, hidden files cannot hide. THIS is the assert that makes the transition safe —
|
|
# an unmount of a populated Longhorn store would orphan every replica on the node.
|
|
- name: "Data partition (transition): read the contents of the retired mountpoint"
|
|
ansible.builtin.find:
|
|
paths: "{{ node_baseline_data_retired_mount }}"
|
|
file_type: any
|
|
hidden: true
|
|
recurse: false
|
|
excludes: ['lost+found']
|
|
register: node_baseline_lh_retired_contents
|
|
when: node_baseline_lh_transition_needed | bool
|
|
|
|
- name: "Data partition (transition): prove this node is safe to re-label"
|
|
ansible.builtin.assert:
|
|
that:
|
|
# 1. The retired mountpoint is STILL A MOUNT, and it is backed by the very device the
|
|
# retired label names. The only state this role knows how to transition is the one
|
|
# the previous version of it produced; anything else is an operator decision.
|
|
- (node_baseline_lh_retired_findmnt.rc | default(1)) == 0
|
|
- >-
|
|
(node_baseline_lh_retired_findmnt.stdout | default('') | trim)
|
|
== (node_baseline_lh_retired_labelled.stdout | default('') | trim)
|
|
# 2. It is EMPTY. Nothing is being orphaned. A populated store means either this is
|
|
# not the node we think it is, or a tenant has already written data — in both cases
|
|
# unmounting and re-labelling under it is destructive and the role must stop.
|
|
- (node_baseline_lh_retired_contents.files | length) == 0
|
|
fail_msg: >-
|
|
REFUSING to transition this node's data partition from
|
|
LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
|
|
to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}.
|
|
device by retired label='{{ node_baseline_lh_retired_labelled.stdout | default('') | trim }}',
|
|
device mounted at the retired mountpoint='{{ node_baseline_lh_retired_findmnt.stdout | default('') | trim }}'
|
|
(findmnt rc={{ node_baseline_lh_retired_findmnt.rc | default('n/a') }}),
|
|
non-lost+found entries={{ node_baseline_lh_retired_contents.files | default([]) | length }}
|
|
{{ node_baseline_lh_retired_contents.files | default([]) | map(attribute='path') | list }}.
|
|
If the partition holds DATA, this transition would orphan it — Longhorn replicas are
|
|
not recoverable from an unmounted, re-labelled filesystem. Migrate the data
|
|
deliberately, or drop node_baseline_data_retired_fs_label and leave the node alone.
|
|
Do NOT relax this assert.
|
|
quiet: true
|
|
when: node_baseline_lh_transition_needed | bool
|
|
|
|
- name: "Data partition (transition): report exactly what is about to change"
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
Transitioning {{ node_baseline_lh_retired_labelled.stdout | trim }} (verified empty)
|
|
from LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
|
|
to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}:
|
|
unmount -> drop its fstab entry -> relabel -> re-mount at the new path.
|
|
No filesystem is created and no data is moved.
|
|
when: node_baseline_lh_transition_needed | bool
|
|
|
|
# e2label refuses to run on a mounted filesystem, and the fstab entry must go in the same
|
|
# breath or the next boot re-mounts the partition at the RETIRED path underneath the new one.
|
|
- name: "Data partition (transition): unmount the retired mountpoint"
|
|
ansible.posix.mount:
|
|
path: "{{ node_baseline_data_retired_mount }}"
|
|
state: unmounted
|
|
when: node_baseline_lh_transition_needed | bool
|
|
|
|
- name: "Data partition (transition): remove the retired mountpoint's /etc/fstab entry"
|
|
ansible.posix.mount:
|
|
path: "{{ node_baseline_data_retired_mount }}"
|
|
state: absent_from_fstab
|
|
backup: true
|
|
when: node_baseline_lh_transition_needed | bool
|
|
|
|
- name: "Data partition (transition): re-stamp the partition with the new filesystem label"
|
|
ansible.builtin.command: >-
|
|
e2label {{ node_baseline_lh_retired_labelled.stdout | trim }}
|
|
{{ node_baseline_data_fs_label }}
|
|
changed_when: true
|
|
when: node_baseline_lh_transition_needed | bool
|
|
|
|
# --- The claim (prod, first run only) -------------------------------------------------
|
|
# Ask the node, do not assume. `--mountpoint` (not `--target`) matches ONLY an exact
|
|
# mountpoint: if /srv is not itself a mount, findmnt exits 1 rather than silently
|
|
# returning the device behind `/` — which is precisely the mistake that would hand the
|
|
# root partition to e2label.
|
|
- name: "Longhorn (claim): resolve the block device mounted at the claim mountpoint"
|
|
ansible.builtin.command: >-
|
|
findmnt --noheadings --first-only --output SOURCE
|
|
--mountpoint {{ node_baseline_longhorn_claim_mountpoint }}
|
|
register: node_baseline_lh_claim_findmnt
|
|
changed_when: false
|
|
failed_when: false
|
|
check_mode: false
|
|
when: node_baseline_lh_claim_needed | bool
|
|
|
|
- name: "Longhorn (claim): the claim mountpoint must actually be a mountpoint"
|
|
ansible.builtin.assert:
|
|
that:
|
|
- node_baseline_lh_claim_findmnt.rc == 0
|
|
- (node_baseline_lh_claim_findmnt.stdout | trim) is match('^/dev/')
|
|
fail_msg: >-
|
|
node_baseline_longhorn_source is `claim-mountpoint` and no filesystem is claimed
|
|
yet, but nothing is mounted at {{ node_baseline_longhorn_claim_mountpoint }} on
|
|
this node (findmnt rc={{ node_baseline_lh_claim_findmnt.rc | default('n/a') }},
|
|
stdout='{{ node_baseline_lh_claim_findmnt.stdout | default('') | trim }}').
|
|
The role will NOT go looking for "a big partition" to claim instead. Either the
|
|
node is partitioned differently from the fleet — in which case fix the node or give
|
|
this host its own declaration — or the claim already happened and the label was
|
|
removed by hand.
|
|
quiet: true
|
|
when: node_baseline_lh_claim_needed | bool
|
|
|
|
- name: "Longhorn (claim): inspect the candidate partition"
|
|
ansible.builtin.command: >-
|
|
lsblk --json --nodeps --bytes
|
|
--output PATH,TYPE,FSTYPE,LABEL,SIZE
|
|
{{ node_baseline_lh_claim_findmnt.stdout | trim }}
|
|
register: node_baseline_lh_claim_lsblk
|
|
changed_when: false
|
|
check_mode: false
|
|
when: node_baseline_lh_claim_needed | bool
|
|
|
|
# The devices this role must never, under any circumstances, hand to e2label. Built from
|
|
# gathered facts (`gather_facts: true` in the play) plus the live swap list, so it is the
|
|
# NODE's account of what is load-bearing, not a hardcoded list of partition names.
|
|
#
|
|
# EXPECT THIS LIST TO BE EMPTY. Since the swap tasks in tasks/main.yml mask swap.target,
|
|
# a baselined node has no active swap for `swapon` to report, so the swap half of
|
|
# node_baseline_lh_protected_devices below contributes nothing. That is not a weakened
|
|
# guard: the claim CANDIDATE is derived from whatever is mounted at
|
|
# node_baseline_longhorn_claim_mountpoint, and a swap partition is neither mounted nor an
|
|
# ext4 filesystem, so it can never be the candidate in the first place. The task stays
|
|
# because the list must be the node's own account of what is load-bearing — including on
|
|
# a node reached before baseline has run, or one an operator has deliberately swapped on.
|
|
- name: "Longhorn (claim): list the devices currently backing active swap"
|
|
ansible.builtin.command: swapon --noheadings --show=NAME
|
|
register: node_baseline_lh_swapdevs
|
|
changed_when: false
|
|
failed_when: false
|
|
check_mode: false
|
|
when: node_baseline_lh_claim_needed | bool
|
|
|
|
# Read the mountpoint's top level. `excludes: lost+found` because every freshly-mkfs'd
|
|
# ext4 has one and it is not data. hidden: true so a dotfile cannot hide from the check.
|
|
- name: "Longhorn (claim): read the contents of the mountpoint to be claimed"
|
|
ansible.builtin.find:
|
|
paths: "{{ node_baseline_longhorn_claim_mountpoint }}"
|
|
file_type: any
|
|
hidden: true
|
|
recurse: false
|
|
excludes: ['lost+found']
|
|
register: node_baseline_lh_claim_contents
|
|
when: node_baseline_lh_claim_needed | bool
|
|
|
|
# THE GUARD WALL. Five independent conditions, every one of which must hold before the
|
|
# role modifies a filesystem it did not create. Any single failure stops the play with
|
|
# the evidence in the message — nothing is unmounted, relabelled or mounted first.
|
|
- name: "Longhorn (claim): prove the candidate partition is safe to claim"
|
|
ansible.builtin.assert:
|
|
that:
|
|
# 1. It is a real block device — a partition or a whole disk. Catches a claim
|
|
# mountpoint that is actually tmpfs, an NFS/CIFS export, or a bind mount, all of
|
|
# which would have produced a non-/dev SOURCE or an lsblk row of another type.
|
|
- node_baseline_lh_candidate.type in ['part', 'disk']
|
|
# 2. It is not backing anything the OS needs. Derived from ansible_mounts + swapon,
|
|
# so a differently-partitioned node is caught by its own facts.
|
|
- node_baseline_lh_candidate.path not in node_baseline_lh_protected_devices
|
|
# 3. It already holds ext4. The claim path RELABELS rather than reformats (see
|
|
# below), which is only meaningful for the filesystem type we are about to
|
|
# declare in fstab. Anything else is an operator decision, not a role's.
|
|
- node_baseline_lh_candidate.fstype == 'ext4'
|
|
# 4. It carries no filesystem label, or already carries ours. A partition labelled
|
|
# `boot`, `data`, `backup`… is somebody's named volume; stamping `longhorn` over
|
|
# it would make it unfindable by its own fstab entry.
|
|
- (node_baseline_lh_candidate.label | default('', true)) in ['', node_baseline_data_fs_label]
|
|
# 5. It is big enough to be the partition we meant. A data partition of a few GB is
|
|
# not "small", it is the wrong partition — and on prod it must hold BOTH tenants.
|
|
- (node_baseline_lh_candidate.size | int) >= (node_baseline_longhorn_min_size_gb | int) * 1024 * 1024 * 1024
|
|
# 6. It is EMPTY. The operator's premise is that prod's /srv holds nothing but the
|
|
# ext4 5% root reserve; this is where that premise is verified per node instead
|
|
# of trusted. If any node has real data on it, the play stops here — the role
|
|
# never wipes data it cannot account for.
|
|
- (node_baseline_lh_claim_contents.files | length) == 0
|
|
fail_msg: >-
|
|
REFUSING to claim {{ node_baseline_lh_candidate.path }}
|
|
(mounted at {{ node_baseline_longhorn_claim_mountpoint }}) for the Longhorn store.
|
|
type={{ node_baseline_lh_candidate.type }},
|
|
fstype={{ node_baseline_lh_candidate.fstype }},
|
|
label='{{ node_baseline_lh_candidate.label | default('', true) }}',
|
|
size={{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB
|
|
(minimum {{ node_baseline_longhorn_min_size_gb }} GiB),
|
|
non-lost+found entries={{ node_baseline_lh_claim_contents.files | length }}
|
|
{{ node_baseline_lh_claim_contents.files | map(attribute='path') | list }},
|
|
devices this node needs and that must never be claimed:
|
|
{{ node_baseline_lh_protected_devices }}.
|
|
This is the destructive step of the role and it fails CLOSED. If the partition
|
|
holds data, back it up or pick a different one; do NOT relax this assert.
|
|
quiet: true
|
|
vars:
|
|
node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
|
|
# The node's OWN account of what it cannot afford to lose: the devices behind the
|
|
# mounts in node_baseline_protected_mounts (from gathered facts, so a node
|
|
# partitioned differently is judged on its own layout) plus every active swap device.
|
|
# Built here rather than hardcoded so no partition-name list can go stale.
|
|
node_baseline_lh_protected_devices: >-
|
|
{{ (ansible_facts.mounts | default([])
|
|
| selectattr('mount', 'in', node_baseline_protected_mounts)
|
|
| map(attribute='device') | list)
|
|
+ (node_baseline_lh_swapdevs.stdout_lines | default([]) | map('trim') | select | list) }}
|
|
when: node_baseline_lh_claim_needed | bool
|
|
|
|
- name: "Longhorn (claim): report exactly what is about to be claimed"
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
Claiming {{ node_baseline_lh_candidate.path }}
|
|
({{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB ext4,
|
|
currently mounted at {{ node_baseline_longhorn_claim_mountpoint }}, verified empty)
|
|
as the data partition: unmount -> drop its fstab entry ->
|
|
relabel LABEL={{ node_baseline_data_fs_label }} ->
|
|
mount at {{ node_baseline_data_mount }}.
|
|
vars:
|
|
node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
|
|
when: node_baseline_lh_claim_needed | bool
|
|
|
|
# --- Why the role must unmount and relabel, not just point fstab at the device ---------
|
|
# The mount task below mounts by `LABEL=`, deliberately: a label survives /dev/sda5 ->
|
|
# /dev/nvme0n1p5 renaming across the fleet and across a controller change, which a device
|
|
# path does not. The installer-created /srv filesystem has NO label (fstab pins it by
|
|
# UUID), so it must be given one. e2label refuses to run on a mounted filesystem, hence
|
|
# the unmount first — and the fstab entry must go in the same breath, or the next boot
|
|
# re-mounts the same partition at /srv underneath Longhorn.
|
|
#
|
|
# RELABEL, NOT REFORMAT. The partition is already ext4 and already verified empty, so
|
|
# mkfs would buy nothing and would destroy anything the emptiness check got wrong.
|
|
# e2label is a metadata write: if some assert above was too permissive, the worst case is
|
|
# a renamed filesystem, not a destroyed one. (This is also why `community.general.filesystem`
|
|
# cannot do this job: it declines to touch a device that already has a filesystem, so it
|
|
# would silently no-op and then the LABEL= mount would fail with no useful message.)
|
|
- name: "Longhorn (claim): unmount the claim mountpoint"
|
|
ansible.posix.mount:
|
|
path: "{{ node_baseline_longhorn_claim_mountpoint }}"
|
|
state: unmounted
|
|
when: node_baseline_lh_claim_needed | bool
|
|
|
|
- name: "Longhorn (claim): remove the claim mountpoint's /etc/fstab entry"
|
|
ansible.posix.mount:
|
|
path: "{{ node_baseline_longhorn_claim_mountpoint }}"
|
|
state: absent_from_fstab
|
|
backup: true
|
|
when: node_baseline_lh_claim_needed | bool
|
|
|
|
- name: "Longhorn (claim): stamp the data-partition filesystem label onto the partition"
|
|
ansible.builtin.command: >-
|
|
e2label {{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0].path }}
|
|
{{ node_baseline_data_fs_label }}
|
|
changed_when: true
|
|
when: node_baseline_lh_claim_needed | bool
|
|
|
|
# --- The dedicated-device path (dev) --------------------------------------------------
|
|
# Unchanged behaviour for `source: device`: data-safe, because community.general.filesystem
|
|
# never reformats a device that already carries a filesystem.
|
|
- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
|
|
community.general.filesystem:
|
|
fstype: ext4
|
|
dev: "{{ node_baseline_longhorn_device }}"
|
|
opts: "-L {{ node_baseline_data_fs_label }}"
|
|
when: node_baseline_longhorn_source == 'device'
|
|
|
|
# --- Common to ALL paths (device, claim, transition) ----------------------------------
|
|
# Mounted by LABEL for every shape, so dev and prod share ONE fstab convention and one
|
|
# mount task. Idempotent: on a converged node this reports ok, which is what makes a
|
|
# re-run a no-op rather than a reformat. It is also what completes the transition above:
|
|
# the retired mountpoint is gone from fstab and the partition now answers to the new label,
|
|
# so this task mounts it at the new path.
|
|
- name: Mount the data partition
|
|
ansible.posix.mount:
|
|
path: "{{ node_baseline_data_mount }}"
|
|
src: "LABEL={{ node_baseline_data_fs_label }}"
|
|
fstype: ext4
|
|
opts: "defaults,nofail,x-systemd.device-timeout=5s"
|
|
state: mounted
|
|
|
|
# --- The per-tenant subdirectories ---------------------------------------------------
|
|
# WHY THIS IS CONDITIONAL RATHER THAN UNCONDITIONAL. On prod the partition serves two
|
|
# tenants and each needs its own subdirectory carved out of it. On DEV neither path is a
|
|
# subdirectory of the mount: Longhorn's dataPath IS the mount root, and k3s's data dir is
|
|
# /var/lib/rancher/k3s on the OS filesystem, owned and created by a LIVE k3s. Creating
|
|
# either there would at best be a no-op and at worst re-mode the running dev cluster's data
|
|
# dir, so the role only creates paths that are STRICT DESCENDANTS of the partition it
|
|
# mounted — i.e. only the directories it is actually responsible for. That test is derived
|
|
# from the paths themselves, so no environment has to remember to switch it off.
|
|
#
|
|
# MODES. Longhorn's dataPath keeps 0755, matching what mkfs gives dev's mount root and what
|
|
# the Longhorn manager expects to find. k3s's data dir is 0700 from the start: it holds the
|
|
# etcd database, the node token and the containerd image store, and k3s would tighten it to
|
|
# 0700 itself on first start — doing it here means it is never briefly world-readable.
|
|
- name: Create the per-tenant data directories on the data partition
|
|
ansible.builtin.file:
|
|
path: "{{ item.path }}"
|
|
state: directory
|
|
owner: root
|
|
group: root
|
|
mode: "{{ item.mode }}"
|
|
loop:
|
|
- { path: "{{ node_baseline_longhorn_data_path }}", mode: "0755", tenant: "Longhorn replicas" }
|
|
- { path: "{{ node_baseline_k3s_data_dir }}", mode: "0700", tenant: "k3s data dir (etcd + containerd)" }
|
|
loop_control:
|
|
label: "{{ item.path }} ({{ item.tenant }})"
|
|
when: item.path.startswith(node_baseline_data_mount ~ '/')
|
|
|
|
# =====================================================================================
|
|
# ETCD DATA DIR
|
|
# =====================================================================================
|
|
|
|
# --- The dedicated-device path (dev) --------------------------------------------------
|
|
# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
|
|
# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
|
|
# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
|
|
# a disk that already carries a filesystem, so re-running against an already-migrated
|
|
# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
|
|
# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
|
|
# role would mount over a populated directory.
|
|
- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
|
|
community.general.filesystem:
|
|
fstype: ext4
|
|
dev: "{{ node_baseline_etcd_device }}"
|
|
opts: "-L {{ node_baseline_etcd_fs_label }}"
|
|
when: node_baseline_etcd_source == 'device'
|
|
|
|
- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
|
|
ansible.posix.mount:
|
|
path: "{{ node_baseline_etcd_mount }}"
|
|
src: "LABEL={{ node_baseline_etcd_fs_label }}"
|
|
fstype: ext4
|
|
# noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
|
|
# missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
|
|
opts: "defaults,noatime"
|
|
dump: "0"
|
|
passno: "2"
|
|
state: mounted
|
|
when: node_baseline_etcd_source == 'device'
|
|
|
|
# --- The no-dedicated-device paths (prod) ---------------------------------------------
|
|
# Prod has ONE physical disk per node, already fully partitioned, with no free slot verified
|
|
# populated — so etcd gets no device of its own. That is an operator decision
|
|
# (PROD-PROVISIONING.md §2, 2026-09-03), and the shared-device I/O contention it accepts is
|
|
# UNCHANGED by the 2026-09-04 layout revision: dev's etcd/Longhorn contention incident was
|
|
# on a spinning HDD, and every prod node is SSD or NVMe, where concurrent rebuild I/O costs
|
|
# latency rather than collapsing it. Accepted to be MEASURED, not assumed.
|
|
#
|
|
# WHAT DID CHANGE ON 2026-09-04 is WHICH FILESYSTEM etcd shares. It used to be the OS
|
|
# filesystem (`root-filesystem`), which on these nodes means /var — a separate partition of
|
|
# 11.8 GiB carrying the containerd image store too. Measured on dev, containerd alone is
|
|
# 21-25 GB per node, so prod would have filled /var during its first image pulls and etcd
|
|
# could not have fsynced its WAL. So k3s's whole data dir moved onto the claimed 424 GB
|
|
# partition and etcd moved with it: `shared-data-partition`. Both enum values are still
|
|
# supported, and the role now PROVES which one is true instead of printing a claim.
|
|
#
|
|
# There is nothing to CREATE here for either shape — the data-partition section above
|
|
# already created node_baseline_k3s_data_dir, and k3s creates server/db inside it — so the
|
|
# role's job is to verify the assumption rather than act on it.
|
|
- name: "etcd (no dedicated device): check for anything already mounted at the etcd data dir"
|
|
ansible.builtin.command: >-
|
|
findmnt --noheadings --first-only --output SOURCE
|
|
--mountpoint {{ node_baseline_etcd_mount }}
|
|
register: node_baseline_etcd_existing_mount
|
|
changed_when: false
|
|
failed_when: false
|
|
check_mode: false
|
|
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
|
|
|
|
- name: "etcd (no dedicated device): assert no dedicated device is already mounted there"
|
|
ansible.builtin.assert:
|
|
that:
|
|
- node_baseline_etcd_existing_mount.rc != 0
|
|
fail_msg: >-
|
|
node_baseline_etcd_source is `{{ node_baseline_etcd_source }}` (etcd deliberately has
|
|
NO dedicated device — PROD-PROVISIONING.md §2) but {{ node_baseline_etcd_mount }} is
|
|
ALREADY a separate mount, backed by
|
|
{{ node_baseline_etcd_existing_mount.stdout | default('') | trim }}. That is a
|
|
contradiction the role will not resolve on its own: either this node was previously
|
|
built with a dedicated etcd device (in which case unmount it deliberately, after
|
|
accounting for the etcd data on it) or the declaration is wrong. Refusing to leave
|
|
the node in a state where the declared layout and the real one disagree.
|
|
quiet: true
|
|
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
|
|
|
|
# Evidence, not decoration: §2 records which filesystem etcd shares, and this establishes
|
|
# WHICH one per node so the claim is auditable from a run log. `findmnt --target` needs an
|
|
# existing path and k3s has not created server/db yet, so walk up to the deepest ancestor
|
|
# that does exist — the same filesystem the data dir will be created on. TARGET is captured
|
|
# separately from the human-readable line because the assert below judges on it.
|
|
- name: "etcd (no dedicated device): resolve the filesystem etcd will actually live on"
|
|
ansible.builtin.shell: |
|
|
set -eu
|
|
p={{ node_baseline_etcd_mount | quote }}
|
|
while [ ! -d "$p" ]; do p=$(dirname "$p"); done
|
|
findmnt --noheadings --first-only --output TARGET --target "$p"
|
|
findmnt --noheadings --first-only --output SOURCE,TARGET,SIZE,AVAIL --target "$p"
|
|
register: node_baseline_etcd_backing_fs
|
|
changed_when: false
|
|
check_mode: false
|
|
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
|
|
|
|
# THE ASSERT THAT MAKES THE WHOLE LAYOUT REVISION SELF-PROVING, and the reason
|
|
# `shared-data-partition` is a distinct enum value rather than a comment. Declaring
|
|
# k3s_data_dir: /data/k3s does NOT by itself put etcd on the data partition: if the mount
|
|
# above had silently failed, /data would be a plain directory on `/` and etcd plus 21-25 GB
|
|
# of container images would land back on the OS filesystem — the exact failure this change
|
|
# exists to prevent, and completely invisible until the disk filled. So the role checks that
|
|
# the filesystem backing the etcd data dir IS the data partition's mount, per node, on every
|
|
# run. `root-filesystem` is deliberately NOT asserted this way: it makes no claim about
|
|
# which filesystem it lands on beyond "not a dedicated one", and the debug line below is the
|
|
# whole record for that shape.
|
|
- name: "etcd (shared-data-partition): assert etcd really lands on the data partition"
|
|
ansible.builtin.assert:
|
|
that:
|
|
- >-
|
|
(node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim)
|
|
== node_baseline_data_mount
|
|
fail_msg: >-
|
|
node_baseline_etcd_source is `shared-data-partition`, which asserts that
|
|
{{ node_baseline_etcd_mount }} sits on the claimed data partition mounted at
|
|
{{ node_baseline_data_mount }} — but the filesystem actually backing it is mounted at
|
|
'{{ node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim }}'
|
|
(full: {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}).
|
|
The data partition is therefore NOT mounted where this layout requires, and k3s would
|
|
put etcd's WAL AND the containerd image store (21-25 GB measured on dev) on that
|
|
filesystem instead. On prod that is /var at 11.8 GiB: it would fill during the first
|
|
image pulls and etcd would stop being able to fsync its WAL. DO NOT install k3s on
|
|
this node until the mount is correct.
|
|
quiet: true
|
|
when: node_baseline_etcd_source == 'shared-data-partition'
|
|
|
|
- name: "etcd (no dedicated device): show which filesystem etcd will live on"
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
etcd data dir {{ node_baseline_etcd_mount }} has NO dedicated device
|
|
(source={{ node_baseline_etcd_source }}); it lands on
|
|
{{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}
|
|
(SOURCE TARGET SIZE AVAIL). Shared-device I/O contention is an accepted, to-be-
|
|
measured risk — §2.
|
|
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
|