feat(prod): kernel fix + disk layout — prod nodes ready for k3s
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
8e913fdd7b
commit
05baae90ec
16 files changed
+1782
-126
No files matched your search
@@ -1,6 +1,6 @@
|
||||
---
|
||||
# TAGS — every task here carries one of `packages`, `system`, `disks` or `dns`. An untagged
|
||||
# run is unchanged (all four run, in this order). The split exists because the PACKAGE floor
|
||||
# TAGS — every task here carries one of `packages`, `kernel`, `system`, `disks` or `dns`. An
|
||||
# untagged run is unchanged (all five run, in this order). The split exists because the PACKAGE floor
|
||||
# and the DATA-DISK work have different readiness: packages are safe on any node from
|
||||
# first contact, while the disk tasks need block devices that a new environment may not
|
||||
# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
|
||||
@@ -12,6 +12,10 @@
|
||||
# mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
|
||||
# at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
|
||||
# ansible-playbook playbooks/node-baseline.yml --tags dns
|
||||
# `kernel` (the backports-kernel block below) is separated for the same reason `packages`
|
||||
# is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
|
||||
# it must be runnable without touching a disk decision that is still open:
|
||||
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
|
||||
|
||||
- name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
|
||||
ansible.builtin.apt:
|
||||
@@ -21,6 +25,237 @@
|
||||
cache_valid_time: 3600
|
||||
tags: [packages]
|
||||
|
||||
# =====================================================================================
|
||||
# BACKPORTS KERNEL — tag `kernel` (EFI memattr; infra/PROD-PROVISIONING.md §0b and §3)
|
||||
# =====================================================================================
|
||||
# A STANDING DEVIATION from the Debian 13 baseline, encoded here so every current AND
|
||||
# FUTURE prod node gets it from provisioning instead of from someone remembering.
|
||||
#
|
||||
# WHAT IT FIXES. HP R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI
|
||||
# Memory Attributes Table. `efi_memattr_init()` in Linux < 6.13 computes
|
||||
# `tbl_size = sizeof(*tbl) + num_entries * desc_size` from two UNVALIDATED u32 fields into a
|
||||
# signed int and `memblock_reserve()`s the product; when bit 31 lands the value sign-extends
|
||||
# through phys_addr_t, `memblock_cap_size()` clamps it to PHYS_ADDR_MAX, and EVERY physical
|
||||
# address above the table is reserved. k3s-prod-2 booted with 2.6 GiB of its 16 GB, twice.
|
||||
# Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus"
|
||||
# (first released in v6.13) validates the size and rejects the table instead. It is absent
|
||||
# from v6.12 and from linux-6.12.y, has no `Cc: stable` and no `Fixes:` tag, so no automatic
|
||||
# stable feed will ever pick it up. There is NO kernel cmdline workaround —
|
||||
# `efi_memattr_init()` runs from `efi_config_parse_tables()` before any `efi=` bail-out could
|
||||
# take effect. Booting a >= 6.13 kernel is the only fix, and ALL THREE prod nodes are on the
|
||||
# same lottery — node 1 and node 3 have merely drawn safe tickets so far.
|
||||
#
|
||||
# WHY IT IS SCOPED OFF FOR DEV. `node_baseline_backports_kernel` defaults to FALSE in
|
||||
# defaults/main.yml and is set true in inventory/group_vars/k3s_prod/vars.yml only. Dev is
|
||||
# Proxmox VMs whose firmware publishes no MAT at all — genuinely unaffected — and pulling a
|
||||
# LIVE cluster onto a backports kernel as a side effect of sharing this role would be a real
|
||||
# change for no benefit. See defaults/main.yml for why this is a variable rather than a
|
||||
# `k3s-prod-*` hostname test.
|
||||
#
|
||||
# IT DOES NOT REBOOT. A new kernel is inert until the node reboots, and this role
|
||||
# deliberately will not take one: k3s-prod-2 hung on a warm reboot on 2026-09-03 — powered,
|
||||
# off the LAN for ~38 minutes, power button required (§0b "SECOND FAULT"). Rebooting is
|
||||
# therefore a SERIALIZED, SUPERVISED operator step — one node at a time, verify each returns
|
||||
# before touching the next, never node 2 first — and a role cannot enforce that. The last
|
||||
# task here reports the pending reboot loudly instead of letting a green play imply the node
|
||||
# is already fixed.
|
||||
- name: Backports kernel for the EFI memattr fix (>= 6.13, commit 8fbe4c49c0cc)
|
||||
when: node_baseline_backports_kernel | bool
|
||||
tags: [kernel]
|
||||
block:
|
||||
|
||||
# The suite name and the running release must agree, or apt resolves no candidate and
|
||||
# `state: latest` reports "ok" while leaving the node on the vulnerable 6.12 kernel —
|
||||
# a green play that fixed nothing. Fail on the mismatch instead (e.g. a node
|
||||
# dist-upgraded to forky while group_vars still says trixie-backports).
|
||||
- name: Assert the configured backports suite matches this node's Debian release
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_facts['distribution'] == 'Debian'
|
||||
- node_baseline_backports_suite == (ansible_facts['distribution_release'] ~ '-backports')
|
||||
fail_msg: >-
|
||||
node_baseline_backports_kernel is enabled but node_baseline_backports_suite
|
||||
({{ node_baseline_backports_suite }}) is not the backports suite for this node's
|
||||
release ({{ ansible_facts['distribution'] }}
|
||||
{{ ansible_facts['distribution_release'] | default('?') }}). apt would resolve no
|
||||
candidate and this play would report success while leaving the node on the
|
||||
vulnerable kernel. Fix the suite in group_vars — do not remove this assert.
|
||||
quiet: true
|
||||
|
||||
# deb822 `.sources`, not a legacy one-line `deb` entry: it is the format Debian 13
|
||||
# prefers, it carries `Signed-By` explicitly (binding this source to the archive keyring
|
||||
# rather than to whatever is in /etc/apt/trusted.gpg.d), and a `copy` of exact content is
|
||||
# idempotent in a way `apt_repository` — which rewrites and reformats — is not.
|
||||
#
|
||||
# THERE IS DELIBERATELY NO /etc/apt/preferences.d PIN, and that is not an oversight. The
|
||||
# backports Release file carries `NotAutomatic: yes` + `ButAutomaticUpgrades: yes`, which
|
||||
# apt already reads as: priority 100 so NOTHING is pulled from backports by accident, but
|
||||
# 500 for a package that was explicitly installed from there so it keeps getting updates.
|
||||
# Writing an explicit `Pin-Priority: 100` looks like belt-and-braces and is actively
|
||||
# HARMFUL — it overrides the ButAutomaticUpgrades half, and the kernel would then never
|
||||
# be upgraded again by `apt upgrade`. Enabling the source IS the whole configuration.
|
||||
- name: Enable the Debian backports apt source (deb822)
|
||||
ansible.builtin.copy:
|
||||
dest: "/etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
|
||||
# A DELIBERATE, STANDING deviation from the Debian 13 baseline, enabled ONLY so this
|
||||
# machine can boot a kernel >= 6.13 that validates the corrupt EFI Memory Attributes
|
||||
# Table its firmware publishes (upstream 8fbe4c49c0cc). See
|
||||
# infra/PROD-PROVISIONING.md §0b/§3, and {{ node_baseline_backports_kernel_stamp }}
|
||||
# on this node.
|
||||
#
|
||||
# Backports is NotAutomatic, so nothing is taken from here unless asked for by name
|
||||
# with `-t {{ node_baseline_backports_suite }}`. Do NOT add an apt pin for it:
|
||||
# ButAutomaticUpgrades is what keeps the installed kernel patched, and an explicit
|
||||
# Pin-Priority would switch that off.
|
||||
Types: deb
|
||||
URIs: {{ node_baseline_backports_mirror }}
|
||||
Suites: {{ node_baseline_backports_suite }}
|
||||
Components: {{ node_baseline_backports_components | join(' ') }}
|
||||
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
|
||||
register: node_baseline_backports_source
|
||||
|
||||
- name: Refresh the apt cache after enabling backports
|
||||
ansible.builtin.apt:
|
||||
update_cache: true
|
||||
when: node_baseline_backports_source is changed
|
||||
|
||||
# `state: latest`, NOT `state: present`. linux-image-amd64 is ALREADY installed on these
|
||||
# nodes from trixie main (6.12.107-1), so `present` would find the name satisfied and do
|
||||
# nothing at all — the single most likely way to write this task and have it silently fix
|
||||
# nothing forever. `latest` + `default_release` resolves the candidate inside backports
|
||||
# and upgrades the metapackage to it. Once the node is current a re-run is a genuine
|
||||
# no-op: no change, no churn, no spurious reboot flag. Scope is exactly
|
||||
# node_baseline_backports_kernel_packages — one metapackage — so nothing else on the node
|
||||
# is dragged out of the main archive.
|
||||
- name: Install the fixed kernel from backports (>= 6.13)
|
||||
ansible.builtin.apt:
|
||||
name: "{{ node_baseline_backports_kernel_packages }}"
|
||||
state: latest
|
||||
default_release: "{{ node_baseline_backports_suite }}"
|
||||
update_cache: true
|
||||
cache_valid_time: 3600
|
||||
register: node_baseline_backports_kernel_apt
|
||||
|
||||
# WHICH VERSION DID THIS NODE ACTUALLY LAND ON. Read back rather than assumed: the
|
||||
# metapackage tracks a moving target, so "we installed linux-image-amd64" is not an
|
||||
# answer anybody can roll back from six months later.
|
||||
- name: Read back the installed backports kernel metapackage version
|
||||
ansible.builtin.command: dpkg-query -W -f '${Version}' {{ node_baseline_backports_kernel_packages | first }}
|
||||
register: node_baseline_backports_kernel_version
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
|
||||
# `linux-version` ships in linux-base and knows Debian's kernel version ordering, which
|
||||
# neither `sort -V` nor a Jinja sort gets right across +deb13 / ABI suffixes. The full
|
||||
# list is the ROLLBACK INVENTORY: every entry is still installed and selectable from
|
||||
# GRUB's "Advanced options" without uninstalling anything.
|
||||
- name: List every kernel image installed on this node, newest first
|
||||
ansible.builtin.shell:
|
||||
cmd: set -o pipefail; linux-version list | linux-version sort --reverse
|
||||
executable: /bin/bash
|
||||
register: node_baseline_installed_kernels
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
|
||||
# The node must be able to explain itself to an operator standing in front of it with no
|
||||
# repo checkout. Content is deliberately time-INVARIANT (no timestamp) so re-running the
|
||||
# role is a real no-op rather than a permanent daily "changed".
|
||||
- name: Record the deviation, the exact version, and the rollback path on the node
|
||||
ansible.builtin.copy:
|
||||
dest: "{{ node_baseline_backports_kernel_stamp }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
|
||||
#
|
||||
# THIS NODE DELIBERATELY DOES NOT RUN THE DEBIAN 13 STOCK KERNEL.
|
||||
#
|
||||
# WHY
|
||||
# This machine's HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table.
|
||||
# Linux < 6.13 reserves memory from it without validating its size, and an unlucky
|
||||
# boot reserves everything above the table: 2.6 GiB of 16 GB, observed twice on
|
||||
# k3s-prod-2. Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size
|
||||
# is clearly bogus" (first released in v6.13) adds the validation; it is NOT in
|
||||
# v6.12 and NOT in linux-6.12.y. There is NO kernel cmdline workaround.
|
||||
# Full write-up: infra/PROD-PROVISIONING.md §0b in the home-network-k3s-impl repo.
|
||||
#
|
||||
# PROOF THE FIX IS ACTIVE (not merely a lucky boot)
|
||||
# BOOT-INDEPENDENT, and this is the one to trust — the string only exists in
|
||||
# >= 6.13, so its presence in the RUNNING image is proof regardless of the boot:
|
||||
# zstd -dc < <(tail -c +$(...)) ... | grep 'Corrupted EFI Memory Attributes'
|
||||
# (roles/node_health does the version check for you; see the health report's
|
||||
# firmware.efi_memattr finding.)
|
||||
#
|
||||
# FROM dmesg, WHICH IS A LOTTERY. efi_memattr_init() on >= 6.13 has THREE gates
|
||||
# and each prints a DIFFERENT line — all three are rejections:
|
||||
# tbl->version > 2 'Unexpected EFI Memory Attributes table version %d'
|
||||
# (NOTE: printed with %d, so garbage may read NEGATIVE)
|
||||
# bad desc_size 'Unexpected EFI Memory Attributes descriptor size ...'
|
||||
# num_entries > 64K '[Firmware Bug]: Corrupted EFI Memory Attributes
|
||||
# Table detected! (version == ..., num_entries == ...)'
|
||||
# Which one appears depends on WHICH field this boot's garbage corrupted, so do
|
||||
# NOT treat the absence of the FW_BUG line as the fix being absent — all three
|
||||
# prod nodes printed the *version* line on 2026-09-04.
|
||||
# The one line that means the fix is NOT active:
|
||||
# 'Failed to map EFI Memory Attributes table @ 0x...' — the kernel trusted the
|
||||
# corrupt table. On a >= 6.13 kernel this should be unreachable; if it appears,
|
||||
# escalate, do not shrug.
|
||||
#
|
||||
# INSTALLED
|
||||
# source /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
|
||||
# package {{ node_baseline_backports_kernel_packages | join(' ') }}
|
||||
# version {{ node_baseline_backports_kernel_version.stdout | default('(not read)') }}
|
||||
# kernels installed on this node, newest first:
|
||||
{% for kv in (node_baseline_installed_kernels.stdout_lines | default([])) %}
|
||||
# {{ kv }}
|
||||
{% endfor %}
|
||||
#
|
||||
# SECURITY COST
|
||||
# Packages from *-backports are provided AS IS. They are NOT covered by the Debian
|
||||
# Security Team's guarantees for the main archive: no security-tracker commitment,
|
||||
# no DSAs. Accepted deliberately, in exchange for the node keeping its RAM.
|
||||
#
|
||||
# ROLLBACK — no repo checkout needed
|
||||
# 1. NON-DESTRUCTIVE, try this first. Every kernel listed above is still installed
|
||||
# and bootable: pick the 6.12 one from GRUB -> "Advanced options for Debian
|
||||
# GNU/Linux". That answers "is the new kernel the problem?" without
|
||||
# uninstalling anything. Note it reinstates the memattr bug for that boot.
|
||||
# 2. PERMANENT. Set node_baseline_backports_kernel: false in
|
||||
# inventory/group_vars/k3s_prod/vars.yml FIRST — otherwise the next Ansible run
|
||||
# puts the backports kernel straight back — then, on this node:
|
||||
# apt-get install --reinstall linux-image-amd64/{{ ansible_facts['distribution_release'] }}
|
||||
# apt-get purge <the backports linux-image-*+deb13-amd64 listed above>
|
||||
# rm /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
|
||||
# apt-get update && update-grub && reboot
|
||||
#
|
||||
# EXIT CONDITION
|
||||
# If 8fbe4c49c0cc is ever picked up into linux-6.12.y (and so into Debian's 6.12
|
||||
# rebase), this deviation can be removed entirely: flip the variable to false,
|
||||
# roll back as above, and delete the `kernel` block from roles/node_baseline.
|
||||
|
||||
# Say the pending reboot out loud. A green play here does NOT mean the node is fixed —
|
||||
# it means the fixed kernel is on disk. See the block header for why the role will not
|
||||
# reboot on its own.
|
||||
- name: Report whether this node is running the kernel it now has installed
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
{{ ('REBOOT REQUIRED — running ' ~ ansible_facts['kernel'] ~ ', newest installed is '
|
||||
~ newest ~ '. The EFI memattr fix is NOT active until this node reboots. Reboot
|
||||
prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER
|
||||
node 2 first (infra/PROD-PROVISIONING.md §0b "SECOND FAULT"). Afterwards confirm
|
||||
with: dmesg | grep -i "Corrupted EFI Memory Attributes Table"')
|
||||
if newest != ansible_facts['kernel']
|
||||
else ('Running the newest installed kernel (' ~ ansible_facts['kernel']
|
||||
~ '); no reboot pending for the memattr fix.') }}
|
||||
vars:
|
||||
newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
|
||||
|
||||
- name: Persist kernel modules for k3s/Cilium/Longhorn
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
|
||||
@@ -79,70 +314,32 @@
|
||||
state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
|
||||
tags: [system]
|
||||
|
||||
# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
|
||||
# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
|
||||
# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
|
||||
# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
|
||||
# which do not exist on bare metal either — a confusing failure that looks like a wiring
|
||||
# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
|
||||
# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
|
||||
- name: Assert this environment's data-disk devices have actually been decided
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- "'BLOCKED' not in node_baseline_longhorn_device"
|
||||
- "'BLOCKED' not in node_baseline_etcd_device"
|
||||
fail_msg: >-
|
||||
node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
|
||||
placeholders for this host's environment
|
||||
(longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
|
||||
The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
|
||||
and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
|
||||
the SMART read that decision waits on can be taken) re-run this play with
|
||||
`--tags packages`, which skips every disk task including this assert.
|
||||
quiet: true
|
||||
tags: [disks]
|
||||
|
||||
- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
|
||||
community.general.filesystem:
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_longhorn_device }}"
|
||||
opts: "-L {{ node_baseline_longhorn_fs_label }}"
|
||||
tags: [disks]
|
||||
|
||||
- name: Mount the Longhorn data disk
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_longhorn_mount }}"
|
||||
src: "LABEL={{ node_baseline_longhorn_fs_label }}"
|
||||
fstype: ext4
|
||||
opts: "defaults,nofail,x-systemd.device-timeout=5s"
|
||||
state: mounted
|
||||
tags: [disks]
|
||||
|
||||
# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
|
||||
# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
|
||||
# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
|
||||
# a disk that already carries a filesystem, so re-running against an already-migrated
|
||||
# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
|
||||
# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
|
||||
# role would mount over a populated directory.
|
||||
- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
|
||||
community.general.filesystem:
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_etcd_device }}"
|
||||
opts: "-L {{ node_baseline_etcd_fs_label }}"
|
||||
tags: [disks]
|
||||
|
||||
- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_etcd_mount }}"
|
||||
src: "LABEL={{ node_baseline_etcd_fs_label }}"
|
||||
fstype: ext4
|
||||
# noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
|
||||
# missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
|
||||
opts: "defaults,noatime"
|
||||
dump: "0"
|
||||
passno: "2"
|
||||
state: mounted
|
||||
# =====================================================================================
|
||||
# DATA DISKS — tag `disks`, in tasks/disks.yml
|
||||
# =====================================================================================
|
||||
# Split into its own file for two reasons. It is the only DESTRUCTIVE part of the role
|
||||
# (prod re-purposes the partition already mounted at /srv as the Longhorn store), so it
|
||||
# should be readable end-to-end on its own; and both environments' storage shapes — dev's
|
||||
# two dedicated virtio disks, prod's single partitioned physical disk with etcd left on
|
||||
# the OS filesystem — are declared there as an explicit enum rather than inferred from
|
||||
# which variables happen to be set. See tasks/disks.yml and infra/PROD-PROVISIONING.md §2.
|
||||
#
|
||||
# The `disks` tag is applied ONCE here, in BOTH places it is needed:
|
||||
# * `tags:` on the include itself, so `--tags packages` never even reaches the include
|
||||
# and a placeholder device path is not so much as templated; and
|
||||
# * `apply: tags:`, so the tag reaches the tasks INSIDE the file. This half is not
|
||||
# optional and not decoration — a DYNAMIC include (`include_tasks`) does NOT propagate
|
||||
# its own tags to its children the way `import_tasks` does, so without `apply` the
|
||||
# included tasks are untagged and `--tags disks` selects the include and then runs
|
||||
# NOTHING inside it. That failure is silent and reports success, which is the worst
|
||||
# shape a guard can have.
|
||||
# A FULL run against a node whose layout is undeclared stops on the assert at the top of
|
||||
# that file instead of failing obscurely at mkfs time.
|
||||
- name: Provision the data disks (Longhorn store, etcd data dir)
|
||||
ansible.builtin.include_tasks:
|
||||
file: disks.yml
|
||||
apply:
|
||||
tags: [disks]
|
||||
tags: [disks]
|
||||
|
||||
- name: Disable swap now
|
||||
|
||||
Reference in new issue
Block a user