jakman85andClaude Opus 5
05baae90ec
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
222 lines
15 KiB
YAML
222 lines
15 KiB
YAML
---
|
|
# node_baseline — first-boot floor for k3s/Cilium/Longhorn, moved out of cloud-init
|
|
# (Proxmox provisioning is API-only). Runs before k3s (site.yml). Idempotent.
|
|
node_baseline_packages:
|
|
# VM-only in practice (binds the virtio-serial channel Proxmox attaches) but harmless to install
|
|
# unconditionally: the task below only *starts* it when that channel exists, so on bare metal
|
|
# (prod) it installs and stays a dormant, disabled unit. Kept in the shared list rather than
|
|
# split per-platform so dev and prod install from the exact same package set (R-ENV-02 reuse).
|
|
- qemu-guest-agent
|
|
- open-iscsi
|
|
- nfs-common
|
|
- curl
|
|
- cryptsetup # Longhorn >=1.12 RequiredPackages check needs the cryptsetup userspace tool (dm_crypt module alone isn't enough)
|
|
# Drive health (smartctl) is a PERMANENT part of a bare-metal node's baseline, not a one-off
|
|
# manual install: prod runs on real disks that can fail/degrade (see infra/PROD-PROVISIONING.md
|
|
# §2/§5 — a ~15-year-old consumer SSD on node 1), where dev's virtio disks never needed it.
|
|
# Installed here (not hand-run with apt-get) so it's declared, idempotent, and present on every
|
|
# node from first baseline — package installation is the role's job, never a manual step
|
|
# (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo).
|
|
- smartmontools
|
|
# --- BACKPORTS KERNEL (EFI memattr, PROD-PROVISIONING.md §0b/§3) ---------------------
|
|
# OFF BY DEFAULT, ON FOR PROD. This is a STANDING, DELIBERATE DEVIATION from the Debian 13
|
|
# baseline — not a one-off `apt install` someone has to remember on the next node.
|
|
#
|
|
# WHY: HP's R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI Memory
|
|
# Attributes Table. Linux < 6.13 does not validate it before `memblock_reserve()`ing memory
|
|
# from its contents, so an unlucky boot reserves everything above the table and the node
|
|
# comes up with a fraction of its RAM (k3s-prod-2: 2.6 GiB of 16 GB, twice). Upstream commit
|
|
# 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" (Ard Biesheuvel,
|
|
# first released in v6.13) adds the validation. It is NOT in v6.12 and NOT in linux-6.12.y —
|
|
# no `Cc: stable`, no `Fixes:`, so no automatic feed will ever pick it up. There is also NO
|
|
# kernel cmdline workaround: efi_memattr_init() runs before any `efi=` bail-out. Booting a
|
|
# >= 6.13 kernel is the only fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
|
|
#
|
|
# WHY A VARIABLE AND NOT A HOSTNAME TEST. The dev nodes are Proxmox VMs whose firmware
|
|
# publishes no MAT at all — they are genuinely unaffected and must NOT be dragged onto a
|
|
# backports kernel as a side effect of sharing this role (R-ENV-02 is "same role, scoped
|
|
# overrides", not "same kernel"). A `when: inventory_hostname is match('k3s-prod-')` would
|
|
# encode the environment into the ROLE, would silently miss a fourth prod node named
|
|
# differently, and could not be flipped for a one-off test host. So it is a plain boolean,
|
|
# default false, turned on in inventory/group_vars/k3s_prod/vars.yml — the SAME seam
|
|
# kube_vip_address / k3s_api_host / cluster_kubeconfig_name already use.
|
|
#
|
|
# SECURITY COST, STATED PLAINLY: packages in *-backports are provided AS-IS. They are NOT
|
|
# covered by the Debian Security Team's guarantees for the main archive — no security-tracker
|
|
# commitment, no DSAs. In exchange for that the node keeps the RAM it paid for. Accepting it
|
|
# is the operator's recorded decision (PROD-PROVISIONING.md §0b/§3), not this role's.
|
|
#
|
|
# EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y (and therefore in Debian's
|
|
# 6.12 rebase), flip this back to false everywhere, `apt purge` the backports kernel, and
|
|
# delete this block. Nothing else in the repo depends on it.
|
|
node_baseline_backports_kernel: false
|
|
# The suite, mirror and components the source line is built from. Split out so a mirror
|
|
# change is a variable, not an edit to a heredoc in tasks/main.yml.
|
|
node_baseline_backports_suite: trixie-backports
|
|
node_baseline_backports_mirror: "http://deb.debian.org/debian"
|
|
# Matches /etc/apt/sources.list on these nodes (main + non-free-firmware). Deliberately NOT
|
|
# contrib/non-free: this deviation exists to get ONE fixed kernel, and widening the archive
|
|
# surface beyond what the node already trusts is a separate decision nobody has taken.
|
|
node_baseline_backports_components:
|
|
- main
|
|
- non-free-firmware
|
|
# The METApackage, not a pinned ABI (e.g. linux-image-6.19.14+deb13-amd64). A frozen ABI
|
|
# would stop receiving even backports' own updates the moment the ABI rolls, leaving the
|
|
# node on a kernel nothing patches — strictly worse than the risk this deviation accepts.
|
|
# The metapackage tracks whatever backports currently ships (>= 6.13 by construction, and
|
|
# far above it in practice), and `ButAutomaticUpgrades: yes` in the backports Release file
|
|
# means `apt upgrade` keeps it current once installed. The EXACT version each node landed on
|
|
# is recorded on the node itself — see node_baseline_backports_kernel_stamp.
|
|
node_baseline_backports_kernel_packages:
|
|
- linux-image-amd64
|
|
# Provenance/rollback note written on the node. An operator standing in front of a prod box
|
|
# with no repo checkout must be able to answer "why is this machine not on the distro
|
|
# kernel, and how do I undo it?" from the machine itself.
|
|
node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel
|
|
|
|
node_baseline_kernel_modules:
|
|
- overlay
|
|
- br_netfilter
|
|
- iscsi_tcp
|
|
- dm_crypt
|
|
- nfs
|
|
|
|
# --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
|
|
# The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
|
|
# a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:
|
|
#
|
|
# node_baseline_longhorn_source:
|
|
# device dedicated block device at node_baseline_longhorn_device (DEV: the
|
|
# tofu-provisioned virtio disk /dev/vdb). mkfs + mount by label.
|
|
# claim-mountpoint re-purpose the partition the node ALREADY has mounted at
|
|
# node_baseline_longhorn_claim_mountpoint (PROD: /srv). The device
|
|
# path is discovered ON THE NODE — it is /dev/sda5 on prod nodes 1
|
|
# and 3 but /dev/nvme0n1p5 on node 2, so no group-wide device value
|
|
# is correct and a hardcoded one would eventually mkfs the wrong
|
|
# partition on a future node. Destructive; heavily guarded — read
|
|
# tasks/disks.yml before changing anything here.
|
|
#
|
|
# node_baseline_etcd_source:
|
|
# device dedicated block device at node_baseline_etcd_device (DEV:
|
|
# /dev/vdc, docs/plans/ssd-etcd-isolation.md — etcd WAL fsync
|
|
# isolated from Longhorn rebuild I/O after dev was burned by HDD
|
|
# contention).
|
|
# root-filesystem etcd lives on the OS filesystem, with NO dedicated device and
|
|
# NO claimed data partition under it. Kept as a valid, supported
|
|
# shape; NOT what prod uses any more (see below).
|
|
# shared-data-partition etcd lives inside node_baseline_k3s_data_dir, which sits on the
|
|
# CLAIMED data partition (PROD). No dedicated device, but also NOT
|
|
# the OS filesystem: the whole point of the 2026-09-04 layout
|
|
# revision is that /var (11.8 GiB) can hold neither the container
|
|
# image store nor etcd's WAL. See infra/PROD-PROVISIONING.md §2.
|
|
#
|
|
# Every one of these is a FIRST-CLASS, explicitly-configured state — an operator decision
|
|
# recorded in infra/PROD-PROVISIONING.md §2 — and NOT the same thing as an unset device
|
|
# var, which still fails the `disks` assert.
|
|
node_baseline_longhorn_source: device
|
|
node_baseline_longhorn_device: /dev/vdb
|
|
# --- The DATA PARTITION: mountpoint + filesystem label -------------------------------
|
|
# NEUTRAL NAMES ON PURPOSE (revised 2026-09-04). On prod this ONE partition now serves TWO
|
|
# tenants — Longhorn's replica store AND k3s's whole data dir (etcd + the containerd image
|
|
# store) — so neither the mountpoint nor the label may name only one of them. Prod overrides
|
|
# both in group_vars/k3s_prod/vars.yml (`/data`, label `k3s-data`).
|
|
#
|
|
# THE DEFAULTS BELOW ARE DEV'S LIVE VALUES AND MUST NOT CHANGE. The dev cluster has 18
|
|
# Longhorn volumes whose replicas live on the partition labelled `longhorn` mounted at
|
|
# /var/lib/longhorn; dev's etcd has its own dedicated device and dev's k3s data dir stays
|
|
# at k3s's own default. Dev's mountpoint IS Longhorn's dataPath there (one tenant, one
|
|
# partition), which is why node_baseline_longhorn_data_path below defaults to the mount.
|
|
node_baseline_data_mount: /var/lib/longhorn
|
|
node_baseline_data_fs_label: longhorn
|
|
# Longhorn's dataPath — the directory Longhorn puts replicas in. On dev that IS the mount
|
|
# root (single tenant). On prod it is a SUBDIRECTORY of the mount, because k3s owns a
|
|
# sibling subdirectory of the same partition. Must agree with the LONGHORN_DATA_PATH knob
|
|
# in clusters/base/cluster-vars.yaml (patched per overlay) — that is the Helm side of the
|
|
# same fact, and the two are checked against each other by nothing but this comment.
|
|
node_baseline_longhorn_data_path: "{{ node_baseline_data_mount }}"
|
|
# k3s's --data-dir. Defers to the CLUSTER-WIDE `k3s_data_dir` when an environment declares
|
|
# one (group_vars), and otherwise restates k3s's own built-in default. It is deliberately
|
|
# NOT a second independent variable: roles/k3s_server reads `k3s_data_dir` to build the
|
|
# install flags, this role reads the same value to create the directory and to place etcd,
|
|
# and one line in group_vars/k3s_prod/vars.yml moves BOTH. Lazy Jinja means the group_vars
|
|
# value (higher precedence than any role default) is what actually resolves here, even
|
|
# though roles/k3s_server is not in the node-baseline play; the default() keeps
|
|
# node-baseline.yml runnable standalone.
|
|
node_baseline_k3s_data_dir: "{{ k3s_data_dir | default('/var/lib/rancher/k3s') }}"
|
|
# Only consulted when node_baseline_longhorn_source is `claim-mountpoint`. Empty by
|
|
# default so a dev host cannot enter the claim path even by accident.
|
|
node_baseline_longhorn_claim_mountpoint: ""
|
|
# --- ONE-TIME TRANSITION KNOBS (delete once the fleet is converged) ------------------
|
|
# A node claimed by an EARLIER version of this role carries the OLD label at the OLD
|
|
# mountpoint, and the idempotency pivot (`blkid -L <current label>`) would therefore read
|
|
# "not claimed yet" and walk into the greenfield claim path — which then fails closed at
|
|
# "the claim mountpoint must actually be a mountpoint", because /srv is long gone. These
|
|
# two vars declare the state to migrate FROM. Empty by default: dev's label and mountpoint
|
|
# are NOT changing, so dev has nothing to transition and cannot enter the transition path.
|
|
node_baseline_data_retired_fs_label: ""
|
|
node_baseline_data_retired_mount: ""
|
|
# Sanity floor for the claim path: a data partition smaller than this is not "a small
|
|
# store", it is the wrong partition. Prod's /srv is 415-425 GiB.
|
|
node_baseline_longhorn_min_size_gb: 100
|
|
# Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
|
|
# so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
|
|
# See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
|
|
# device renames (matches the data partition's LABEL= approach).
|
|
node_baseline_etcd_source: device
|
|
node_baseline_etcd_device: /dev/vdc
|
|
# DERIVED, never restated: etcd's db always lives at <k3s data dir>/server/db, so moving
|
|
# the data dir moves etcd with it and the two cannot drift. Dev resolves to the unchanged
|
|
# /var/lib/rancher/k3s/server/db (where dev's dedicated etcd SSD is mounted); prod resolves
|
|
# to /data/k3s/server/db, inside the claimed partition.
|
|
node_baseline_etcd_mount: "{{ node_baseline_k3s_data_dir }}/server/db"
|
|
node_baseline_etcd_fs_label: k3s-etcd
|
|
# Mounts whose backing devices the claim path must NEVER touch. Used to build the
|
|
# protected-device set from the node's own gathered facts (plus live swap), so a node
|
|
# partitioned differently from the fleet is judged on its own layout rather than against
|
|
# a hardcoded list of partition names.
|
|
node_baseline_protected_mounts:
|
|
- /
|
|
- /boot
|
|
- /boot/efi
|
|
- /usr
|
|
- /var
|
|
- /home
|
|
- /tmp
|
|
|
|
# --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
|
|
# These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml
|
|
# (the same seam kube_vip_address / k3s_api_host / cluster_kubeconfig_name use) — NOT in
|
|
# clusters/base/cluster-vars.yaml, which is a Flux postBuild ConfigMap that no Ansible
|
|
# code reads.
|
|
#
|
|
# WHY: the nodes shipped with ONE upstream resolver. PiHole answers the local jackson.lan
|
|
# zone instantly but its external recursion flaps (measured 0/25 for registry-1.docker.io /
|
|
# quay.io / ghcr.io while gitea.jackson.lan resolved 5/5), and one upstream is a hard SPOF:
|
|
# it broke containerd image pulls ("lookup registry-1.docker.io: Try again") AND CoreDNS,
|
|
# which forwards to /etc/resolv.conf and flooded its log with i/o timeouts to 192.168.1.18.
|
|
#
|
|
# ORDER IS THE POLICY. This is a RESILIENCE fix, not a bypass: the first entry stays PiHole,
|
|
# so *.jackson.lan and network-wide ad-blocking keep working exactly as before, and the
|
|
# second is only ever consulted when the first fails. Both write paths preserve that order
|
|
# (systemd-resolved tries the servers of one scope in order; glibc walks `nameserver` lines
|
|
# in order). Never put a public resolver first, and never enable `rotate` — either would
|
|
# silently take the local zone and the ad-block policy out of the path.
|
|
#
|
|
# Keep in parity with infra/tofu (var.dns_server, var.network_gateway), which sets the same
|
|
# two servers via cloud-init so a freshly rebuilt dev node boots with this list already.
|
|
node_baseline_dns_servers:
|
|
- 192.168.1.18 # PiHole — local jackson.lan zone + ad-blocking. PRIMARY, keep first.
|
|
- 192.168.1.1 # pfSense gateway — reliable external recursion. FALLBACK only.
|
|
# Search domain(s). An EMPTY list means "configure no search domain" and is a legitimate
|
|
# value (prod uses it — its local domain is not a decided thing), not a missing one.
|
|
node_baseline_dns_search:
|
|
- jackson.lan
|
|
# resolv.conf `options` for the DIRECT-file mechanism only (systemd-resolved does its own
|
|
# per-server failover timing and ignores these). glibc's defaults are timeout:5 attempts:2,
|
|
# i.e. up to 20s of stalled lookups before it reaches the second server — long enough for
|
|
# containerd to give up on a pull. These shorten the failover, and deliberately do NOT
|
|
# include `rotate`, which would round-robin the servers and break PiHole-first ordering.
|
|
node_baseline_dns_options:
|
|
- timeout:2
|
|
- attempts:2
|