jakman85andClaude Opus 5
05baae90ec
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
111 lines
6.4 KiB
YAML
111 lines
6.4 KiB
YAML
---
|
|
# Longhorn — distributed replicated block storage across the 3 nodes (ADR-0003).
|
|
# Serves R-STATE-01 (state survives destroy/rebuild) and R-STATE-02 (3-way replication).
|
|
#
|
|
# Dev-appropriate values. Environment differences enter ONLY via $${...} substitution from
|
|
# cluster-vars (R-ENV-02); everything else is identical base config for dev and prod.
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: longhorn
|
|
namespace: longhorn-system
|
|
spec:
|
|
interval: 30m
|
|
# Longhorn upgrades roll every workload + engine image on slow HDD-backed nodes and can exceed the
|
|
# 5m Helm default; a short timeout caused a W5 mid-upgrade wedge (data migrated to the new minor but
|
|
# workloads rolled back, and the new-version data refuses the old-version manager). Give it room.
|
|
timeout: 15m0s
|
|
chart:
|
|
spec:
|
|
chart: longhorn
|
|
version: ">=1.12.0 <1.13.0" # W5 upgrade 1.11->1.12 (resolves 1.12.1; supports k8s 1.36). Renovate maintains
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: longhorn
|
|
namespace: longhorn-system
|
|
install:
|
|
createNamespace: false # namespace.yaml owns it (GitOps-managed, labelled)
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
# NEVER auto-rollback a Longhorn minor upgrade: once the on-disk data migrates to the new minor,
|
|
# the older manager fatally refuses it ("downgrading from vX to vY is not supported"). Rolling
|
|
# back the workloads while data is already migrated wedges the cluster. Fail forward instead.
|
|
remediateLastFailure: false
|
|
values:
|
|
# We own the StorageClasses (storageclasses.yaml) so the default 3-replica class and the bulk
|
|
# tier are explicit and GitOps-managed. Disable the chart's built-in default SC so there is
|
|
# exactly one default-class annotation in the cluster.
|
|
persistence:
|
|
defaultClass: false
|
|
|
|
defaultSettings:
|
|
# --- data path (R-STATE-02, R-ENV-02) ---
|
|
# The on-node directory every replica is written into. A cluster-vars KNOB, not a literal,
|
|
# because the two environments mount their data disk differently and a hardcoded path would
|
|
# put one of them on the OS filesystem:
|
|
# dev /var/lib/longhorn — a dedicated ext4 disk (/dev/vdb) mounted there by
|
|
# roles/node_baseline. The mount root IS the dataPath: one disk,
|
|
# one tenant. 18 live volumes; this value must never change.
|
|
# prod /data/longhorn — a SUBDIRECTORY of the claimed ~424 GB partition mounted at
|
|
# /data, whose sibling /data/k3s holds k3s's entire data dir
|
|
# (etcd + the containerd image store, moved off the 11.8 GiB
|
|
# /var). Two tenants, so Longhorn gets a subdirectory rather
|
|
# than the mount root. infra/PROD-PROVISIONING.md §2.
|
|
# [parity] MUST equal roles/node_baseline's node_baseline_longhorn_data_path for the same
|
|
# environment — Ansible creates and mounts the directory, Helm tells Longhorn to use it.
|
|
# Set in BOTH overlays (clusters/dev, clusters/prod); base default is the dev value.
|
|
defaultDataPath: "${LONGHORN_DATA_PATH}"
|
|
|
|
# --- replication (R-STATE-02) ---
|
|
# Global fallback replica count. The `longhorn` StorageClass sets numberOfReplicas explicitly
|
|
# (authoritative); this keeps any volume created without an explicit count consistent.
|
|
# Substituted from cluster-vars.longhorn_default_replica_count (=3 in dev).
|
|
defaultReplicaCount: ${LONGHORN_DEFAULT_REPLICA_COUNT}
|
|
|
|
# --- replica anti-affinity across nodes (R-STATE-02 durability guarantee) ---
|
|
# HARD anti-affinity: never place two replicas of the same volume on the same node, so a
|
|
# single node loss cannot take down more than one replica. On a 3-node / 3-replica cluster
|
|
# this yields exactly one replica per node.
|
|
replicaSoftAntiAffinity: false
|
|
# Single zone (3 mini-PCs, no failure zones): allow replicas within the one zone.
|
|
replicaZoneSoftAntiAffinity: true
|
|
# When a downed node returns, rebalance replicas back toward one-per-node (best-effort so it
|
|
# never blocks scheduling).
|
|
replicaAutoBalance: "best-effort"
|
|
|
|
# --- rebuild I/O throttle for slow disks (etcd WAL fsync protection) ---
|
|
# The k3s VM disks sit on HDDs, so a full-replica rebuild's sequential I/O can saturate the
|
|
# spindle and starve etcd's WAL fsync — which crash-loops etcd and flaps the kube-vip VIP.
|
|
# Cap Longhorn to rebuilding ONE replica per node at a time so a node bounce / replica
|
|
# replenishment can never storm the HDDs. Deliberately conservative for this slow-disk cluster
|
|
# (Longhorn's chart default is higher). Pair with replica-replenishment-wait-interval=600
|
|
# (Longhorn default) so brief node bounces don't trigger replenishment at all.
|
|
concurrentReplicaRebuildPerNodeLimit: 1
|
|
|
|
# --- capacity guardrails (docs/04 "reserve disk headroom") ---
|
|
# Reserve 20% of the dedicated data disk so a node never fills to 100% (which wedges the
|
|
# kubelet and Longhorn alike). Dev disks are small, so headroom matters.
|
|
storageReservedPercentageForDefaultDisk: "20"
|
|
# Stop scheduling new replicas onto a disk once free space drops below 25%.
|
|
storageMinimalAvailablePercentage: "25"
|
|
# Dev: 200% thin over-provisioning — config-only volumes use little actual space, so the small
|
|
# 39Gi data disks host the full app catalog. Physical fill risk is low (watch monitoring PVCs).
|
|
# Prod-grade alternative: grow the Longhorn data disks and return to 1:1 (WS-STORAGE).
|
|
storageOverProvisioningPercentage: "200"
|
|
|
|
# --- misc dev-appropriate defaults ---
|
|
# Keep replicas spread for durability rather than co-locating with the pod.
|
|
defaultDataLocality: "disabled"
|
|
# Reclaim replica space of deleted/failed volumes promptly in dev.
|
|
orphanAutoDeletion: true
|
|
|
|
# Longhorn's own manager/UI stay in-cluster; UI exposure (if any) is gated behind
|
|
# oauth2-proxy forward-auth (ADR-0014) by WS-SEC — not exposed here.
|
|
|
|
# TODO(WS-STORAGE, Phase 4): off-cluster backup target (defaultSettings.backupTarget +
|
|
# backupTargetCredentialSecret) is configured alongside Velero (ADR-0003, R-STATE-03).
|
|
# Credentials referenced BY NAME only, SOPS-encrypted (WS-SEC) — never inline here.
|