jakman85andClaude Opus 5
05baae90ec
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
85 lines
5.5 KiB
Markdown
85 lines
5.5 KiB
Markdown
# platform/longhorn
|
|
|
|
**Longhorn** — distributed replicated block storage across the 3 nodes.
|
|
|
|
- **Governing ADR:** [ADR-0003 — Distributed storage: Longhorn (+ bulk tier + Velero)](../../../home-network-k3s/decisions/adr-0003-storage-longhorn.md)
|
|
- **Spec:** [docs/04 — Storage & State](../../../home-network-k3s/docs/04-storage-and-state.md)
|
|
- **Serves:** R-STATE-01, R-STATE-02, R-STATE-04 (R-STATE-03 is Velero's, Phase 4)
|
|
- **Owner:** WS-STORAGE
|
|
|
|
## What this is
|
|
|
|
A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`es:
|
|
|
|
- **`longhorn`** — the cluster's **only default** StorageClass, **3 replicas** with **hard node
|
|
anti-affinity** (one replica per node). Durable home for app/config state, databases, and
|
|
KubeVirt VM disks (via CDI). Replica count is the `${LONGHORN_DEFAULT_REPLICA_COUNT}` overlay
|
|
knob (cluster-vars, =3 in dev).
|
|
- **`bulk`** — thin indirection for the **external** (outside-cluster) media tier: Plex library,
|
|
download staging, arr handoff. **STUB placeholder** modelled as an external NFS mount; its backing
|
|
is **OQ-2 (unresolved)** and will not provision until answered. Plex's metadata DB stays on the
|
|
3-replica `longhorn` class regardless (docs/04, ADR-0003).
|
|
|
|
## Requirement mapping
|
|
|
|
| Requirement | Summary | How this component serves it |
|
|
|---|---|---|
|
|
| **R-STATE-01** | Stateful data survives destroy/rebuild of the workload | Volumes are Longhorn PVCs, independent of pod/VM lifecycle; `reclaimPolicy: Delete` on the class ties data to the PVC, not the workload. Destroy the pod/Deployment/VM → PVC + data persist → rebuild re-attaches. (Per-service rebuild test is the Phase-2 exit gate; T-1.5 proves node-level rebuild.) |
|
|
| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the node's data disk at `${LONGHORN_DATA_PATH}` (a `cluster-vars` knob: dev `/var/lib/longhorn`, a dedicated disk; prod `/data/longhorn`, a subdirectory of the partition it shares with k3s's data dir). `replicaAutoBalance: best-effort` re-spreads on node return. |
|
|
| **R-STATE-04** | Bulk media has a capacity strategy distinct from the replicated tier | Separate `bulk` StorageClass (NON-replicated, external NFS mount) keeps multi-TB media off the 3x tier; only Plex's small metadata DB uses `longhorn`. |
|
|
|
|
## Dev-scoped choices (change for prod)
|
|
|
|
- **`reclaimPolicy: Delete`** on `longhorn` — convenient in dev; prod may prefer `Retain` for
|
|
accidental-delete protection (backups still cover DR — R-STATE-03).
|
|
- **`storageOverProvisioningPercentage: 100`** — no thin over-provisioning in dev (small disks);
|
|
prod may raise it.
|
|
- **`storageReservedPercentageForDefaultDisk: 20` / `storageMinimalAvailablePercentage: 25`** —
|
|
headroom guardrails tuned for small dev disks; revisit for prod disk sizes.
|
|
- **Replica count = 3** comes from `cluster-vars.longhorn_default_replica_count`; both dev and prod
|
|
are 3 today (3 nodes), so the value is identical — the knob exists so it is never forked per env.
|
|
|
|
## Guardrails baked in (docs/04)
|
|
|
|
- Reserved disk headroom (never fill a node to 100%).
|
|
- Hard replica anti-affinity across nodes (no two replicas of a volume on one node).
|
|
- Single-zone anti-affinity relaxed (3 mini-PCs = one failure zone).
|
|
- Volume/disk health + degraded-replica alerting is WS-OBS's job (docs/10) — not configured here.
|
|
|
|
## Node prerequisites (already handled, informational)
|
|
|
|
Each node presents ext4 storage at `${LONGHORN_DATA_PATH}` and installs `open-iscsi` +
|
|
`nfs-common` + kernel modules — all via `roles/node_baseline` (`infra/cloud-init/` is a README
|
|
now; provisioning became API-only). This component assumes that floor. The shape differs per
|
|
environment, which is why the path is a knob:
|
|
|
|
| | |
|
|
|---|---|
|
|
| dev | a **dedicated** disk (`/dev/vdb`, `LABEL=longhorn`) mounted at `/var/lib/longhorn` — the mount root *is* the dataPath |
|
|
| prod | `/data/longhorn`, a **subdirectory** of the ~424 GB partition (`LABEL=k3s-data`) mounted at `/data`, whose sibling `/data/k3s` holds k3s's entire data dir (etcd + the containerd image store, moved off the 11.8 GiB `/var` — `infra/PROD-PROVISIONING.md` §2) |
|
|
|
|
`[parity]`: the value for an environment MUST equal `node_baseline_longhorn_data_path` in that
|
|
environment's Ansible group_vars. Ansible creates and mounts the directory; this component tells
|
|
Longhorn to use it.
|
|
|
|
## Contract (consumed by everyone with state)
|
|
|
|
Per `roadmap/workstreams.md`, WS-STORAGE exposes the `longhorn` (replicated) and `bulk` (media)
|
|
StorageClasses; WS-APPS/WS-VM/WS-OBS reference them from ServiceDefinitions. **Do not place bulk
|
|
media on the replicated tier, or app DBs on the bulk tier.**
|
|
|
|
## Open decisions / TODO (later phases)
|
|
|
|
- **OQ-2 (human):** decide the `bulk`-tier backing (NAS/NFS host + share, vs dedicated large disk).
|
|
Until answered, the `bulk` class is a non-functional placeholder (`server`/`share` = REPLACE-ME).
|
|
Blocks `bulk` finalize; does **not** block the core platform (docs/04).
|
|
- **Version pin (WS-STORAGE):** `>=1.12.0 <1.13.0` is a range; pin exactly at bring-up, Renovate
|
|
maintains.
|
|
- **Phase 4 (WS-STORAGE):** off-cluster backup target via `defaultSettings.backupTarget` +
|
|
SOPS-encrypted credential secret, wired alongside `platform/velero/` (R-STATE-03).
|
|
|
|
## Build check
|
|
|
|
`kubectl kustomize platform/longhorn` renders the namespace, HelmRepository, HelmRelease, and both
|
|
StorageClasses. (Structural only — not applied to a cluster here; the PM validates + deploys.)
|