The operator has a matched 32 GB pair to install. k3s-prod-1 is the node that
needs it: DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (16 GB, DDR4-3200, 1Rx8) and DIMM3
a 16ATF2G64HZ-2G6E1 (16 GB, DDR4-2667, 2Rx8), so capacity is correct at 32 GB
but the pair runs at the slower 2667. Nothing is faulty. k3s-prod-3 is a matched
Crucial pair.
The procedure exists because this node is the awkward one to take down: it held
24 Longhorn replicas, it is one of three etcd members (so the drain leaves
quorum at 2 of 3 with no further margin), and its rebuilds are the load that
starved etcd once already. So the runbook front-loads the checks that decide
whether to start at all, and treats a BLOCKED drain as the safety property
working rather than an obstacle to force past.
Two things it deliberately does not assert:
- node-drain-policy is not pinned in platform/longhorn/helmrelease.yaml, so the
runbook has the operator READ the live Setting and set it before draining if
it is not block-if-contains-last-replica, instead of assuming the default.
- Configured Memory Speed, not Speed, is named as the field that proves the
swap worked — a matched pair being clocked down still reports the rated Speed.
It also flags the capacity decision as a decision: 2x16 GB keeps the three nodes
symmetric as PROD-PROVISIONING.md §2 assumes, while 2x32 GB makes node 1 the
asymmetric one on memory as well as disk.
Recorded caveats: the node-health artifacts de-duplicate DIMM part numbers, so
they can prove "two different modules" but cannot distinguish one module from
two identical ones — verification therefore collects dmidecode live. And node 2's
inventory is stale and self-contradictory (documented as the only single-DIMM
node at 16 GB, yet reporting ~31.1 GiB), so it needs re-inventory before anyone
claims three-way symmetry. This is NOT the §0b "do not buy RAM" case, which was
an EFI memattr kernel bug on node 2.
Wired into the configMapGenerator so it is served at /runbooks/node-memory-swap
rather than existing only in git.
Co-Authored-By: Claude Opus 5 <[email protected]>
