They do not, and the correction changes the operator's plan rather than just a
detail.
Verified live: all three CNPG clusters run 3 instances spread one per node by
required anti-affinity, and k3s-prod-1 holds a REPLICA of each (keycloak-db-2,
nextcloud-db-2, semaphore-db-2). No primary is on node 1, so draining it forces
no failover at all — the best state this maintenance could ask for. The original
sentence was carried over from earlier session state and was simply stale; the
runbook now prints the command to re-derive it, because primaries move on their
own after any restart or failover.
Adds the warning that matters more. CNPGPrimariesCoLocated is currently firing
for k3s-prod-2, which holds both the keycloak-db and semaphore-db primaries. The
instinct is to tidy that before a maintenance window. Doing so would be actively
harmful here: rebalancing means promoting a replica, the only node with spare
primary capacity is node 1, and so "fixing" the alert would put a primary on the
very node about to be powered off — converting a no-failover drain into a
two-database failover. Rebalance after node 1 is back and caught up.
Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
The operator has a matched 32 GB pair to install. k3s-prod-1 is the node that
needs it: DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (16 GB, DDR4-3200, 1Rx8) and DIMM3
a 16ATF2G64HZ-2G6E1 (16 GB, DDR4-2667, 2Rx8), so capacity is correct at 32 GB
but the pair runs at the slower 2667. Nothing is faulty. k3s-prod-3 is a matched
Crucial pair.
The procedure exists because this node is the awkward one to take down: it held
24 Longhorn replicas, it is one of three etcd members (so the drain leaves
quorum at 2 of 3 with no further margin), and its rebuilds are the load that
starved etcd once already. So the runbook front-loads the checks that decide
whether to start at all, and treats a BLOCKED drain as the safety property
working rather than an obstacle to force past.
Two things it deliberately does not assert:
- node-drain-policy is not pinned in platform/longhorn/helmrelease.yaml, so the
runbook has the operator READ the live Setting and set it before draining if
it is not block-if-contains-last-replica, instead of assuming the default.
- Configured Memory Speed, not Speed, is named as the field that proves the
swap worked — a matched pair being clocked down still reports the rated Speed.
It also flags the capacity decision as a decision: 2x16 GB keeps the three nodes
symmetric as PROD-PROVISIONING.md §2 assumes, while 2x32 GB makes node 1 the
asymmetric one on memory as well as disk.
Recorded caveats: the node-health artifacts de-duplicate DIMM part numbers, so
they can prove "two different modules" but cannot distinguish one module from
two identical ones — verification therefore collects dmidecode live. And node 2's
inventory is stale and self-contradictory (documented as the only single-DIMM
node at 16 GB, yet reporting ~31.1 GiB), so it needs re-inventory before anyone
claims three-way symmetry. This is NOT the §0b "do not buy RAM" case, which was
an EFI memattr kernel bug on node 2.
Wired into the configMapGenerator so it is served at /runbooks/node-memory-swap
rather than existing only in git.
Co-Authored-By: Claude Opus 5 <[email protected]>
jakman85andClaude Opus 5