fix(node_health): SMART verdict was wrong by 4096x — judge on portable signals

The first real SMART run FAILED k3s-prod-1's drive on reallocated=61440. That verdict
was WRONG, and the failure mode is instructive: the check trusted a VENDOR-DEFINED raw
value as if it were an ATA-standard sector count.

Micron TN-FD-20 ("SMART Attributes for the C400 SSD") names attribute 5 "Reallocated
BLOCK Count" and defines its raw as grown-bad-blocks x Block_Sector_Count, where
Block_Sector_Count is 256 x 16 = 4096 for the 128Gb-NAND parts (256GB/512GB). This is
the 512GB C400-MTFDDAC512MAM: 61440 / 4096 = 15. Triply corroborated on the drive itself
— attr 170 Grown_Failing_Block_Ct = 15, attr 196 Reallocated_Event_Count = 15, attr 5
normalised value 100 against thresh 10, and `when_failed` empty on every attribute.
FIFTEEN grown bad blocks, not sixty-one thousand bad sectors.

smartctl(8) states it outright: since ATA/ATAPI-4 rev 4 the meaning of raw Attribute
fields is "entirely vendor-specific". So a raw value can never be a portable verdict.

FAIL now comes only from signals that mean the same thing on every drive: the drive's own
smart_status.passed; NORMALISED value vs the MANUFACTURER's own threshold via smartctl's
when_failed ("now"/"past"/""), split Pre-fail -> FAIL and Old_age-or-past -> WARN; the
ATA-standard counters 197/198/187 (199 CRC is WARN-only — it is link, not media); and for
NVMe critical_warning / media_errors / percentage_used. Wear reads from the NORMALISED
lifetime indicator (202/231/233, or NVMe percentage_used) — a percentage by definition.

Vendor raw counts are still collected and PRINTED WITH THEIR NUMBERS, and may raise a
WARN, but can never drive a FAIL. Attribute 5's raw is now shown beside the normalised
reading that contradicts it, labelled vendor-encoded, with a mechanically-derived
"(= 15 x 4096)". Endurance attributes are matched by THE NAME smartctl's drivedb resolved
for that model, not by bare ID — drivedb is the portability layer; on an unrecognised
drive the name is Unknown_Attribute, nothing matches, and the field stays null with no
claim made. The STORAGE section now prints 170/172/173/174/189/206, whose previous
omission is exactly what let the misread stand unchallenged.

Also documented from TN-FD-20: on this family attr 197 "will always be 0, as reallocation
will be done on-the-fly" — so pending=0 is NOT evidence of health here. The role reports
the number and infers nothing from it.

Branches real hardware cannot exercise were verified with a harness rendering the actual
role expressions against synthetic smartctl payloads: virtio/no-stdout -> SKIP; NVMe
healthy -> PASS; NVMe degraded -> FAIL; a genuinely dying ATA drive (attr 5 normalised 8
<= thresh 10, pending 48) -> FAIL driven by the threshold trip and standard counters, not
the raw; unknown-vendor attrs -> PASS with the unnamed counter ignored; failed-in-the-past
-> WARN.

NEW VERDICTS. k3s-prod-1: 0 FAIL / 4 WARN / 12 PASS / 1 SKIP — SMART is a WARN naming the
real findings (31% of rated life consumed, 15 grown bad blocks, 187 erase failures, 32
unexpected power losses on a drive with no power-loss protection). k3s-dev-1: unchanged,
all three virtio disks still SKIP with their stated reason.

BREAKING (JSON artifact): the fact `reallocated_sectors` is renamed and gone — that name
asserted a unit ATA does not guarantee. Replaced by realloc_attr (the whole attribute,
normalised + raw), realloc_events, and grown_bad_blocks. Nothing in the repo reads it.

Serves: R-ENV-03, R-OBS-04.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-01 00:40:29 -05:00
1 parent ce1966acb0
commit 0b3db6e778
5 files changed
+398 -29

No files matched your search

+67 -2
View File
@@ -39,7 +39,7 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
| id | what |
|---|---|
| `access.privilege` | whether `become` works at all (gates the root-only checks) |
| `storage.smart.<disk>` | overall-health, power-on hours, lifetime remaining, reallocated + pending sectors, CRC errors, drive temperature — **root only** |
| `storage.smart.<disk>` | overall-health, normalised-vs-threshold trips, pending/uncorrectable/CRC counters, lifetime remaining, wear + endurance context, drive temperature — **root only**. See [How the SMART verdict is reached](#how-the-smart-verdict-is-reached) |
| `storage.io_errors` | kernel SCSI/ATA `ioerr_cnt` / `iotmo_cnt` per disk (no root needed) |
| `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
| `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
@@ -79,13 +79,78 @@ Equally verified: `failed_when: false` does **not** catch a missing-sudo-passwor
escalation probe uses `ignore_errors` — anything else turns "no vault yet" into a hard,
misleading play failure.
## How the SMART verdict is reached
**SMART raw values are vendor-defined and this role does not judge on them.** smartctl(8)
is explicit: "starting with ATA/ATAPI-4, revision 4, the meaning of these Attribute fields
has been made entirely vendor-specific."
This is not theoretical. The first real run of this role FAILED `k3s-prod-1`'s
Crucial/Micron C400 (`C400-MTFDDAC512MAM`, fw `070H`) on `reallocated=61440`, read as a
plain bad-sector count. Micron's own TN-FD-20 *SMART Attributes for the C400 SSD* defines
attribute 5 not as a sector count but as **Reallocated Block Count**, whose raw is "the
grown NAND Flash bad-block count multiplied by the Block_Sector_Count, which is 256 x 8 for
64Gb NAND ... or 256 x 16 for 128Gb NAND (256GB and 512GB)". For this 512GB drive that
multiplier is 256 x 16 = 4096, and 61440 / 4096 = **15** — corroborated on the same drive by
attribute 170 `Grown_Failing_Block_Ct` = 15 and attribute 196 `Reallocated_Event_Count` = 15.
Fifteen grown bad blocks, not sixty-one thousand bad sectors. The verdict was wrong by a
factor of 4096 because it assumed a unit the ATA spec does not guarantee.
### What the check judges on instead
Vendor-**independent** signals only, each of which means the same thing on every drive:
1. **The drive's own overall-health self-assessment** (`smart_status.passed`) — FAIL.
2. **Normalised VALUE vs the MANUFACTURER'S OWN THRESHOLD**, taken from smartctl's
`when_failed` field (`"now"` / `"past"` / `""`). Per smartctl(8), `FAILING_NOW` means the
normalised value is at or below the threshold *the manufacturer set for that attribute on
that model*; a dash means it is OK now and has never failed. This needs no per-vendor
knowledge from us — the vendor already encoded the answer. Pre-fail failing now → FAIL;
Old_age failing now, or anything that failed in the past → WARN.
3. **ATA-standard error counters** whose raw meaning is a plain event count everywhere:
197 pending, 198 offline-uncorrectable, 187 reported-uncorrectable → WARN/FAIL;
199 CRC (SATA link, not media) → WARN only.
4. **NVMe's structured health log** — `critical_warning`, `media_errors`, `percentage_used`.
5. **Wear from the NORMALISED lifetime indicator** (ATA 202/231/233 `value`, or NVMe
`percentage_used`) — a percentage by definition, and therefore portable where raw wear
counts are not.
### What is reported but never produces a FAIL
Vendor raw counts are real information and are all printed with their numbers — grown
bad-block count, reallocation events (196), erase/program failures, unexpected power losses,
and attribute 5's raw shown **beside** the normalised reading that contradicts it. They can
raise a **WARN**, never a FAIL, because their units are vendor-defined.
The endurance counters live in the vendor-specific attribute-ID range (170-174), so they are
matched by **the name smartctl's drive database resolved for that model**, not by a bare ID:
`drivedb.h` is smartmontools' own per-(model, firmware) translation, so keying off its
resolved name borrows that portability instead of hardcoding one vendor's layout. On a drive
drivedb does not recognise, the name is `Unknown_Attribute`, nothing matches, and the field
stays `null` — no claim is made either way.
### Two honesty traps this check does not fall into
- **Zero is not a clean bill of health.** Micron TN-FD-20 states attribute 197 on the C400
"will always be 0, as reallocation will be done on-the-fly". A non-zero 197 is a finding;
a zero one proves nothing. The role reports the number and does not infer health from it.
- **A quiet drive is not necessarily a good drive.** `overall-health=PASSED` with every
threshold un-tripped can still sit alongside a third of rated life consumed, grown bad
blocks, erase failures and unprotected power losses — which is exactly `k3s-prod-1`. Those
produce a WARN *with the numbers*, because a check that called that drive "fine" would be
worse than the false FAIL it replaced.
## Known limits
- Without root: **no SMART** (wear, power-on hours, reallocated/pending sectors, drive
- Without root: **no SMART** (wear, power-on hours, error counters, drive
temperature) and no DMI serial numbers. On prod that means creating
`inventory/group_vars/k3s_prod/vault.yml` (see that directory's README) and running
`node-baseline` so `smartmontools` exists — in that order.
- EDAC reports nothing on platforms with no ECC memory controller bound. That is
"ECC status unknown", not "no errors", and the role says so.
- SMART wear and endurance context depends on smartmontools' `drivedb.h` knowing the drive.
On an unrecognised model the vendor attributes are `Unknown_Attribute`; the portable
signals (overall health, normalised-vs-threshold, 197/198/187/199, NVMe log) still apply,
but the wear/endurance context SKIPs rather than guessing.
- `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or
unsupported commands during discovery. Attribution needs `dmesg`, which needs root.