fix(node_health): SMART verdict was wrong by 4096x — judge on portable signals
The first real SMART run FAILED k3s-prod-1's drive on reallocated=61440. That verdict
was WRONG, and the failure mode is instructive: the check trusted a VENDOR-DEFINED raw
value as if it were an ATA-standard sector count.
Micron TN-FD-20 ("SMART Attributes for the C400 SSD") names attribute 5 "Reallocated
BLOCK Count" and defines its raw as grown-bad-blocks x Block_Sector_Count, where
Block_Sector_Count is 256 x 16 = 4096 for the 128Gb-NAND parts (256GB/512GB). This is
the 512GB C400-MTFDDAC512MAM: 61440 / 4096 = 15. Triply corroborated on the drive itself
— attr 170 Grown_Failing_Block_Ct = 15, attr 196 Reallocated_Event_Count = 15, attr 5
normalised value 100 against thresh 10, and `when_failed` empty on every attribute.
FIFTEEN grown bad blocks, not sixty-one thousand bad sectors.
smartctl(8) states it outright: since ATA/ATAPI-4 rev 4 the meaning of raw Attribute
fields is "entirely vendor-specific". So a raw value can never be a portable verdict.
FAIL now comes only from signals that mean the same thing on every drive: the drive's own
smart_status.passed; NORMALISED value vs the MANUFACTURER's own threshold via smartctl's
when_failed ("now"/"past"/""), split Pre-fail -> FAIL and Old_age-or-past -> WARN; the
ATA-standard counters 197/198/187 (199 CRC is WARN-only — it is link, not media); and for
NVMe critical_warning / media_errors / percentage_used. Wear reads from the NORMALISED
lifetime indicator (202/231/233, or NVMe percentage_used) — a percentage by definition.
Vendor raw counts are still collected and PRINTED WITH THEIR NUMBERS, and may raise a
WARN, but can never drive a FAIL. Attribute 5's raw is now shown beside the normalised
reading that contradicts it, labelled vendor-encoded, with a mechanically-derived
"(= 15 x 4096)". Endurance attributes are matched by THE NAME smartctl's drivedb resolved
for that model, not by bare ID — drivedb is the portability layer; on an unrecognised
drive the name is Unknown_Attribute, nothing matches, and the field stays null with no
claim made. The STORAGE section now prints 170/172/173/174/189/206, whose previous
omission is exactly what let the misread stand unchallenged.
Also documented from TN-FD-20: on this family attr 197 "will always be 0, as reallocation
will be done on-the-fly" — so pending=0 is NOT evidence of health here. The role reports
the number and infers nothing from it.
Branches real hardware cannot exercise were verified with a harness rendering the actual
role expressions against synthetic smartctl payloads: virtio/no-stdout -> SKIP; NVMe
healthy -> PASS; NVMe degraded -> FAIL; a genuinely dying ATA drive (attr 5 normalised 8
<= thresh 10, pending 48) -> FAIL driven by the threshold trip and standard counters, not
the raw; unknown-vendor attrs -> PASS with the unnamed counter ignored; failed-in-the-past
-> WARN.
NEW VERDICTS. k3s-prod-1: 0 FAIL / 4 WARN / 12 PASS / 1 SKIP — SMART is a WARN naming the
real findings (31% of rated life consumed, 15 grown bad blocks, 187 erase failures, 32
unexpected power losses on a drive with no power-loss protection). k3s-dev-1: unchanged,
all three virtio disks still SKIP with their stated reason.
BREAKING (JSON artifact): the fact `reallocated_sectors` is renamed and gone — that name
asserted a unit ATA does not guarantee. Replaced by realloc_attr (the whole attribute,
normalised + raw), realloc_events, and grown_bad_blocks. Nothing in the repo reads it.
Serves: R-ENV-03, R-OBS-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
ce1966acb0
commit
0b3db6e778
5 files changed
+398
-29
No files matched your search
@@ -39,7 +39,7 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
|
||||
| id | what |
|
||||
|---|---|
|
||||
| `access.privilege` | whether `become` works at all (gates the root-only checks) |
|
||||
| `storage.smart.<disk>` | overall-health, power-on hours, lifetime remaining, reallocated + pending sectors, CRC errors, drive temperature — **root only** |
|
||||
| `storage.smart.<disk>` | overall-health, normalised-vs-threshold trips, pending/uncorrectable/CRC counters, lifetime remaining, wear + endurance context, drive temperature — **root only**. See [How the SMART verdict is reached](#how-the-smart-verdict-is-reached) |
|
||||
| `storage.io_errors` | kernel SCSI/ATA `ioerr_cnt` / `iotmo_cnt` per disk (no root needed) |
|
||||
| `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
|
||||
| `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
|
||||
@@ -79,13 +79,78 @@ Equally verified: `failed_when: false` does **not** catch a missing-sudo-passwor
|
||||
escalation probe uses `ignore_errors` — anything else turns "no vault yet" into a hard,
|
||||
misleading play failure.
|
||||
|
||||
## How the SMART verdict is reached
|
||||
|
||||
**SMART raw values are vendor-defined and this role does not judge on them.** smartctl(8)
|
||||
is explicit: "starting with ATA/ATAPI-4, revision 4, the meaning of these Attribute fields
|
||||
has been made entirely vendor-specific."
|
||||
|
||||
This is not theoretical. The first real run of this role FAILED `k3s-prod-1`'s
|
||||
Crucial/Micron C400 (`C400-MTFDDAC512MAM`, fw `070H`) on `reallocated=61440`, read as a
|
||||
plain bad-sector count. Micron's own TN-FD-20 *SMART Attributes for the C400 SSD* defines
|
||||
attribute 5 not as a sector count but as **Reallocated Block Count**, whose raw is "the
|
||||
grown NAND Flash bad-block count multiplied by the Block_Sector_Count, which is 256 x 8 for
|
||||
64Gb NAND ... or 256 x 16 for 128Gb NAND (256GB and 512GB)". For this 512GB drive that
|
||||
multiplier is 256 x 16 = 4096, and 61440 / 4096 = **15** — corroborated on the same drive by
|
||||
attribute 170 `Grown_Failing_Block_Ct` = 15 and attribute 196 `Reallocated_Event_Count` = 15.
|
||||
Fifteen grown bad blocks, not sixty-one thousand bad sectors. The verdict was wrong by a
|
||||
factor of 4096 because it assumed a unit the ATA spec does not guarantee.
|
||||
|
||||
### What the check judges on instead
|
||||
|
||||
Vendor-**independent** signals only, each of which means the same thing on every drive:
|
||||
|
||||
1. **The drive's own overall-health self-assessment** (`smart_status.passed`) — FAIL.
|
||||
2. **Normalised VALUE vs the MANUFACTURER'S OWN THRESHOLD**, taken from smartctl's
|
||||
`when_failed` field (`"now"` / `"past"` / `""`). Per smartctl(8), `FAILING_NOW` means the
|
||||
normalised value is at or below the threshold *the manufacturer set for that attribute on
|
||||
that model*; a dash means it is OK now and has never failed. This needs no per-vendor
|
||||
knowledge from us — the vendor already encoded the answer. Pre-fail failing now → FAIL;
|
||||
Old_age failing now, or anything that failed in the past → WARN.
|
||||
3. **ATA-standard error counters** whose raw meaning is a plain event count everywhere:
|
||||
197 pending, 198 offline-uncorrectable, 187 reported-uncorrectable → WARN/FAIL;
|
||||
199 CRC (SATA link, not media) → WARN only.
|
||||
4. **NVMe's structured health log** — `critical_warning`, `media_errors`, `percentage_used`.
|
||||
5. **Wear from the NORMALISED lifetime indicator** (ATA 202/231/233 `value`, or NVMe
|
||||
`percentage_used`) — a percentage by definition, and therefore portable where raw wear
|
||||
counts are not.
|
||||
|
||||
### What is reported but never produces a FAIL
|
||||
|
||||
Vendor raw counts are real information and are all printed with their numbers — grown
|
||||
bad-block count, reallocation events (196), erase/program failures, unexpected power losses,
|
||||
and attribute 5's raw shown **beside** the normalised reading that contradicts it. They can
|
||||
raise a **WARN**, never a FAIL, because their units are vendor-defined.
|
||||
|
||||
The endurance counters live in the vendor-specific attribute-ID range (170-174), so they are
|
||||
matched by **the name smartctl's drive database resolved for that model**, not by a bare ID:
|
||||
`drivedb.h` is smartmontools' own per-(model, firmware) translation, so keying off its
|
||||
resolved name borrows that portability instead of hardcoding one vendor's layout. On a drive
|
||||
drivedb does not recognise, the name is `Unknown_Attribute`, nothing matches, and the field
|
||||
stays `null` — no claim is made either way.
|
||||
|
||||
### Two honesty traps this check does not fall into
|
||||
|
||||
- **Zero is not a clean bill of health.** Micron TN-FD-20 states attribute 197 on the C400
|
||||
"will always be 0, as reallocation will be done on-the-fly". A non-zero 197 is a finding;
|
||||
a zero one proves nothing. The role reports the number and does not infer health from it.
|
||||
- **A quiet drive is not necessarily a good drive.** `overall-health=PASSED` with every
|
||||
threshold un-tripped can still sit alongside a third of rated life consumed, grown bad
|
||||
blocks, erase failures and unprotected power losses — which is exactly `k3s-prod-1`. Those
|
||||
produce a WARN *with the numbers*, because a check that called that drive "fine" would be
|
||||
worse than the false FAIL it replaced.
|
||||
|
||||
## Known limits
|
||||
|
||||
- Without root: **no SMART** (wear, power-on hours, reallocated/pending sectors, drive
|
||||
- Without root: **no SMART** (wear, power-on hours, error counters, drive
|
||||
temperature) and no DMI serial numbers. On prod that means creating
|
||||
`inventory/group_vars/k3s_prod/vault.yml` (see that directory's README) and running
|
||||
`node-baseline` so `smartmontools` exists — in that order.
|
||||
- EDAC reports nothing on platforms with no ECC memory controller bound. That is
|
||||
"ECC status unknown", not "no errors", and the role says so.
|
||||
- SMART wear and endurance context depends on smartmontools' `drivedb.h` knowing the drive.
|
||||
On an unrecognised model the vendor attributes are `Unknown_Attribute`; the portable
|
||||
signals (overall health, normalised-vs-threshold, 197/198/187/199, NVMe log) still apply,
|
||||
but the wear/endurance context SKIPs rather than guessing.
|
||||
- `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or
|
||||
unsupported commands during discovery. Attribution needs `dmesg`, which needs root.
|
||||
Reference in new issue
Block a user