fix(node_health): SMART verdict was wrong by 4096x — judge on portable signals

The first real SMART run FAILED k3s-prod-1's drive on reallocated=61440. That verdict
was WRONG, and the failure mode is instructive: the check trusted a VENDOR-DEFINED raw
value as if it were an ATA-standard sector count.

Micron TN-FD-20 ("SMART Attributes for the C400 SSD") names attribute 5 "Reallocated
BLOCK Count" and defines its raw as grown-bad-blocks x Block_Sector_Count, where
Block_Sector_Count is 256 x 16 = 4096 for the 128Gb-NAND parts (256GB/512GB). This is
the 512GB C400-MTFDDAC512MAM: 61440 / 4096 = 15. Triply corroborated on the drive itself
— attr 170 Grown_Failing_Block_Ct = 15, attr 196 Reallocated_Event_Count = 15, attr 5
normalised value 100 against thresh 10, and `when_failed` empty on every attribute.
FIFTEEN grown bad blocks, not sixty-one thousand bad sectors.

smartctl(8) states it outright: since ATA/ATAPI-4 rev 4 the meaning of raw Attribute
fields is "entirely vendor-specific". So a raw value can never be a portable verdict.

FAIL now comes only from signals that mean the same thing on every drive: the drive's own
smart_status.passed; NORMALISED value vs the MANUFACTURER's own threshold via smartctl's
when_failed ("now"/"past"/""), split Pre-fail -> FAIL and Old_age-or-past -> WARN; the
ATA-standard counters 197/198/187 (199 CRC is WARN-only — it is link, not media); and for
NVMe critical_warning / media_errors / percentage_used. Wear reads from the NORMALISED
lifetime indicator (202/231/233, or NVMe percentage_used) — a percentage by definition.

Vendor raw counts are still collected and PRINTED WITH THEIR NUMBERS, and may raise a
WARN, but can never drive a FAIL. Attribute 5's raw is now shown beside the normalised
reading that contradicts it, labelled vendor-encoded, with a mechanically-derived
"(= 15 x 4096)". Endurance attributes are matched by THE NAME smartctl's drivedb resolved
for that model, not by bare ID — drivedb is the portability layer; on an unrecognised
drive the name is Unknown_Attribute, nothing matches, and the field stays null with no
claim made. The STORAGE section now prints 170/172/173/174/189/206, whose previous
omission is exactly what let the misread stand unchallenged.

Also documented from TN-FD-20: on this family attr 197 "will always be 0, as reallocation
will be done on-the-fly" — so pending=0 is NOT evidence of health here. The role reports
the number and infers nothing from it.

Branches real hardware cannot exercise were verified with a harness rendering the actual
role expressions against synthetic smartctl payloads: virtio/no-stdout -> SKIP; NVMe
healthy -> PASS; NVMe degraded -> FAIL; a genuinely dying ATA drive (attr 5 normalised 8
<= thresh 10, pending 48) -> FAIL driven by the threshold trip and standard counters, not
the raw; unknown-vendor attrs -> PASS with the unnamed counter ignored; failed-in-the-past
-> WARN.

NEW VERDICTS. k3s-prod-1: 0 FAIL / 4 WARN / 12 PASS / 1 SKIP — SMART is a WARN naming the
real findings (31% of rated life consumed, 15 grown bad blocks, 187 erase failures, 32
unexpected power losses on a drive with no power-loss protection). k3s-dev-1: unchanged,
all three virtio disks still SKIP with their stated reason.

BREAKING (JSON artifact): the fact `reallocated_sectors` is renamed and gone — that name
asserted a unit ATA does not guarantee. Replaced by realloc_attr (the whole attribute,
normalised + raw), realloc_events, and grown_bad_blocks. Nothing in the repo reads it.

Serves: R-ENV-03, R-OBS-04.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

infra/ansible/roles/node_health/defaults/main.yml

@@ -60,13 +60,65 @@ node_health_thresholds:
   # collector looks behind the 127.0.0.53 stub rather than counting it as one server.
   # The fix side of this is the `dns`-tagged block in roles/node_baseline.
   min_nameservers: 2
-  # SMART (only evaluated when smartctl actually ran)
-  smart_reallocated_warn: 1
-  smart_reallocated_fail: 50
+  # --- SMART (only evaluated when smartctl actually ran) -----------------------------
+  #
+  # WHY THESE THRESHOLDS AND NOT "reallocated sectors" ANY MORE.
+  # The first real run of this role FAILED k3s-prod-1's Crucial/Micron C400 on
+  # `reallocated=61440`, read as a plain bad-sector count. That verdict was WRONG, and it
+  # was wrong for a structural reason, not a tuning one: SMART *raw* values are vendor
+  # defined. smartctl(8) says so outright -- "starting with ATA/ATAPI-4, revision 4, the
+  # meaning of these Attribute fields has been made entirely vendor-specific". Micron's own
+  # TN-FD-20 (SMART Attributes for the C400 SSD) defines ID 5 not as a sector count but as
+  # "Reallocated Block Count", raw = "the grown NAND Flash bad-block count multiplied by the
+  # Block_Sector_Count, which is 256 x 8 for 64Gb NAND ... or 256 x 16 for 128Gb NAND (256GB
+  # and 512GB)". 256 x 16 = 4096, and 61440 / 4096 = 15 -- corroborated independently by
+  # ID 170 Grown_Failing_Block_Ct = 15 and ID 196 Reallocated_Event_Count = 15. Fifteen
+  # grown bad blocks, not sixty-one thousand bad sectors.
+  #
+  # So no threshold below is ever applied to attribute 5's raw value. The verdict comes from
+  # the signals smartctl itself defines as vendor-INDEPENDENT:
+  #   * the drive's own overall-health self-assessment (smart_status.passed);
+  #   * NORMALISED value vs THRESHOLD, i.e. smartctl's when_failed ("now"/"past"), which is
+  #     the manufacturer's OWN trip point for that attribute on that drive;
+  #   * the ATA-standard error counters whose raw meaning is a plain event count everywhere:
+  #     197 pending, 198 offline-uncorrectable, 187 reported-uncorrectable, 199 CRC;
+  #   * NVMe's structured health log (critical_warning, media_errors, percentage_used).
+  # Vendor raw counts are still collected and printed -- they are real information -- but
+  # they can only ever raise a WARN with the number shown, never a FAIL on their own.
+  #
+  # Hard-error counters (197/198/187 raw, NVMe media_errors). A plain event count on every
+  # drive that implements them. NOTE the converse is not evidence: some drives (the C400
+  # among them, per TN-FD-20) hard-code 197 to 0 because reallocation happens on the fly, so
+  # "pending = 0" proves nothing. Nonzero is a finding; zero is not a clean bill of health.
   smart_pending_warn: 1
   smart_pending_fail: 10
+  smart_uncorrectable_warn: 1
+  smart_uncorrectable_fail: 10
+  # 199 UDMA_CRC_Error_Count -- SATA link/cable integrity, not media. WARN only: it is a
+  # lifetime count that never resets, so a historic cable reseat should not FAIL a node.
+  smart_crc_warn: 1
+  # Wear, from the NORMALISED lifetime indicator (ATA 202/231/233 value, or NVMe
+  # percentage_used) -- a percentage by definition, so portable in a way raw wear counts
+  # are not.  ..._remaining_* are the "this drive is nearly used up" bands; ..._used_pct_warn
+  # is the softer "this is not a fresh drive, know that before you put etcd on it" notice
+  # that PROD-PROVISIONING.md Section 2 actually needs.
   smart_lifetime_remaining_warn_pct: 20
   smart_lifetime_remaining_fail_pct: 10
+  smart_life_used_pct_warn: 25
+  # Reallocation ACTIVITY. 196 Reallocated_Event_Count is an ATA-standard *event* count
+  # (one per remap operation) and is the portable stand-in for attribute 5's unportable raw.
+  # WARN-only by design: remapping is what a drive is supposed to do, and how many events
+  # are "too many" is drive-specific -- the manufacturer already encodes that answer in
+  # attribute 5's normalised threshold, which the when_failed check above honours.
+  smart_realloc_events_warn: 1
+  # Endurance-stress counters, matched by the NAME smartctl's drive database resolved for
+  # this model rather than by a bare vendor ID -- drivedb IS the portability layer. On a
+  # drive smartctl does not recognise these simply do not match and nothing is claimed.
+  smart_erase_fail_warn: 1
+  smart_program_fail_warn: 1
+  # Unexpected power loss on a consumer SSD with no power-loss protection: each one is a
+  # chance of a torn/lost in-flight write. Directly relevant to etcd's fsync contract.
+  smart_unexpected_power_loss_warn: 10
   smart_temp_warn_c: 55
   smart_temp_fail_c: 70
   # Power-on hours past which a consumer SSD is simply old. Informational — age alone is
@@ -76,6 +128,11 @@ node_health_thresholds:
 
 # ATA SMART attribute IDs surfaced verbatim in the report. Vendors disagree on names, so
 # the report prints whatever the drive calls them and the checks key off the IDs.
-node_health_smart_attribute_ids: [5, 9, 177, 179, 181, 182, 183, 184, 187, 188, 190,
-                                  194, 195, 196, 197, 198, 199, 202, 231, 232, 233,
-                                  241, 242, 246]
+# 170-174 and 189 are the Micron/Crucial (and, by convention, most SSD controllers')
+# endurance and grown-bad-block counters; they are the CONTEXT that made the C400 verdict
+# above provable, and leaving them out of the report is what let a single misread raw value
+# stand unchallenged. 1 and 12 are included so the error-rate and power-cycle baselines are
+# visible next to them.
+node_health_smart_attribute_ids: [1, 5, 9, 12, 170, 171, 172, 173, 174, 177, 179, 181,
+                                  182, 183, 184, 187, 188, 189, 190, 194, 195, 196, 197,
+                                  198, 199, 202, 206, 231, 232, 233, 241, 242, 246]