fix(node_health): SMART verdict was wrong by 4096x — judge on portable signals

The first real SMART run FAILED k3s-prod-1's drive on reallocated=61440. That verdict
was WRONG, and the failure mode is instructive: the check trusted a VENDOR-DEFINED raw
value as if it were an ATA-standard sector count.

Micron TN-FD-20 ("SMART Attributes for the C400 SSD") names attribute 5 "Reallocated
BLOCK Count" and defines its raw as grown-bad-blocks x Block_Sector_Count, where
Block_Sector_Count is 256 x 16 = 4096 for the 128Gb-NAND parts (256GB/512GB). This is
the 512GB C400-MTFDDAC512MAM: 61440 / 4096 = 15. Triply corroborated on the drive itself
— attr 170 Grown_Failing_Block_Ct = 15, attr 196 Reallocated_Event_Count = 15, attr 5
normalised value 100 against thresh 10, and `when_failed` empty on every attribute.
FIFTEEN grown bad blocks, not sixty-one thousand bad sectors.

smartctl(8) states it outright: since ATA/ATAPI-4 rev 4 the meaning of raw Attribute
fields is "entirely vendor-specific". So a raw value can never be a portable verdict.

FAIL now comes only from signals that mean the same thing on every drive: the drive's own
smart_status.passed; NORMALISED value vs the MANUFACTURER's own threshold via smartctl's
when_failed ("now"/"past"/""), split Pre-fail -> FAIL and Old_age-or-past -> WARN; the
ATA-standard counters 197/198/187 (199 CRC is WARN-only — it is link, not media); and for
NVMe critical_warning / media_errors / percentage_used. Wear reads from the NORMALISED
lifetime indicator (202/231/233, or NVMe percentage_used) — a percentage by definition.

Vendor raw counts are still collected and PRINTED WITH THEIR NUMBERS, and may raise a
WARN, but can never drive a FAIL. Attribute 5's raw is now shown beside the normalised
reading that contradicts it, labelled vendor-encoded, with a mechanically-derived
"(= 15 x 4096)". Endurance attributes are matched by THE NAME smartctl's drivedb resolved
for that model, not by bare ID — drivedb is the portability layer; on an unrecognised
drive the name is Unknown_Attribute, nothing matches, and the field stays null with no
claim made. The STORAGE section now prints 170/172/173/174/189/206, whose previous
omission is exactly what let the misread stand unchallenged.

Also documented from TN-FD-20: on this family attr 197 "will always be 0, as reallocation
will be done on-the-fly" — so pending=0 is NOT evidence of health here. The role reports
the number and infers nothing from it.

Branches real hardware cannot exercise were verified with a harness rendering the actual
role expressions against synthetic smartctl payloads: virtio/no-stdout -> SKIP; NVMe
healthy -> PASS; NVMe degraded -> FAIL; a genuinely dying ATA drive (attr 5 normalised 8
<= thresh 10, pending 48) -> FAIL driven by the threshold trip and standard counters, not
the raw; unknown-vendor attrs -> PASS with the unnamed counter ignored; failed-in-the-past
-> WARN.

NEW VERDICTS. k3s-prod-1: 0 FAIL / 4 WARN / 12 PASS / 1 SKIP — SMART is a WARN naming the
real findings (31% of rated life consumed, 15 grown bad blocks, 187 erase failures, 32
unexpected power losses on a drive with no power-loss protection). k3s-dev-1: unchanged,
all three virtio disks still SKIP with their stated reason.

BREAKING (JSON artifact): the fact `reallocated_sectors` is renamed and gone — that name
asserted a unit ATA does not guarantee. Replaced by realloc_attr (the whole attribute,
normalised + raw), realloc_events, and grown_bad_blocks. Nothing in the repo reads it.

Serves: R-ENV-03, R-OBS-04.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

infra/ansible/roles/node_health/README.md

@@ -39,7 +39,7 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 | id | what |
 |---|---|
 | `access.privilege` | whether `become` works at all (gates the root-only checks) |
-| `storage.smart.<disk>` | overall-health, power-on hours, lifetime remaining, reallocated + pending sectors, CRC errors, drive temperature — **root only** |
+| `storage.smart.<disk>` | overall-health, normalised-vs-threshold trips, pending/uncorrectable/CRC counters, lifetime remaining, wear + endurance context, drive temperature — **root only**. See [How the SMART verdict is reached](#how-the-smart-verdict-is-reached) |
 | `storage.io_errors` | kernel SCSI/ATA `ioerr_cnt` / `iotmo_cnt` per disk (no root needed) |
 | `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
 | `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
@@ -79,13 +79,78 @@ Equally verified: `failed_when: false` does **not** catch a missing-sudo-passwor
 escalation probe uses `ignore_errors` — anything else turns "no vault yet" into a hard,
 misleading play failure.
 
+## How the SMART verdict is reached
+
+**SMART raw values are vendor-defined and this role does not judge on them.** smartctl(8)
+is explicit: "starting with ATA/ATAPI-4, revision 4, the meaning of these Attribute fields
+has been made entirely vendor-specific."
+
+This is not theoretical. The first real run of this role FAILED `k3s-prod-1`'s
+Crucial/Micron C400 (`C400-MTFDDAC512MAM`, fw `070H`) on `reallocated=61440`, read as a
+plain bad-sector count. Micron's own TN-FD-20 *SMART Attributes for the C400 SSD* defines
+attribute 5 not as a sector count but as **Reallocated Block Count**, whose raw is "the
+grown NAND Flash bad-block count multiplied by the Block_Sector_Count, which is 256 x 8 for
+64Gb NAND ... or 256 x 16 for 128Gb NAND (256GB and 512GB)". For this 512GB drive that
+multiplier is 256 x 16 = 4096, and 61440 / 4096 = **15** — corroborated on the same drive by
+attribute 170 `Grown_Failing_Block_Ct` = 15 and attribute 196 `Reallocated_Event_Count` = 15.
+Fifteen grown bad blocks, not sixty-one thousand bad sectors. The verdict was wrong by a
+factor of 4096 because it assumed a unit the ATA spec does not guarantee.
+
+### What the check judges on instead
+
+Vendor-**independent** signals only, each of which means the same thing on every drive:
+
+1. **The drive's own overall-health self-assessment** (`smart_status.passed`) — FAIL.
+2. **Normalised VALUE vs the MANUFACTURER'S OWN THRESHOLD**, taken from smartctl's
+   `when_failed` field (`"now"` / `"past"` / `""`). Per smartctl(8), `FAILING_NOW` means the
+   normalised value is at or below the threshold *the manufacturer set for that attribute on
+   that model*; a dash means it is OK now and has never failed. This needs no per-vendor
+   knowledge from us — the vendor already encoded the answer. Pre-fail failing now → FAIL;
+   Old_age failing now, or anything that failed in the past → WARN.
+3. **ATA-standard error counters** whose raw meaning is a plain event count everywhere:
+   197 pending, 198 offline-uncorrectable, 187 reported-uncorrectable → WARN/FAIL;
+   199 CRC (SATA link, not media) → WARN only.
+4. **NVMe's structured health log** — `critical_warning`, `media_errors`, `percentage_used`.
+5. **Wear from the NORMALISED lifetime indicator** (ATA 202/231/233 `value`, or NVMe
+   `percentage_used`) — a percentage by definition, and therefore portable where raw wear
+   counts are not.
+
+### What is reported but never produces a FAIL
+
+Vendor raw counts are real information and are all printed with their numbers — grown
+bad-block count, reallocation events (196), erase/program failures, unexpected power losses,
+and attribute 5's raw shown **beside** the normalised reading that contradicts it. They can
+raise a **WARN**, never a FAIL, because their units are vendor-defined.
+
+The endurance counters live in the vendor-specific attribute-ID range (170-174), so they are
+matched by **the name smartctl's drive database resolved for that model**, not by a bare ID:
+`drivedb.h` is smartmontools' own per-(model, firmware) translation, so keying off its
+resolved name borrows that portability instead of hardcoding one vendor's layout. On a drive
+drivedb does not recognise, the name is `Unknown_Attribute`, nothing matches, and the field
+stays `null` — no claim is made either way.
+
+### Two honesty traps this check does not fall into
+
+- **Zero is not a clean bill of health.** Micron TN-FD-20 states attribute 197 on the C400
+  "will always be 0, as reallocation will be done on-the-fly". A non-zero 197 is a finding;
+  a zero one proves nothing. The role reports the number and does not infer health from it.
+- **A quiet drive is not necessarily a good drive.** `overall-health=PASSED` with every
+  threshold un-tripped can still sit alongside a third of rated life consumed, grown bad
+  blocks, erase failures and unprotected power losses — which is exactly `k3s-prod-1`. Those
+  produce a WARN *with the numbers*, because a check that called that drive "fine" would be
+  worse than the false FAIL it replaced.
+
 ## Known limits
 
-- Without root: **no SMART** (wear, power-on hours, reallocated/pending sectors, drive
+- Without root: **no SMART** (wear, power-on hours, error counters, drive
   temperature) and no DMI serial numbers. On prod that means creating
   `inventory/group_vars/k3s_prod/vault.yml` (see that directory's README) and running
   `node-baseline` so `smartmontools` exists — in that order.
 - EDAC reports nothing on platforms with no ECC memory controller bound. That is
   "ECC status unknown", not "no errors", and the role says so.
+- SMART wear and endurance context depends on smartmontools' `drivedb.h` knowing the drive.
+  On an unrecognised model the vendor attributes are `Unknown_Attribute`; the portable
+  signals (overall health, normalised-vs-threshold, 197/198/187/199, NVMe log) still apply,
+  but the wear/endurance context SKIPs rather than guessing.
 - `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or
   unsupported commands during discovery. Attribution needs `dmesg`, which needs root.

infra/ansible/roles/node_health/defaults/main.yml

@@ -60,13 +60,65 @@ node_health_thresholds:
   # collector looks behind the 127.0.0.53 stub rather than counting it as one server.
   # The fix side of this is the `dns`-tagged block in roles/node_baseline.
   min_nameservers: 2
-  # SMART (only evaluated when smartctl actually ran)
+  # --- SMART (only evaluated when smartctl actually ran) -----------------------------
-  smart_reallocated_warn: 1
+  #
-  smart_reallocated_fail: 50
+  # WHY THESE THRESHOLDS AND NOT "reallocated sectors" ANY MORE.
+  # The first real run of this role FAILED k3s-prod-1's Crucial/Micron C400 on
+  # `reallocated=61440`, read as a plain bad-sector count. That verdict was WRONG, and it
+  # was wrong for a structural reason, not a tuning one: SMART *raw* values are vendor
+  # defined. smartctl(8) says so outright -- "starting with ATA/ATAPI-4, revision 4, the
+  # meaning of these Attribute fields has been made entirely vendor-specific". Micron's own
+  # TN-FD-20 (SMART Attributes for the C400 SSD) defines ID 5 not as a sector count but as
+  # "Reallocated Block Count", raw = "the grown NAND Flash bad-block count multiplied by the
+  # Block_Sector_Count, which is 256 x 8 for 64Gb NAND ... or 256 x 16 for 128Gb NAND (256GB
+  # and 512GB)". 256 x 16 = 4096, and 61440 / 4096 = 15 -- corroborated independently by
+  # ID 170 Grown_Failing_Block_Ct = 15 and ID 196 Reallocated_Event_Count = 15. Fifteen
+  # grown bad blocks, not sixty-one thousand bad sectors.
+  #
+  # So no threshold below is ever applied to attribute 5's raw value. The verdict comes from
+  # the signals smartctl itself defines as vendor-INDEPENDENT:
+  #   * the drive's own overall-health self-assessment (smart_status.passed);
+  #   * NORMALISED value vs THRESHOLD, i.e. smartctl's when_failed ("now"/"past"), which is
+  #     the manufacturer's OWN trip point for that attribute on that drive;
+  #   * the ATA-standard error counters whose raw meaning is a plain event count everywhere:
+  #     197 pending, 198 offline-uncorrectable, 187 reported-uncorrectable, 199 CRC;
+  #   * NVMe's structured health log (critical_warning, media_errors, percentage_used).
+  # Vendor raw counts are still collected and printed -- they are real information -- but
+  # they can only ever raise a WARN with the number shown, never a FAIL on their own.
+  #
+  # Hard-error counters (197/198/187 raw, NVMe media_errors). A plain event count on every
+  # drive that implements them. NOTE the converse is not evidence: some drives (the C400
+  # among them, per TN-FD-20) hard-code 197 to 0 because reallocation happens on the fly, so
+  # "pending = 0" proves nothing. Nonzero is a finding; zero is not a clean bill of health.
   smart_pending_warn: 1
   smart_pending_fail: 10
+  smart_uncorrectable_warn: 1
+  smart_uncorrectable_fail: 10
+  # 199 UDMA_CRC_Error_Count -- SATA link/cable integrity, not media. WARN only: it is a
+  # lifetime count that never resets, so a historic cable reseat should not FAIL a node.
+  smart_crc_warn: 1
+  # Wear, from the NORMALISED lifetime indicator (ATA 202/231/233 value, or NVMe
+  # percentage_used) -- a percentage by definition, so portable in a way raw wear counts
+  # are not.  ..._remaining_* are the "this drive is nearly used up" bands; ..._used_pct_warn
+  # is the softer "this is not a fresh drive, know that before you put etcd on it" notice
+  # that PROD-PROVISIONING.md Section 2 actually needs.
   smart_lifetime_remaining_warn_pct: 20
   smart_lifetime_remaining_fail_pct: 10
+  smart_life_used_pct_warn: 25
+  # Reallocation ACTIVITY. 196 Reallocated_Event_Count is an ATA-standard *event* count
+  # (one per remap operation) and is the portable stand-in for attribute 5's unportable raw.
+  # WARN-only by design: remapping is what a drive is supposed to do, and how many events
+  # are "too many" is drive-specific -- the manufacturer already encodes that answer in
+  # attribute 5's normalised threshold, which the when_failed check above honours.
+  smart_realloc_events_warn: 1
+  # Endurance-stress counters, matched by the NAME smartctl's drive database resolved for
+  # this model rather than by a bare vendor ID -- drivedb IS the portability layer. On a
+  # drive smartctl does not recognise these simply do not match and nothing is claimed.
+  smart_erase_fail_warn: 1
+  smart_program_fail_warn: 1
+  # Unexpected power loss on a consumer SSD with no power-loss protection: each one is a
+  # chance of a torn/lost in-flight write. Directly relevant to etcd's fsync contract.
+  smart_unexpected_power_loss_warn: 10
   smart_temp_warn_c: 55
   smart_temp_fail_c: 70
   # Power-on hours past which a consumer SSD is simply old. Informational — age alone is
@@ -76,6 +128,11 @@ node_health_thresholds:
 
 # ATA SMART attribute IDs surfaced verbatim in the report. Vendors disagree on names, so
 # the report prints whatever the drive calls them and the checks key off the IDs.
-node_health_smart_attribute_ids: [5, 9, 177, 179, 181, 182, 183, 184, 187, 188, 190,
+# 170-174 and 189 are the Micron/Crucial (and, by convention, most SSD controllers')
-                                  194, 195, 196, 197, 198, 199, 202, 231, 232, 233,
+# endurance and grown-bad-block counters; they are the CONTEXT that made the C400 verdict
-                                  241, 242, 246]
+# above provable, and leaving them out of the report is what let a single misread raw value
+# stand unchallenged. 1 and 12 are included so the error-rate and power-cycle baselines are
+# visible next to them.
+node_health_smart_attribute_ids: [1, 5, 9, 12, 170, 171, 172, 173, 174, 177, 179, 181,
+                                  182, 183, 184, 187, 188, 189, 190, 194, 195, 196, 197,
+                                  198, 199, 202, 206, 231, 232, 233, 241, 242, 246]

infra/ansible/roles/node_health/tasks/assess.yml

@@ -57,6 +57,14 @@
       'detail': node_health_root_skip_reason }] }}"
 
 # --- Storage ------------------------------------------------------------------------
+# SMART VERDICTS — READ defaults/main.yml's SMART threshold block BEFORE EDITING THIS.
+# Short version: this check judges a drive on the signals smartctl(8) itself defines as
+# vendor-independent (the drive's own overall-health verdict; NORMALISED value vs the
+# MANUFACTURER'S OWN threshold, via smartctl's `when_failed`; and the ATA-standard error
+# counters 197/198/187/199 whose raw meaning is a plain event count everywhere, plus NVMe's
+# structured health log). Vendor raw counts are reported with their numbers and can raise a
+# WARN, but they can never produce a FAIL — because their units are vendor-defined and
+# reading one as if it were a portable sector count is exactly the defect this replaces.
 - name: Assess — SMART per physical disk
   ansible.builtin.set_fact:
     node_health_checks: "{{ node_health_checks + [{
@@ -71,22 +79,80 @@
     ansible_become: false
     th: "{{ node_health_thresholds }}"
     disk_smart: "{{ (node_health_smart | selectattr('path', 'equalto', item.path) | list | first) | default({}) }}"
-    # Sub-verdicts. Each is SKIP unless the number was actually read.
+
+    # --- Sub-verdicts. Each is SKIP unless the number behind it was actually read; a SKIP is
+    # an honest "unknown" and never contributes a PASS. ------------------------------------
+
+    # 1. The drive's own overall-health self-assessment. Vendor-independent by construction.
     s_health: >-
       {{ 'SKIP' if (disk_smart.passed is not defined or disk_smart.passed is none)
          else ('PASS' if disk_smart.passed else 'FAIL') }}
-    s_realloc: >-
+    # 2. NVMe critical warning bitfield (spare below threshold, media in read-only, etc.).
-      {{ 'SKIP' if (disk_smart.reallocated_sectors is not defined or disk_smart.reallocated_sectors is not number)
+    s_nvme_crit: >-
-         else ('FAIL' if disk_smart.reallocated_sectors | int >= th.smart_reallocated_fail
+      {{ 'SKIP' if (disk_smart.nvme_critical_warning is not defined or disk_smart.nvme_critical_warning is not number)
-               else ('WARN' if disk_smart.reallocated_sectors | int >= th.smart_reallocated_warn else 'PASS')) }}
+         else ('PASS' if disk_smart.nvme_critical_warning | int == 0 else 'FAIL') }}
+    # 3. THE portable per-attribute failure signal: normalised value at or below the
+    #    threshold the MANUFACTURER set for that attribute on that model. Pre-fail failing
+    #    now = FAIL. Old_age failing now, or anything that failed in the past, = WARN.
+    s_thresh: >-
+      {{ 'SKIP' if not (disk_smart.thresholds_present | default(false))
+         else ('FAIL' if (disk_smart.failing_now_prefail | default([]) | length) > 0
+               else ('WARN' if ((disk_smart.failing_now | default([]) | length) > 0
+                                or (disk_smart.failed_past | default([]) | length) > 0)
+                     else 'PASS')) }}
+    # 4. ATA-standard hard-error counters. 197 pending; 198/187 + NVMe media_errors are
+    #    uncorrectable reads — data the drive could NOT return. Same meaning on every drive.
     s_pending: >-
       {{ 'SKIP' if (disk_smart.pending_sectors is not defined or disk_smart.pending_sectors is not number)
          else ('FAIL' if disk_smart.pending_sectors | int >= th.smart_pending_fail
                else ('WARN' if disk_smart.pending_sectors | int >= th.smart_pending_warn else 'PASS')) }}
+    uncorr_values: >-
+      {{ [disk_smart.offline_uncorrectable | default(none),
+          disk_smart.reported_uncorrect | default(none),
+          disk_smart.nvme_media_errors | default(none)] | select('number') | list }}
+    s_uncorr: >-
+      {{ 'SKIP' if (uncorr_values | length) == 0
+         else ('FAIL' if (uncorr_values | max) >= th.smart_uncorrectable_fail
+               else ('WARN' if (uncorr_values | max) >= th.smart_uncorrectable_warn else 'PASS')) }}
+    # 5. 199 CRC — SATA link integrity (cable/backplane), not media. WARN only: the counter
+    #    never resets, so one historic bad seating must not condemn the node forever.
+    s_crc: >-
+      {{ 'SKIP' if (disk_smart.crc_errors is not defined or disk_smart.crc_errors is not number)
+         else ('WARN' if disk_smart.crc_errors | int >= th.smart_crc_warn else 'PASS') }}
+    # 6. Wear, from the NORMALISED lifetime indicator — a percentage by definition.
+    life_rem: "{{ disk_smart.lifetime_remaining_pct | default(none) }}"
+    life_used: "{{ (100 - (life_rem | int)) if (life_rem is number) else none }}"
     s_life: >-
-      {{ 'SKIP' if (disk_smart.lifetime_remaining_pct is not defined or disk_smart.lifetime_remaining_pct is not number)
+      {{ 'SKIP' if (life_rem is not number)
-         else ('FAIL' if disk_smart.lifetime_remaining_pct | int <= th.smart_lifetime_remaining_fail_pct
+         else ('FAIL' if life_rem | int <= th.smart_lifetime_remaining_fail_pct
-               else ('WARN' if disk_smart.lifetime_remaining_pct | int <= th.smart_lifetime_remaining_warn_pct else 'PASS')) }}
+               else ('WARN' if (life_rem | int <= th.smart_lifetime_remaining_warn_pct
+                                or life_used | int >= th.smart_life_used_pct_warn)
+                     else 'PASS')) }}
+    # 7. Reallocation ACTIVITY. Portable evidence first (196 event count, and the grown
+    #    bad-block counter smartctl's drivedb named for this model). Only if neither exists
+    #    do we fall back to attribute 5's raw — and then ONLY to say "nonzero, so something
+    #    was remapped", which is portable, never to interpret its magnitude, which is not.
+    remap_evidence: >-
+      {{ [disk_smart.realloc_events | default(none),
+          disk_smart.grown_bad_blocks | default(none)] | select('number') | list }}
+    realloc_raw: "{{ (disk_smart.realloc_attr | default({}, true)).get('raw', {}).get('value', none) }}"
+    s_remap: >-
+      {{ ('WARN' if (remap_evidence | max) >= th.smart_realloc_events_warn else 'PASS')
+           if (remap_evidence | length) > 0
+         else (('WARN' if realloc_raw | int > 0 else 'PASS') if realloc_raw is number else 'SKIP') }}
+    # 8. Endurance stress — erase/program failures are NAND wearing out under the FTL.
+    s_erase: >-
+      {{ 'SKIP' if (disk_smart.erase_fail_count is not defined or disk_smart.erase_fail_count is not number)
+         else ('WARN' if disk_smart.erase_fail_count | int >= th.smart_erase_fail_warn else 'PASS') }}
+    s_program: >-
+      {{ 'SKIP' if (disk_smart.program_fail_count is not defined or disk_smart.program_fail_count is not number)
+         else ('WARN' if disk_smart.program_fail_count | int >= th.smart_program_fail_warn else 'PASS') }}
+    # 9. Unexpected power loss. On a consumer SSD with no power-loss protection each one is
+    #    an opportunity for a torn or lost in-flight write — which is precisely the guarantee
+    #    etcd's fsync contract depends on. Directly relevant to PROD-PROVISIONING.md Section 2.
+    s_ploss: >-
+      {{ 'SKIP' if (disk_smart.unexpected_power_loss is not defined or disk_smart.unexpected_power_loss is not number)
+         else ('WARN' if disk_smart.unexpected_power_loss | int >= th.smart_unexpected_power_loss_warn else 'PASS') }}
     s_temp: >-
       {{ 'SKIP' if (disk_smart.temperature_c is not defined or disk_smart.temperature_c is not number)
          else ('FAIL' if disk_smart.temperature_c | int >= th.smart_temp_fail_c
@@ -94,8 +160,82 @@
     # Worst sub-verdict wins. `select('in', ...)` preserves the order of the left-hand
     # list, so the LAST survivor is the most severe present.
     smart_status: >-
-      {{ (['PASS', 'WARN', 'FAIL'] | select('in', [s_health, s_realloc, s_pending, s_life, s_temp]) | list | last)
+      {{ (['PASS', 'WARN', 'FAIL'] | select('in', [s_health, s_nvme_crit, s_thresh, s_pending,
+           s_uncorr, s_crc, s_life, s_remap, s_erase, s_program, s_ploss, s_temp]) | list | last)
          | default('SKIP', true) }}
+
+    # --- Detail. Every WARN/FAIL must arrive WITH the number that caused it; a verdict the
+    # operator cannot check is not much better than a wrong one. -------------------------
+    smart_findings: >-
+      {%- set n = namespace(l=[]) -%}
+      {%- if s_health == 'FAIL' -%}
+      {%- set _ = n.l.append('the drive reports its OWN overall-health self-assessment as FAILED') -%}
+      {%- endif -%}
+      {%- if s_nvme_crit == 'FAIL' -%}
+      {%- set _ = n.l.append('NVMe critical_warning=' ~ disk_smart.nvme_critical_warning ~ ' (non-zero)') -%}
+      {%- endif -%}
+      {%- for a in (disk_smart.failing_now_prefail | default([])) -%}
+      {%- set _ = n.l.append('Pre-fail attribute ' ~ a.id ~ ' ' ~ a.name ~ ' is AT OR BELOW its manufacturer threshold NOW (value='
+                             ~ a.value ~ ', thresh=' ~ a.thresh ~ ')') -%}
+      {%- endfor -%}
+      {%- for a in (disk_smart.failed_past | default([])) -%}
+      {%- set _ = n.l.append('attribute ' ~ a.id ~ ' ' ~ a.name ~ ' crossed its manufacturer threshold IN THE PAST (worst='
+                             ~ a.worst ~ ', thresh=' ~ a.thresh ~ ')') -%}
+      {%- endfor -%}
+      {%- if s_pending in ['WARN', 'FAIL'] -%}
+      {%- set _ = n.l.append('current pending sectors (197)=' ~ disk_smart.pending_sectors) -%}
+      {%- endif -%}
+      {%- if s_uncorr in ['WARN', 'FAIL'] -%}
+      {%- set _ = n.l.append('UNCORRECTABLE read errors — offline(198)=' ~ (disk_smart.offline_uncorrectable if disk_smart.offline_uncorrectable is number else 'n/a')
+                             ~ ', reported(187)=' ~ (disk_smart.reported_uncorrect if disk_smart.reported_uncorrect is number else 'n/a')
+                             ~ ', nvme media_errors=' ~ (disk_smart.nvme_media_errors if disk_smart.nvme_media_errors is number else 'n/a')) -%}
+      {%- endif -%}
+      {%- if s_crc == 'WARN' -%}
+      {%- set _ = n.l.append('SATA link CRC errors (199)=' ~ disk_smart.crc_errors ~ ' — cable/backplane, not media') -%}
+      {%- endif -%}
+      {%- if s_life in ['WARN', 'FAIL'] -%}
+      {%- set _ = n.l.append(life_used ~ '% of rated life CONSUMED (' ~ life_rem
+                             ~ '% remaining, from the normalised wear indicator)') -%}
+      {%- endif -%}
+      {%- if s_remap == 'WARN' and (remap_evidence | length) > 0 -%}
+      {%- set _ = n.l.append((remap_evidence | max) ~ ' grown bad block(s)/reallocation event(s)'
+                             ~ (' [196 events=' ~ disk_smart.realloc_events ~ ']' if disk_smart.realloc_events is number else '')
+                             ~ (' [grown failing blocks=' ~ disk_smart.grown_bad_blocks ~ ']' if disk_smart.grown_bad_blocks is number else '')) -%}
+      {%- elif s_remap == 'WARN' -%}
+      {%- set _ = n.l.append('the drive reports a non-zero reallocation attribute (5 raw=' ~ realloc_raw
+                             ~ '); its UNIT is vendor-defined, so the count is not interpreted here') -%}
+      {%- endif -%}
+      {%- if s_erase == 'WARN' -%}
+      {%- set _ = n.l.append('NAND erase failures=' ~ disk_smart.erase_fail_count) -%}
+      {%- endif -%}
+      {%- if s_program == 'WARN' -%}
+      {%- set _ = n.l.append('NAND program failures=' ~ disk_smart.program_fail_count) -%}
+      {%- endif -%}
+      {%- if s_ploss == 'WARN' -%}
+      {%- set _ = n.l.append('unexpected power losses=' ~ disk_smart.unexpected_power_loss
+                             ~ ' — a consumer SSD has no power-loss protection, so each one risks a torn or lost in-flight write') -%}
+      {%- endif -%}
+      {%- if s_temp in ['WARN', 'FAIL'] -%}
+      {%- set _ = n.l.append('temperature=' ~ disk_smart.temperature_c ~ 'C') -%}
+      {%- endif -%}
+      {{ n.l | join('; ') }}
+    # The raw value that started this. Shown ALWAYS when it is non-zero and contradicts the
+    # normalised reading, explicitly labelled as vendor-encoded and explicitly excluded from
+    # the verdict — so the next operator to see 61440 sees why it is not 61440 bad sectors.
+    realloc_raw_note: >-
+      {%- if realloc_raw is number and realloc_raw | int > 0 -%}
+      {%- set a = disk_smart.realloc_attr -%}
+      attribute 5 ({{ a.name }}) raw={{ realloc_raw }} normalised value={{ a.value }} vs thresh={{ a.thresh }}
+      {%- if (remap_evidence | length) > 0 and (remap_evidence | max) > 0
+             and realloc_raw | int != (remap_evidence | max) | int
+             and realloc_raw | int % (remap_evidence | max) | int == 0 -%}
+      {{ ' (= ' ~ (remap_evidence | max) ~ ' x ' ~ (realloc_raw | int // (remap_evidence | max) | int) ~ ')' }}
+      {%- endif -%}
+      . SMART raw values are VENDOR-DEFINED (smartctl(8): "entirely vendor-specific") and this
+      one is NOT a portable bad-sector count — it is reported as context and is NOT used for
+      the verdict. The normalised value/threshold pair beside it is the manufacturer's own
+      trip point and IS.
+      {%- endif -%}
     smart_detail: >-
       {{ ('not read — ' ~ node_health_root_skip_reason)
            if not (node_health_root_available | bool)
@@ -108,10 +248,19 @@
            if not (disk_smart.available | default(false))
          else ('overall-health=' ~ ('PASSED' if disk_smart.passed else 'FAILED')
                ~ ', power_on_hours=' ~ (disk_smart.power_on_hours | default('?'))
-               ~ ', lifetime_remaining=' ~ (disk_smart.lifetime_remaining_pct | default('?')) ~ '%'
+               ~ ', power_cycles=' ~ (disk_smart.power_cycles | default('?'))
-               ~ ', reallocated=' ~ (disk_smart.reallocated_sectors | default('?'))
+               ~ ', life_remaining=' ~ ((life_rem | string ~ '%') if life_rem is number else 'not reported')
-               ~ ', pending=' ~ (disk_smart.pending_sectors | default('?'))
+               ~ ', pending(197)=' ~ (disk_smart.pending_sectors if disk_smart.pending_sectors is number else 'n/a')
-               ~ ', temp=' ~ (disk_smart.temperature_c | default('?')) ~ 'C') }}
+               ~ ', offline_uncorrectable(198)=' ~ (disk_smart.offline_uncorrectable if disk_smart.offline_uncorrectable is number else 'n/a')
+               ~ ', reported_uncorrect(187)=' ~ (disk_smart.reported_uncorrect if disk_smart.reported_uncorrect is number else 'n/a')
+               ~ ', crc(199)=' ~ (disk_smart.crc_errors if disk_smart.crc_errors is number else 'n/a')
+               ~ (('. NO attribute is at or below its manufacturer threshold; closest is '
+                   ~ disk_smart.prefail_closest.id ~ ' ' ~ disk_smart.prefail_closest.name
+                   ~ ' value=' ~ disk_smart.prefail_closest.value
+                   ~ ' vs thresh=' ~ disk_smart.prefail_closest.thresh)
+                  if (s_thresh == 'PASS' and (disk_smart.prefail_closest | default(none)) is mapping) else '')
+               ~ (('. FINDINGS: ' ~ smart_findings) if smart_findings | trim else '')
+               ~ (('. CONTEXT: ' ~ realloc_raw_note) if realloc_raw_note | trim else '')) }}
 
 - name: Assess — kernel-visible I/O errors per disk (no root needed)
   ansible.builtin.set_fact:

infra/ansible/roles/node_health/tasks/collect_smart.yml

@@ -43,12 +43,79 @@
     smart_json: "{{ (smart_result.stdout | default('') | trim) | default('{}', true) | from_json }}"
     smart_attrs: "{{ smart_json.ata_smart_attributes.table | default([]) }}"
     smart_nvme: "{{ smart_json.nvme_smart_health_information_log | default({}) }}"
-    # Normalised ("value") wear indicators, in vendor-preference order. Micron/Crucial use
+
-    # 202 Percent_Lifetime_Remaining; many others use 231 SSD_Life_Left or 233. The RAW
+    # --- VENDOR-INDEPENDENT FAILURE SIGNAL: normalised value vs the drive's own threshold.
-    # value is kept alongside in `attributes` so nothing is hidden by this normalisation.
+    # smartctl computes this for us and emits it as `when_failed`, per smartctl(8): "If the
+    # Attribute's current Normalized value is less than or equal to the threshold value, then
+    # the WHEN_FAILED column will display FAILING_NOW. If not, but the worst recorded value is
+    # less than or equal to the threshold value, then this column will display In_the_past. If
+    # the WHEN_FAILED column has no entry ... then this Attribute is OK now (not failing) and
+    # has also never failed in the past." In JSON those become "now" / "past" / "". We match
+    # case-insensitively on 'past' rather than on an exact token so the same code reads either
+    # the JSON spelling or the plain-text one if a future smartctl changes it.
+    #
+    # This is the ONLY comparison in this role that can condemn a drive on an attribute the
+    # role does not itself understand, and that is the point: the manufacturer picked the
+    # threshold, for that attribute, on that model. It needs no per-vendor knowledge from us.
+    smart_failed_attrs: >-
+      {{ smart_attrs | selectattr('when_failed', 'defined')
+         | rejectattr('when_failed', 'equalto', '') | list }}
+    smart_failed_past: "{{ smart_failed_attrs | selectattr('when_failed', 'search', '(?i)past') | list }}"
+    smart_failing_now: "{{ smart_failed_attrs | rejectattr('when_failed', 'search', '(?i)past') | list }}"
+    # Of the failing ones, which are Pre-fail (imminent failure) vs Old_age (design life
+    # exceeded)? smartctl decodes that flag from the drive too, so this stays vendor-neutral.
+    smart_failing_now_prefail: "{{ smart_failing_now | selectattr('flags.prefailure', 'defined') | selectattr('flags.prefailure') | list }}"
+    # How close is the drive to ANY of its own trip points? The smallest (value - thresh)
+    # across all Pre-fail attributes that actually carry a threshold (thresh 0 = "cannot
+    # trip", per ATA, so it is excluded rather than counted as a margin of `value`).
+    # Reported as context; a shrinking margin is the early warning the raw values cannot give.
+    smart_prefail_margin_json: >-
+      {%- set ns = namespace(best=none) -%}
+      {%- for a in smart_attrs -%}
+      {%- if (a.flags.prefailure | default(false)) and ((a.thresh | default(0)) | int) > 0 -%}
+      {%- set m = ((a.value | default(0)) | int) - ((a.thresh) | int) -%}
+      {%- if ns.best is none or m < ns.best.margin -%}
+      {%- set ns.best = {'id': a.id, 'name': a.name, 'value': (a.value | int),
+                         'worst': ((a.worst | default(a.value)) | int),
+                         'thresh': (a.thresh | int), 'margin': m} -%}
+      {%- endif -%}
+      {%- endif -%}
+      {%- endfor -%}
+      {{ ns.best | to_json }}
+
+    # --- Normalised ("value") wear indicators, in vendor-preference order. Micron/Crucial use
+    # 202 Perc_Rated_Life_Used; many others use 231 SSD_Life_Left or 233. All three express
+    # the SAME thing in the normalised field — a percentage of rated life REMAINING — which is
+    # exactly why the normalised field is usable across vendors where their raws are not.
     smart_life_202: "{{ smart_attrs | selectattr('id', 'equalto', 202) | map(attribute='value') | first | default(none) }}"
     smart_life_231: "{{ smart_attrs | selectattr('id', 'equalto', 231) | map(attribute='value') | first | default(none) }}"
     smart_life_233: "{{ smart_attrs | selectattr('id', 'equalto', 233) | map(attribute='value') | first | default(none) }}"
+
+    # --- VENDOR CONTEXT. Everything below is collected and printed but NEVER produces a FAIL.
+    # Attribute 5 is kept in full — id, drive's own name, normalised value/worst/thresh AND
+    # raw — precisely so the report can show the raw next to the normalised reading that
+    # contradicts it, instead of silently dropping the number that started this.
+    smart_attr5: "{{ (smart_attrs | selectattr('id', 'equalto', 5) | list | first) | default(none) }}"
+    # 196 Reallocated_Event_Count — an ATA-standard *event* count (one tick per remap
+    # operation), the portable stand-in for attribute 5's unportable raw.
+    smart_realloc_events: "{{ smart_attrs | selectattr('id', 'equalto', 196) | map(attribute='raw.value') | first | default(none) }}"
+    # Endurance counters matched by the NAME smartctl's drive database resolved for this
+    # model, not by a bare vendor attribute ID. IDs 170-174 are in the vendor-specific range
+    # and mean different things on different controllers; drivedb is smartmontools' own
+    # per-(model, firmware) translation of them, so matching its resolved name borrows that
+    # knowledge instead of hardcoding one vendor's. On a drive drivedb does not recognise the
+    # name is "Unknown_Attribute", nothing matches, and these stay null — no claim is made.
+    smart_grown_blocks: >-
+      {{ smart_attrs | selectattr('name', 'search', '(?i)(grown|runtime)[_ -]?(failing|bad)?[_ -]?block')
+         | map(attribute='raw.value') | first | default(none) }}
+    smart_erase_fail: "{{ smart_attrs | selectattr('name', 'search', '(?i)erase[_ -]?fail') | map(attribute='raw.value') | first | default(none) }}"
+    smart_program_fail: "{{ smart_attrs | selectattr('name', 'search', '(?i)program[_ -]?fail|prog[_ -]?fail') | map(attribute='raw.value') | first | default(none) }}"
+    # '(?i)power[_ -]?loss[_ -]?(ct|cnt|count)' deliberately does NOT match Intel's
+    # Power_Loss_Cap_Test (a capacitor self-test result, not a count of power losses).
+    smart_power_loss: >-
+      {{ smart_attrs | selectattr('name', 'search', '(?i)(unexpect\w*|unsafe)[_ -]?(power|shutdown)|power[_ -]?loss[_ -]?(ct|cnt|count)')
+         | map(attribute='raw.value') | first | default(none) }}
+
     smart_disk:
       path: "{{ smart_result.item.path }}"
       # smartctl ran, but did it understand the device?
@@ -64,11 +131,17 @@
       power_on_hours: "{{ smart_json.power_on_time.hours | default(none) }}"
       power_cycles: "{{ smart_json.power_cycle_count | default(none) }}"
       temperature_c: "{{ smart_json.temperature.current | default(none) }}"
-      reallocated_sectors: "{{ smart_attrs | selectattr('id', 'equalto', 5) | map(attribute='raw.value') | first | default(none) }}"
+      # Normalised-vs-threshold verdict inputs (see the long note above).
+      failing_now: "{{ smart_failing_now }}"
+      failing_now_prefail: "{{ smart_failing_now_prefail }}"
+      failed_past: "{{ smart_failed_past }}"
+      thresholds_present: "{{ (smart_attrs | selectattr('thresh', 'defined') | rejectattr('thresh', 'equalto', 0) | list | length) > 0 }}"
+      prefail_closest: "{{ smart_prefail_margin_json | from_json }}"
+      # ATA-standard error counters — same meaning on every drive that implements them.
       pending_sectors: "{{ smart_attrs | selectattr('id', 'equalto', 197) | map(attribute='raw.value') | first | default(none) }}"
       offline_uncorrectable: "{{ smart_attrs | selectattr('id', 'equalto', 198) | map(attribute='raw.value') | first | default(none) }}"
-      crc_errors: "{{ smart_attrs | selectattr('id', 'equalto', 199) | map(attribute='raw.value') | first | default(none) }}"
       reported_uncorrect: "{{ smart_attrs | selectattr('id', 'equalto', 187) | map(attribute='raw.value') | first | default(none) }}"
+      crc_errors: "{{ smart_attrs | selectattr('id', 'equalto', 199) | map(attribute='raw.value') | first | default(none) }}"
       host_writes_raw: "{{ smart_attrs | selectattr('id', 'equalto', 241) | map(attribute='raw.value') | first | default(none) }}"
       # NVMe reports wear directly as "percentage_used"; ATA SSDs report a countdown.
       lifetime_remaining_pct: >-
@@ -77,6 +150,18 @@
                  else (smart_life_231 if smart_life_231 is not none else smart_life_233)) }}
       nvme_media_errors: "{{ smart_nvme.media_errors | default(none) }}"
       nvme_critical_warning: "{{ smart_nvme.critical_warning | default(none) }}"
+      nvme_unsafe_shutdowns: "{{ smart_nvme.unsafe_shutdowns | default(none) }}"
+      # --- Vendor context. Never a verdict on its own; always printed with its own numbers.
+      realloc_events: "{{ smart_realloc_events }}"
+      grown_bad_blocks: "{{ smart_grown_blocks }}"
+      erase_fail_count: "{{ smart_erase_fail }}"
+      program_fail_count: "{{ smart_program_fail }}"
+      unexpected_power_loss: "{{ smart_power_loss if smart_power_loss is not none else (smart_nvme.unsafe_shutdowns | default(none)) }}"
+      # Attribute 5, whole and labelled, so the raw is visible BESIDE the normalised reading
+      # rather than standing in for it. `reallocated_sectors` is deliberately gone: that name
+      # asserted a unit the ATA spec does not guarantee, and asserting it is what produced the
+      # false FAIL on k3s-prod-1 (see defaults/main.yml).
+      realloc_attr: "{{ smart_attr5 }}"
       # Every attribute the operator asked to see, verbatim, names and all — vendors
       # disagree on naming, so the checks key off IDs and the report shows the drive's
       # own labels.

infra/ansible/roles/node_health/templates/summary.txt.j2

@@ -61,10 +61,23 @@
 {% set sm = (node_health_smart | selectattr('path', 'equalto', d.path) | list | first) | default(none) %}
 {% if sm and sm.available %}
    SMART      overall-health={{ 'PASSED' if sm.passed else 'FAILED' }}  power_on_hours={{ sm.power_on_hours | default('?') }}  power_cycles={{ sm.power_cycles | default('?') }}
-              lifetime_remaining={{ sm.lifetime_remaining_pct | default('?') }}%  temp={{ sm.temperature_c | default('?') }}C
+              lifetime_remaining={{ (sm.lifetime_remaining_pct | string ~ '%') if sm.lifetime_remaining_pct is not none else 'not reported' }}  temp={{ (sm.temperature_c | string ~ 'C') if sm.temperature_c is not none else 'not reported by this drive' }}
-              reallocated={{ sm.reallocated_sectors | default('?') }}  pending={{ sm.pending_sectors | default('?') }}  offline_uncorrectable={{ sm.offline_uncorrectable | default('?') }}  crc_errors={{ sm.crc_errors | default('?') }}
+              pending(197)={{ sm.pending_sectors | default('?') }}  offline_uncorrectable(198)={{ sm.offline_uncorrectable | default('?') }}  reported_uncorrect(187)={{ sm.reported_uncorrect | default('?') }}  crc(199)={{ sm.crc_errors | default('?') }}
+{% if sm.failing_now or sm.failed_past %}
+              THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ a.thresh }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ a.thresh }}]{% endfor %}
+
+{% elif sm.thresholds_present %}
+              no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ sm.prefail_closest.thresh }}{% endif %}
+
+{% endif %}
+              vendor context (NOT used for the verdict — raw units are vendor-defined):
+              reallocation events(196)={{ sm.realloc_events if sm.realloc_events is not none else 'n/a' }}  grown bad blocks={{ sm.grown_bad_blocks if sm.grown_bad_blocks is not none else 'n/a' }}  erase fails={{ sm.erase_fail_count if sm.erase_fail_count is not none else 'n/a' }}  program fails={{ sm.program_fail_count if sm.program_fail_count is not none else 'n/a' }}  unexpected power loss={{ sm.unexpected_power_loss if sm.unexpected_power_loss is not none else 'n/a' }}
+{% if sm.realloc_attr %}
+              attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ sm.realloc_attr.thresh }} (the signal that is judged)
+{% endif %}
 {% for a in sm.attributes | default([]) %}
-              [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}
+              [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %}  <<< when_failed={{ a.when_failed }}{% endif %}
+
 {% endfor %}
 {% else %}
    SMART      NOT READ — {{ (node_health_root_skip_reason if not node_health_root_available else ('smartmontools not installed on this node (ships with roles/node_baseline; this role never installs packages)' if not node_health_facts.tools.smartctl else 'device exposes no SMART data')) | regex_replace('\s+', ' ') | trim }}