From 0b3db6e7784df2da729a58384a83be7676f5f2bd Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 00:40:29 -0500 Subject: [PATCH] =?UTF-8?q?fix(node=5Fhealth):=20SMART=20verdict=20was=20w?= =?UTF-8?q?rong=20by=204096x=20=E2=80=94=20judge=20on=20portable=20signals?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first real SMART run FAILED k3s-prod-1's drive on reallocated=61440. That verdict was WRONG, and the failure mode is instructive: the check trusted a VENDOR-DEFINED raw value as if it were an ATA-standard sector count. Micron TN-FD-20 ("SMART Attributes for the C400 SSD") names attribute 5 "Reallocated BLOCK Count" and defines its raw as grown-bad-blocks x Block_Sector_Count, where Block_Sector_Count is 256 x 16 = 4096 for the 128Gb-NAND parts (256GB/512GB). This is the 512GB C400-MTFDDAC512MAM: 61440 / 4096 = 15. Triply corroborated on the drive itself — attr 170 Grown_Failing_Block_Ct = 15, attr 196 Reallocated_Event_Count = 15, attr 5 normalised value 100 against thresh 10, and `when_failed` empty on every attribute. FIFTEEN grown bad blocks, not sixty-one thousand bad sectors. smartctl(8) states it outright: since ATA/ATAPI-4 rev 4 the meaning of raw Attribute fields is "entirely vendor-specific". So a raw value can never be a portable verdict. FAIL now comes only from signals that mean the same thing on every drive: the drive's own smart_status.passed; NORMALISED value vs the MANUFACTURER's own threshold via smartctl's when_failed ("now"/"past"/""), split Pre-fail -> FAIL and Old_age-or-past -> WARN; the ATA-standard counters 197/198/187 (199 CRC is WARN-only — it is link, not media); and for NVMe critical_warning / media_errors / percentage_used. Wear reads from the NORMALISED lifetime indicator (202/231/233, or NVMe percentage_used) — a percentage by definition. Vendor raw counts are still collected and PRINTED WITH THEIR NUMBERS, and may raise a WARN, but can never drive a FAIL. Attribute 5's raw is now shown beside the normalised reading that contradicts it, labelled vendor-encoded, with a mechanically-derived "(= 15 x 4096)". Endurance attributes are matched by THE NAME smartctl's drivedb resolved for that model, not by bare ID — drivedb is the portability layer; on an unrecognised drive the name is Unknown_Attribute, nothing matches, and the field stays null with no claim made. The STORAGE section now prints 170/172/173/174/189/206, whose previous omission is exactly what let the misread stand unchallenged. Also documented from TN-FD-20: on this family attr 197 "will always be 0, as reallocation will be done on-the-fly" — so pending=0 is NOT evidence of health here. The role reports the number and infers nothing from it. Branches real hardware cannot exercise were verified with a harness rendering the actual role expressions against synthetic smartctl payloads: virtio/no-stdout -> SKIP; NVMe healthy -> PASS; NVMe degraded -> FAIL; a genuinely dying ATA drive (attr 5 normalised 8 <= thresh 10, pending 48) -> FAIL driven by the threshold trip and standard counters, not the raw; unknown-vendor attrs -> PASS with the unnamed counter ignored; failed-in-the-past -> WARN. NEW VERDICTS. k3s-prod-1: 0 FAIL / 4 WARN / 12 PASS / 1 SKIP — SMART is a WARN naming the real findings (31% of rated life consumed, 15 grown bad blocks, 187 erase failures, 32 unexpected power losses on a drive with no power-loss protection). k3s-dev-1: unchanged, all three virtio disks still SKIP with their stated reason. BREAKING (JSON artifact): the fact `reallocated_sectors` is renamed and gone — that name asserted a unit ATA does not guarantee. Replaced by realloc_attr (the whole attribute, normalised + raw), realloc_events, and grown_bad_blocks. Nothing in the repo reads it. Serves: R-ENV-03, R-OBS-04. Co-Authored-By: Claude Opus 5 (1M context) --- infra/ansible/roles/node_health/README.md | 69 ++++++- .../roles/node_health/defaults/main.yml | 69 ++++++- .../roles/node_health/tasks/assess.yml | 175 ++++++++++++++++-- .../roles/node_health/tasks/collect_smart.yml | 95 +++++++++- .../node_health/templates/summary.txt.j2 | 19 +- 5 files changed, 398 insertions(+), 29 deletions(-) diff --git a/infra/ansible/roles/node_health/README.md b/infra/ansible/roles/node_health/README.md index e2eddd8..a18e949 100644 --- a/infra/ansible/roles/node_health/README.md +++ b/infra/ansible/roles/node_health/README.md @@ -39,7 +39,7 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0 | id | what | |---|---| | `access.privilege` | whether `become` works at all (gates the root-only checks) | -| `storage.smart.` | overall-health, power-on hours, lifetime remaining, reallocated + pending sectors, CRC errors, drive temperature — **root only** | +| `storage.smart.` | overall-health, normalised-vs-threshold trips, pending/uncorrectable/CRC counters, lifetime remaining, wear + endurance context, drive temperature — **root only**. See [How the SMART verdict is reached](#how-the-smart-verdict-is-reached) | | `storage.io_errors` | kernel SCSI/ATA `ioerr_cnt` / `iotmo_cnt` per disk (no root needed) | | `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes | | `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` | @@ -79,13 +79,78 @@ Equally verified: `failed_when: false` does **not** catch a missing-sudo-passwor escalation probe uses `ignore_errors` — anything else turns "no vault yet" into a hard, misleading play failure. +## How the SMART verdict is reached + +**SMART raw values are vendor-defined and this role does not judge on them.** smartctl(8) +is explicit: "starting with ATA/ATAPI-4, revision 4, the meaning of these Attribute fields +has been made entirely vendor-specific." + +This is not theoretical. The first real run of this role FAILED `k3s-prod-1`'s +Crucial/Micron C400 (`C400-MTFDDAC512MAM`, fw `070H`) on `reallocated=61440`, read as a +plain bad-sector count. Micron's own TN-FD-20 *SMART Attributes for the C400 SSD* defines +attribute 5 not as a sector count but as **Reallocated Block Count**, whose raw is "the +grown NAND Flash bad-block count multiplied by the Block_Sector_Count, which is 256 x 8 for +64Gb NAND ... or 256 x 16 for 128Gb NAND (256GB and 512GB)". For this 512GB drive that +multiplier is 256 x 16 = 4096, and 61440 / 4096 = **15** — corroborated on the same drive by +attribute 170 `Grown_Failing_Block_Ct` = 15 and attribute 196 `Reallocated_Event_Count` = 15. +Fifteen grown bad blocks, not sixty-one thousand bad sectors. The verdict was wrong by a +factor of 4096 because it assumed a unit the ATA spec does not guarantee. + +### What the check judges on instead + +Vendor-**independent** signals only, each of which means the same thing on every drive: + +1. **The drive's own overall-health self-assessment** (`smart_status.passed`) — FAIL. +2. **Normalised VALUE vs the MANUFACTURER'S OWN THRESHOLD**, taken from smartctl's + `when_failed` field (`"now"` / `"past"` / `""`). Per smartctl(8), `FAILING_NOW` means the + normalised value is at or below the threshold *the manufacturer set for that attribute on + that model*; a dash means it is OK now and has never failed. This needs no per-vendor + knowledge from us — the vendor already encoded the answer. Pre-fail failing now → FAIL; + Old_age failing now, or anything that failed in the past → WARN. +3. **ATA-standard error counters** whose raw meaning is a plain event count everywhere: + 197 pending, 198 offline-uncorrectable, 187 reported-uncorrectable → WARN/FAIL; + 199 CRC (SATA link, not media) → WARN only. +4. **NVMe's structured health log** — `critical_warning`, `media_errors`, `percentage_used`. +5. **Wear from the NORMALISED lifetime indicator** (ATA 202/231/233 `value`, or NVMe + `percentage_used`) — a percentage by definition, and therefore portable where raw wear + counts are not. + +### What is reported but never produces a FAIL + +Vendor raw counts are real information and are all printed with their numbers — grown +bad-block count, reallocation events (196), erase/program failures, unexpected power losses, +and attribute 5's raw shown **beside** the normalised reading that contradicts it. They can +raise a **WARN**, never a FAIL, because their units are vendor-defined. + +The endurance counters live in the vendor-specific attribute-ID range (170-174), so they are +matched by **the name smartctl's drive database resolved for that model**, not by a bare ID: +`drivedb.h` is smartmontools' own per-(model, firmware) translation, so keying off its +resolved name borrows that portability instead of hardcoding one vendor's layout. On a drive +drivedb does not recognise, the name is `Unknown_Attribute`, nothing matches, and the field +stays `null` — no claim is made either way. + +### Two honesty traps this check does not fall into + +- **Zero is not a clean bill of health.** Micron TN-FD-20 states attribute 197 on the C400 + "will always be 0, as reallocation will be done on-the-fly". A non-zero 197 is a finding; + a zero one proves nothing. The role reports the number and does not infer health from it. +- **A quiet drive is not necessarily a good drive.** `overall-health=PASSED` with every + threshold un-tripped can still sit alongside a third of rated life consumed, grown bad + blocks, erase failures and unprotected power losses — which is exactly `k3s-prod-1`. Those + produce a WARN *with the numbers*, because a check that called that drive "fine" would be + worse than the false FAIL it replaced. + ## Known limits -- Without root: **no SMART** (wear, power-on hours, reallocated/pending sectors, drive +- Without root: **no SMART** (wear, power-on hours, error counters, drive temperature) and no DMI serial numbers. On prod that means creating `inventory/group_vars/k3s_prod/vault.yml` (see that directory's README) and running `node-baseline` so `smartmontools` exists — in that order. - EDAC reports nothing on platforms with no ECC memory controller bound. That is "ECC status unknown", not "no errors", and the role says so. +- SMART wear and endurance context depends on smartmontools' `drivedb.h` knowing the drive. + On an unrecognised model the vendor attributes are `Unknown_Attribute`; the portable + signals (overall health, normalised-vs-threshold, 197/198/187/199, NVMe log) still apply, + but the wear/endurance context SKIPs rather than guessing. - `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or unsupported commands during discovery. Attribution needs `dmesg`, which needs root. diff --git a/infra/ansible/roles/node_health/defaults/main.yml b/infra/ansible/roles/node_health/defaults/main.yml index 406412c..7a2fc72 100644 --- a/infra/ansible/roles/node_health/defaults/main.yml +++ b/infra/ansible/roles/node_health/defaults/main.yml @@ -60,13 +60,65 @@ node_health_thresholds: # collector looks behind the 127.0.0.53 stub rather than counting it as one server. # The fix side of this is the `dns`-tagged block in roles/node_baseline. min_nameservers: 2 - # SMART (only evaluated when smartctl actually ran) - smart_reallocated_warn: 1 - smart_reallocated_fail: 50 + # --- SMART (only evaluated when smartctl actually ran) ----------------------------- + # + # WHY THESE THRESHOLDS AND NOT "reallocated sectors" ANY MORE. + # The first real run of this role FAILED k3s-prod-1's Crucial/Micron C400 on + # `reallocated=61440`, read as a plain bad-sector count. That verdict was WRONG, and it + # was wrong for a structural reason, not a tuning one: SMART *raw* values are vendor + # defined. smartctl(8) says so outright -- "starting with ATA/ATAPI-4, revision 4, the + # meaning of these Attribute fields has been made entirely vendor-specific". Micron's own + # TN-FD-20 (SMART Attributes for the C400 SSD) defines ID 5 not as a sector count but as + # "Reallocated Block Count", raw = "the grown NAND Flash bad-block count multiplied by the + # Block_Sector_Count, which is 256 x 8 for 64Gb NAND ... or 256 x 16 for 128Gb NAND (256GB + # and 512GB)". 256 x 16 = 4096, and 61440 / 4096 = 15 -- corroborated independently by + # ID 170 Grown_Failing_Block_Ct = 15 and ID 196 Reallocated_Event_Count = 15. Fifteen + # grown bad blocks, not sixty-one thousand bad sectors. + # + # So no threshold below is ever applied to attribute 5's raw value. The verdict comes from + # the signals smartctl itself defines as vendor-INDEPENDENT: + # * the drive's own overall-health self-assessment (smart_status.passed); + # * NORMALISED value vs THRESHOLD, i.e. smartctl's when_failed ("now"/"past"), which is + # the manufacturer's OWN trip point for that attribute on that drive; + # * the ATA-standard error counters whose raw meaning is a plain event count everywhere: + # 197 pending, 198 offline-uncorrectable, 187 reported-uncorrectable, 199 CRC; + # * NVMe's structured health log (critical_warning, media_errors, percentage_used). + # Vendor raw counts are still collected and printed -- they are real information -- but + # they can only ever raise a WARN with the number shown, never a FAIL on their own. + # + # Hard-error counters (197/198/187 raw, NVMe media_errors). A plain event count on every + # drive that implements them. NOTE the converse is not evidence: some drives (the C400 + # among them, per TN-FD-20) hard-code 197 to 0 because reallocation happens on the fly, so + # "pending = 0" proves nothing. Nonzero is a finding; zero is not a clean bill of health. smart_pending_warn: 1 smart_pending_fail: 10 + smart_uncorrectable_warn: 1 + smart_uncorrectable_fail: 10 + # 199 UDMA_CRC_Error_Count -- SATA link/cable integrity, not media. WARN only: it is a + # lifetime count that never resets, so a historic cable reseat should not FAIL a node. + smart_crc_warn: 1 + # Wear, from the NORMALISED lifetime indicator (ATA 202/231/233 value, or NVMe + # percentage_used) -- a percentage by definition, so portable in a way raw wear counts + # are not. ..._remaining_* are the "this drive is nearly used up" bands; ..._used_pct_warn + # is the softer "this is not a fresh drive, know that before you put etcd on it" notice + # that PROD-PROVISIONING.md Section 2 actually needs. smart_lifetime_remaining_warn_pct: 20 smart_lifetime_remaining_fail_pct: 10 + smart_life_used_pct_warn: 25 + # Reallocation ACTIVITY. 196 Reallocated_Event_Count is an ATA-standard *event* count + # (one per remap operation) and is the portable stand-in for attribute 5's unportable raw. + # WARN-only by design: remapping is what a drive is supposed to do, and how many events + # are "too many" is drive-specific -- the manufacturer already encodes that answer in + # attribute 5's normalised threshold, which the when_failed check above honours. + smart_realloc_events_warn: 1 + # Endurance-stress counters, matched by the NAME smartctl's drive database resolved for + # this model rather than by a bare vendor ID -- drivedb IS the portability layer. On a + # drive smartctl does not recognise these simply do not match and nothing is claimed. + smart_erase_fail_warn: 1 + smart_program_fail_warn: 1 + # Unexpected power loss on a consumer SSD with no power-loss protection: each one is a + # chance of a torn/lost in-flight write. Directly relevant to etcd's fsync contract. + smart_unexpected_power_loss_warn: 10 smart_temp_warn_c: 55 smart_temp_fail_c: 70 # Power-on hours past which a consumer SSD is simply old. Informational — age alone is @@ -76,6 +128,11 @@ node_health_thresholds: # ATA SMART attribute IDs surfaced verbatim in the report. Vendors disagree on names, so # the report prints whatever the drive calls them and the checks key off the IDs. -node_health_smart_attribute_ids: [5, 9, 177, 179, 181, 182, 183, 184, 187, 188, 190, - 194, 195, 196, 197, 198, 199, 202, 231, 232, 233, - 241, 242, 246] +# 170-174 and 189 are the Micron/Crucial (and, by convention, most SSD controllers') +# endurance and grown-bad-block counters; they are the CONTEXT that made the C400 verdict +# above provable, and leaving them out of the report is what let a single misread raw value +# stand unchallenged. 1 and 12 are included so the error-rate and power-cycle baselines are +# visible next to them. +node_health_smart_attribute_ids: [1, 5, 9, 12, 170, 171, 172, 173, 174, 177, 179, 181, + 182, 183, 184, 187, 188, 189, 190, 194, 195, 196, 197, + 198, 199, 202, 206, 231, 232, 233, 241, 242, 246] diff --git a/infra/ansible/roles/node_health/tasks/assess.yml b/infra/ansible/roles/node_health/tasks/assess.yml index d5d3237..3023397 100644 --- a/infra/ansible/roles/node_health/tasks/assess.yml +++ b/infra/ansible/roles/node_health/tasks/assess.yml @@ -57,6 +57,14 @@ 'detail': node_health_root_skip_reason }] }}" # --- Storage ------------------------------------------------------------------------ +# SMART VERDICTS — READ defaults/main.yml's SMART threshold block BEFORE EDITING THIS. +# Short version: this check judges a drive on the signals smartctl(8) itself defines as +# vendor-independent (the drive's own overall-health verdict; NORMALISED value vs the +# MANUFACTURER'S OWN threshold, via smartctl's `when_failed`; and the ATA-standard error +# counters 197/198/187/199 whose raw meaning is a plain event count everywhere, plus NVMe's +# structured health log). Vendor raw counts are reported with their numbers and can raise a +# WARN, but they can never produce a FAIL — because their units are vendor-defined and +# reading one as if it were a portable sector count is exactly the defect this replaces. - name: Assess — SMART per physical disk ansible.builtin.set_fact: node_health_checks: "{{ node_health_checks + [{ @@ -71,22 +79,80 @@ ansible_become: false th: "{{ node_health_thresholds }}" disk_smart: "{{ (node_health_smart | selectattr('path', 'equalto', item.path) | list | first) | default({}) }}" - # Sub-verdicts. Each is SKIP unless the number was actually read. + + # --- Sub-verdicts. Each is SKIP unless the number behind it was actually read; a SKIP is + # an honest "unknown" and never contributes a PASS. ------------------------------------ + + # 1. The drive's own overall-health self-assessment. Vendor-independent by construction. s_health: >- {{ 'SKIP' if (disk_smart.passed is not defined or disk_smart.passed is none) else ('PASS' if disk_smart.passed else 'FAIL') }} - s_realloc: >- - {{ 'SKIP' if (disk_smart.reallocated_sectors is not defined or disk_smart.reallocated_sectors is not number) - else ('FAIL' if disk_smart.reallocated_sectors | int >= th.smart_reallocated_fail - else ('WARN' if disk_smart.reallocated_sectors | int >= th.smart_reallocated_warn else 'PASS')) }} + # 2. NVMe critical warning bitfield (spare below threshold, media in read-only, etc.). + s_nvme_crit: >- + {{ 'SKIP' if (disk_smart.nvme_critical_warning is not defined or disk_smart.nvme_critical_warning is not number) + else ('PASS' if disk_smart.nvme_critical_warning | int == 0 else 'FAIL') }} + # 3. THE portable per-attribute failure signal: normalised value at or below the + # threshold the MANUFACTURER set for that attribute on that model. Pre-fail failing + # now = FAIL. Old_age failing now, or anything that failed in the past, = WARN. + s_thresh: >- + {{ 'SKIP' if not (disk_smart.thresholds_present | default(false)) + else ('FAIL' if (disk_smart.failing_now_prefail | default([]) | length) > 0 + else ('WARN' if ((disk_smart.failing_now | default([]) | length) > 0 + or (disk_smart.failed_past | default([]) | length) > 0) + else 'PASS')) }} + # 4. ATA-standard hard-error counters. 197 pending; 198/187 + NVMe media_errors are + # uncorrectable reads — data the drive could NOT return. Same meaning on every drive. s_pending: >- {{ 'SKIP' if (disk_smart.pending_sectors is not defined or disk_smart.pending_sectors is not number) else ('FAIL' if disk_smart.pending_sectors | int >= th.smart_pending_fail else ('WARN' if disk_smart.pending_sectors | int >= th.smart_pending_warn else 'PASS')) }} + uncorr_values: >- + {{ [disk_smart.offline_uncorrectable | default(none), + disk_smart.reported_uncorrect | default(none), + disk_smart.nvme_media_errors | default(none)] | select('number') | list }} + s_uncorr: >- + {{ 'SKIP' if (uncorr_values | length) == 0 + else ('FAIL' if (uncorr_values | max) >= th.smart_uncorrectable_fail + else ('WARN' if (uncorr_values | max) >= th.smart_uncorrectable_warn else 'PASS')) }} + # 5. 199 CRC — SATA link integrity (cable/backplane), not media. WARN only: the counter + # never resets, so one historic bad seating must not condemn the node forever. + s_crc: >- + {{ 'SKIP' if (disk_smart.crc_errors is not defined or disk_smart.crc_errors is not number) + else ('WARN' if disk_smart.crc_errors | int >= th.smart_crc_warn else 'PASS') }} + # 6. Wear, from the NORMALISED lifetime indicator — a percentage by definition. + life_rem: "{{ disk_smart.lifetime_remaining_pct | default(none) }}" + life_used: "{{ (100 - (life_rem | int)) if (life_rem is number) else none }}" s_life: >- - {{ 'SKIP' if (disk_smart.lifetime_remaining_pct is not defined or disk_smart.lifetime_remaining_pct is not number) - else ('FAIL' if disk_smart.lifetime_remaining_pct | int <= th.smart_lifetime_remaining_fail_pct - else ('WARN' if disk_smart.lifetime_remaining_pct | int <= th.smart_lifetime_remaining_warn_pct else 'PASS')) }} + {{ 'SKIP' if (life_rem is not number) + else ('FAIL' if life_rem | int <= th.smart_lifetime_remaining_fail_pct + else ('WARN' if (life_rem | int <= th.smart_lifetime_remaining_warn_pct + or life_used | int >= th.smart_life_used_pct_warn) + else 'PASS')) }} + # 7. Reallocation ACTIVITY. Portable evidence first (196 event count, and the grown + # bad-block counter smartctl's drivedb named for this model). Only if neither exists + # do we fall back to attribute 5's raw — and then ONLY to say "nonzero, so something + # was remapped", which is portable, never to interpret its magnitude, which is not. + remap_evidence: >- + {{ [disk_smart.realloc_events | default(none), + disk_smart.grown_bad_blocks | default(none)] | select('number') | list }} + realloc_raw: "{{ (disk_smart.realloc_attr | default({}, true)).get('raw', {}).get('value', none) }}" + s_remap: >- + {{ ('WARN' if (remap_evidence | max) >= th.smart_realloc_events_warn else 'PASS') + if (remap_evidence | length) > 0 + else (('WARN' if realloc_raw | int > 0 else 'PASS') if realloc_raw is number else 'SKIP') }} + # 8. Endurance stress — erase/program failures are NAND wearing out under the FTL. + s_erase: >- + {{ 'SKIP' if (disk_smart.erase_fail_count is not defined or disk_smart.erase_fail_count is not number) + else ('WARN' if disk_smart.erase_fail_count | int >= th.smart_erase_fail_warn else 'PASS') }} + s_program: >- + {{ 'SKIP' if (disk_smart.program_fail_count is not defined or disk_smart.program_fail_count is not number) + else ('WARN' if disk_smart.program_fail_count | int >= th.smart_program_fail_warn else 'PASS') }} + # 9. Unexpected power loss. On a consumer SSD with no power-loss protection each one is + # an opportunity for a torn or lost in-flight write — which is precisely the guarantee + # etcd's fsync contract depends on. Directly relevant to PROD-PROVISIONING.md Section 2. + s_ploss: >- + {{ 'SKIP' if (disk_smart.unexpected_power_loss is not defined or disk_smart.unexpected_power_loss is not number) + else ('WARN' if disk_smart.unexpected_power_loss | int >= th.smart_unexpected_power_loss_warn else 'PASS') }} s_temp: >- {{ 'SKIP' if (disk_smart.temperature_c is not defined or disk_smart.temperature_c is not number) else ('FAIL' if disk_smart.temperature_c | int >= th.smart_temp_fail_c @@ -94,8 +160,82 @@ # Worst sub-verdict wins. `select('in', ...)` preserves the order of the left-hand # list, so the LAST survivor is the most severe present. smart_status: >- - {{ (['PASS', 'WARN', 'FAIL'] | select('in', [s_health, s_realloc, s_pending, s_life, s_temp]) | list | last) + {{ (['PASS', 'WARN', 'FAIL'] | select('in', [s_health, s_nvme_crit, s_thresh, s_pending, + s_uncorr, s_crc, s_life, s_remap, s_erase, s_program, s_ploss, s_temp]) | list | last) | default('SKIP', true) }} + + # --- Detail. Every WARN/FAIL must arrive WITH the number that caused it; a verdict the + # operator cannot check is not much better than a wrong one. ------------------------- + smart_findings: >- + {%- set n = namespace(l=[]) -%} + {%- if s_health == 'FAIL' -%} + {%- set _ = n.l.append('the drive reports its OWN overall-health self-assessment as FAILED') -%} + {%- endif -%} + {%- if s_nvme_crit == 'FAIL' -%} + {%- set _ = n.l.append('NVMe critical_warning=' ~ disk_smart.nvme_critical_warning ~ ' (non-zero)') -%} + {%- endif -%} + {%- for a in (disk_smart.failing_now_prefail | default([])) -%} + {%- set _ = n.l.append('Pre-fail attribute ' ~ a.id ~ ' ' ~ a.name ~ ' is AT OR BELOW its manufacturer threshold NOW (value=' + ~ a.value ~ ', thresh=' ~ a.thresh ~ ')') -%} + {%- endfor -%} + {%- for a in (disk_smart.failed_past | default([])) -%} + {%- set _ = n.l.append('attribute ' ~ a.id ~ ' ' ~ a.name ~ ' crossed its manufacturer threshold IN THE PAST (worst=' + ~ a.worst ~ ', thresh=' ~ a.thresh ~ ')') -%} + {%- endfor -%} + {%- if s_pending in ['WARN', 'FAIL'] -%} + {%- set _ = n.l.append('current pending sectors (197)=' ~ disk_smart.pending_sectors) -%} + {%- endif -%} + {%- if s_uncorr in ['WARN', 'FAIL'] -%} + {%- set _ = n.l.append('UNCORRECTABLE read errors — offline(198)=' ~ (disk_smart.offline_uncorrectable if disk_smart.offline_uncorrectable is number else 'n/a') + ~ ', reported(187)=' ~ (disk_smart.reported_uncorrect if disk_smart.reported_uncorrect is number else 'n/a') + ~ ', nvme media_errors=' ~ (disk_smart.nvme_media_errors if disk_smart.nvme_media_errors is number else 'n/a')) -%} + {%- endif -%} + {%- if s_crc == 'WARN' -%} + {%- set _ = n.l.append('SATA link CRC errors (199)=' ~ disk_smart.crc_errors ~ ' — cable/backplane, not media') -%} + {%- endif -%} + {%- if s_life in ['WARN', 'FAIL'] -%} + {%- set _ = n.l.append(life_used ~ '% of rated life CONSUMED (' ~ life_rem + ~ '% remaining, from the normalised wear indicator)') -%} + {%- endif -%} + {%- if s_remap == 'WARN' and (remap_evidence | length) > 0 -%} + {%- set _ = n.l.append((remap_evidence | max) ~ ' grown bad block(s)/reallocation event(s)' + ~ (' [196 events=' ~ disk_smart.realloc_events ~ ']' if disk_smart.realloc_events is number else '') + ~ (' [grown failing blocks=' ~ disk_smart.grown_bad_blocks ~ ']' if disk_smart.grown_bad_blocks is number else '')) -%} + {%- elif s_remap == 'WARN' -%} + {%- set _ = n.l.append('the drive reports a non-zero reallocation attribute (5 raw=' ~ realloc_raw + ~ '); its UNIT is vendor-defined, so the count is not interpreted here') -%} + {%- endif -%} + {%- if s_erase == 'WARN' -%} + {%- set _ = n.l.append('NAND erase failures=' ~ disk_smart.erase_fail_count) -%} + {%- endif -%} + {%- if s_program == 'WARN' -%} + {%- set _ = n.l.append('NAND program failures=' ~ disk_smart.program_fail_count) -%} + {%- endif -%} + {%- if s_ploss == 'WARN' -%} + {%- set _ = n.l.append('unexpected power losses=' ~ disk_smart.unexpected_power_loss + ~ ' — a consumer SSD has no power-loss protection, so each one risks a torn or lost in-flight write') -%} + {%- endif -%} + {%- if s_temp in ['WARN', 'FAIL'] -%} + {%- set _ = n.l.append('temperature=' ~ disk_smart.temperature_c ~ 'C') -%} + {%- endif -%} + {{ n.l | join('; ') }} + # The raw value that started this. Shown ALWAYS when it is non-zero and contradicts the + # normalised reading, explicitly labelled as vendor-encoded and explicitly excluded from + # the verdict — so the next operator to see 61440 sees why it is not 61440 bad sectors. + realloc_raw_note: >- + {%- if realloc_raw is number and realloc_raw | int > 0 -%} + {%- set a = disk_smart.realloc_attr -%} + attribute 5 ({{ a.name }}) raw={{ realloc_raw }} normalised value={{ a.value }} vs thresh={{ a.thresh }} + {%- if (remap_evidence | length) > 0 and (remap_evidence | max) > 0 + and realloc_raw | int != (remap_evidence | max) | int + and realloc_raw | int % (remap_evidence | max) | int == 0 -%} + {{ ' (= ' ~ (remap_evidence | max) ~ ' x ' ~ (realloc_raw | int // (remap_evidence | max) | int) ~ ')' }} + {%- endif -%} + . SMART raw values are VENDOR-DEFINED (smartctl(8): "entirely vendor-specific") and this + one is NOT a portable bad-sector count — it is reported as context and is NOT used for + the verdict. The normalised value/threshold pair beside it is the manufacturer's own + trip point and IS. + {%- endif -%} smart_detail: >- {{ ('not read — ' ~ node_health_root_skip_reason) if not (node_health_root_available | bool) @@ -108,10 +248,19 @@ if not (disk_smart.available | default(false)) else ('overall-health=' ~ ('PASSED' if disk_smart.passed else 'FAILED') ~ ', power_on_hours=' ~ (disk_smart.power_on_hours | default('?')) - ~ ', lifetime_remaining=' ~ (disk_smart.lifetime_remaining_pct | default('?')) ~ '%' - ~ ', reallocated=' ~ (disk_smart.reallocated_sectors | default('?')) - ~ ', pending=' ~ (disk_smart.pending_sectors | default('?')) - ~ ', temp=' ~ (disk_smart.temperature_c | default('?')) ~ 'C') }} + ~ ', power_cycles=' ~ (disk_smart.power_cycles | default('?')) + ~ ', life_remaining=' ~ ((life_rem | string ~ '%') if life_rem is number else 'not reported') + ~ ', pending(197)=' ~ (disk_smart.pending_sectors if disk_smart.pending_sectors is number else 'n/a') + ~ ', offline_uncorrectable(198)=' ~ (disk_smart.offline_uncorrectable if disk_smart.offline_uncorrectable is number else 'n/a') + ~ ', reported_uncorrect(187)=' ~ (disk_smart.reported_uncorrect if disk_smart.reported_uncorrect is number else 'n/a') + ~ ', crc(199)=' ~ (disk_smart.crc_errors if disk_smart.crc_errors is number else 'n/a') + ~ (('. NO attribute is at or below its manufacturer threshold; closest is ' + ~ disk_smart.prefail_closest.id ~ ' ' ~ disk_smart.prefail_closest.name + ~ ' value=' ~ disk_smart.prefail_closest.value + ~ ' vs thresh=' ~ disk_smart.prefail_closest.thresh) + if (s_thresh == 'PASS' and (disk_smart.prefail_closest | default(none)) is mapping) else '') + ~ (('. FINDINGS: ' ~ smart_findings) if smart_findings | trim else '') + ~ (('. CONTEXT: ' ~ realloc_raw_note) if realloc_raw_note | trim else '')) }} - name: Assess — kernel-visible I/O errors per disk (no root needed) ansible.builtin.set_fact: diff --git a/infra/ansible/roles/node_health/tasks/collect_smart.yml b/infra/ansible/roles/node_health/tasks/collect_smart.yml index ccabe84..fc991fa 100644 --- a/infra/ansible/roles/node_health/tasks/collect_smart.yml +++ b/infra/ansible/roles/node_health/tasks/collect_smart.yml @@ -43,12 +43,79 @@ smart_json: "{{ (smart_result.stdout | default('') | trim) | default('{}', true) | from_json }}" smart_attrs: "{{ smart_json.ata_smart_attributes.table | default([]) }}" smart_nvme: "{{ smart_json.nvme_smart_health_information_log | default({}) }}" - # Normalised ("value") wear indicators, in vendor-preference order. Micron/Crucial use - # 202 Percent_Lifetime_Remaining; many others use 231 SSD_Life_Left or 233. The RAW - # value is kept alongside in `attributes` so nothing is hidden by this normalisation. + + # --- VENDOR-INDEPENDENT FAILURE SIGNAL: normalised value vs the drive's own threshold. + # smartctl computes this for us and emits it as `when_failed`, per smartctl(8): "If the + # Attribute's current Normalized value is less than or equal to the threshold value, then + # the WHEN_FAILED column will display FAILING_NOW. If not, but the worst recorded value is + # less than or equal to the threshold value, then this column will display In_the_past. If + # the WHEN_FAILED column has no entry ... then this Attribute is OK now (not failing) and + # has also never failed in the past." In JSON those become "now" / "past" / "". We match + # case-insensitively on 'past' rather than on an exact token so the same code reads either + # the JSON spelling or the plain-text one if a future smartctl changes it. + # + # This is the ONLY comparison in this role that can condemn a drive on an attribute the + # role does not itself understand, and that is the point: the manufacturer picked the + # threshold, for that attribute, on that model. It needs no per-vendor knowledge from us. + smart_failed_attrs: >- + {{ smart_attrs | selectattr('when_failed', 'defined') + | rejectattr('when_failed', 'equalto', '') | list }} + smart_failed_past: "{{ smart_failed_attrs | selectattr('when_failed', 'search', '(?i)past') | list }}" + smart_failing_now: "{{ smart_failed_attrs | rejectattr('when_failed', 'search', '(?i)past') | list }}" + # Of the failing ones, which are Pre-fail (imminent failure) vs Old_age (design life + # exceeded)? smartctl decodes that flag from the drive too, so this stays vendor-neutral. + smart_failing_now_prefail: "{{ smart_failing_now | selectattr('flags.prefailure', 'defined') | selectattr('flags.prefailure') | list }}" + # How close is the drive to ANY of its own trip points? The smallest (value - thresh) + # across all Pre-fail attributes that actually carry a threshold (thresh 0 = "cannot + # trip", per ATA, so it is excluded rather than counted as a margin of `value`). + # Reported as context; a shrinking margin is the early warning the raw values cannot give. + smart_prefail_margin_json: >- + {%- set ns = namespace(best=none) -%} + {%- for a in smart_attrs -%} + {%- if (a.flags.prefailure | default(false)) and ((a.thresh | default(0)) | int) > 0 -%} + {%- set m = ((a.value | default(0)) | int) - ((a.thresh) | int) -%} + {%- if ns.best is none or m < ns.best.margin -%} + {%- set ns.best = {'id': a.id, 'name': a.name, 'value': (a.value | int), + 'worst': ((a.worst | default(a.value)) | int), + 'thresh': (a.thresh | int), 'margin': m} -%} + {%- endif -%} + {%- endif -%} + {%- endfor -%} + {{ ns.best | to_json }} + + # --- Normalised ("value") wear indicators, in vendor-preference order. Micron/Crucial use + # 202 Perc_Rated_Life_Used; many others use 231 SSD_Life_Left or 233. All three express + # the SAME thing in the normalised field — a percentage of rated life REMAINING — which is + # exactly why the normalised field is usable across vendors where their raws are not. smart_life_202: "{{ smart_attrs | selectattr('id', 'equalto', 202) | map(attribute='value') | first | default(none) }}" smart_life_231: "{{ smart_attrs | selectattr('id', 'equalto', 231) | map(attribute='value') | first | default(none) }}" smart_life_233: "{{ smart_attrs | selectattr('id', 'equalto', 233) | map(attribute='value') | first | default(none) }}" + + # --- VENDOR CONTEXT. Everything below is collected and printed but NEVER produces a FAIL. + # Attribute 5 is kept in full — id, drive's own name, normalised value/worst/thresh AND + # raw — precisely so the report can show the raw next to the normalised reading that + # contradicts it, instead of silently dropping the number that started this. + smart_attr5: "{{ (smart_attrs | selectattr('id', 'equalto', 5) | list | first) | default(none) }}" + # 196 Reallocated_Event_Count — an ATA-standard *event* count (one tick per remap + # operation), the portable stand-in for attribute 5's unportable raw. + smart_realloc_events: "{{ smart_attrs | selectattr('id', 'equalto', 196) | map(attribute='raw.value') | first | default(none) }}" + # Endurance counters matched by the NAME smartctl's drive database resolved for this + # model, not by a bare vendor attribute ID. IDs 170-174 are in the vendor-specific range + # and mean different things on different controllers; drivedb is smartmontools' own + # per-(model, firmware) translation of them, so matching its resolved name borrows that + # knowledge instead of hardcoding one vendor's. On a drive drivedb does not recognise the + # name is "Unknown_Attribute", nothing matches, and these stay null — no claim is made. + smart_grown_blocks: >- + {{ smart_attrs | selectattr('name', 'search', '(?i)(grown|runtime)[_ -]?(failing|bad)?[_ -]?block') + | map(attribute='raw.value') | first | default(none) }} + smart_erase_fail: "{{ smart_attrs | selectattr('name', 'search', '(?i)erase[_ -]?fail') | map(attribute='raw.value') | first | default(none) }}" + smart_program_fail: "{{ smart_attrs | selectattr('name', 'search', '(?i)program[_ -]?fail|prog[_ -]?fail') | map(attribute='raw.value') | first | default(none) }}" + # '(?i)power[_ -]?loss[_ -]?(ct|cnt|count)' deliberately does NOT match Intel's + # Power_Loss_Cap_Test (a capacitor self-test result, not a count of power losses). + smart_power_loss: >- + {{ smart_attrs | selectattr('name', 'search', '(?i)(unexpect\w*|unsafe)[_ -]?(power|shutdown)|power[_ -]?loss[_ -]?(ct|cnt|count)') + | map(attribute='raw.value') | first | default(none) }} + smart_disk: path: "{{ smart_result.item.path }}" # smartctl ran, but did it understand the device? @@ -64,11 +131,17 @@ power_on_hours: "{{ smart_json.power_on_time.hours | default(none) }}" power_cycles: "{{ smart_json.power_cycle_count | default(none) }}" temperature_c: "{{ smart_json.temperature.current | default(none) }}" - reallocated_sectors: "{{ smart_attrs | selectattr('id', 'equalto', 5) | map(attribute='raw.value') | first | default(none) }}" + # Normalised-vs-threshold verdict inputs (see the long note above). + failing_now: "{{ smart_failing_now }}" + failing_now_prefail: "{{ smart_failing_now_prefail }}" + failed_past: "{{ smart_failed_past }}" + thresholds_present: "{{ (smart_attrs | selectattr('thresh', 'defined') | rejectattr('thresh', 'equalto', 0) | list | length) > 0 }}" + prefail_closest: "{{ smart_prefail_margin_json | from_json }}" + # ATA-standard error counters — same meaning on every drive that implements them. pending_sectors: "{{ smart_attrs | selectattr('id', 'equalto', 197) | map(attribute='raw.value') | first | default(none) }}" offline_uncorrectable: "{{ smart_attrs | selectattr('id', 'equalto', 198) | map(attribute='raw.value') | first | default(none) }}" - crc_errors: "{{ smart_attrs | selectattr('id', 'equalto', 199) | map(attribute='raw.value') | first | default(none) }}" reported_uncorrect: "{{ smart_attrs | selectattr('id', 'equalto', 187) | map(attribute='raw.value') | first | default(none) }}" + crc_errors: "{{ smart_attrs | selectattr('id', 'equalto', 199) | map(attribute='raw.value') | first | default(none) }}" host_writes_raw: "{{ smart_attrs | selectattr('id', 'equalto', 241) | map(attribute='raw.value') | first | default(none) }}" # NVMe reports wear directly as "percentage_used"; ATA SSDs report a countdown. lifetime_remaining_pct: >- @@ -77,6 +150,18 @@ else (smart_life_231 if smart_life_231 is not none else smart_life_233)) }} nvme_media_errors: "{{ smart_nvme.media_errors | default(none) }}" nvme_critical_warning: "{{ smart_nvme.critical_warning | default(none) }}" + nvme_unsafe_shutdowns: "{{ smart_nvme.unsafe_shutdowns | default(none) }}" + # --- Vendor context. Never a verdict on its own; always printed with its own numbers. + realloc_events: "{{ smart_realloc_events }}" + grown_bad_blocks: "{{ smart_grown_blocks }}" + erase_fail_count: "{{ smart_erase_fail }}" + program_fail_count: "{{ smart_program_fail }}" + unexpected_power_loss: "{{ smart_power_loss if smart_power_loss is not none else (smart_nvme.unsafe_shutdowns | default(none)) }}" + # Attribute 5, whole and labelled, so the raw is visible BESIDE the normalised reading + # rather than standing in for it. `reallocated_sectors` is deliberately gone: that name + # asserted a unit the ATA spec does not guarantee, and asserting it is what produced the + # false FAIL on k3s-prod-1 (see defaults/main.yml). + realloc_attr: "{{ smart_attr5 }}" # Every attribute the operator asked to see, verbatim, names and all — vendors # disagree on naming, so the checks key off IDs and the report shows the drive's # own labels. diff --git a/infra/ansible/roles/node_health/templates/summary.txt.j2 b/infra/ansible/roles/node_health/templates/summary.txt.j2 index 6babeb8..23a32ba 100644 --- a/infra/ansible/roles/node_health/templates/summary.txt.j2 +++ b/infra/ansible/roles/node_health/templates/summary.txt.j2 @@ -61,10 +61,23 @@ {% set sm = (node_health_smart | selectattr('path', 'equalto', d.path) | list | first) | default(none) %} {% if sm and sm.available %} SMART overall-health={{ 'PASSED' if sm.passed else 'FAILED' }} power_on_hours={{ sm.power_on_hours | default('?') }} power_cycles={{ sm.power_cycles | default('?') }} - lifetime_remaining={{ sm.lifetime_remaining_pct | default('?') }}% temp={{ sm.temperature_c | default('?') }}C - reallocated={{ sm.reallocated_sectors | default('?') }} pending={{ sm.pending_sectors | default('?') }} offline_uncorrectable={{ sm.offline_uncorrectable | default('?') }} crc_errors={{ sm.crc_errors | default('?') }} + lifetime_remaining={{ (sm.lifetime_remaining_pct | string ~ '%') if sm.lifetime_remaining_pct is not none else 'not reported' }} temp={{ (sm.temperature_c | string ~ 'C') if sm.temperature_c is not none else 'not reported by this drive' }} + pending(197)={{ sm.pending_sectors | default('?') }} offline_uncorrectable(198)={{ sm.offline_uncorrectable | default('?') }} reported_uncorrect(187)={{ sm.reported_uncorrect | default('?') }} crc(199)={{ sm.crc_errors | default('?') }} +{% if sm.failing_now or sm.failed_past %} + THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ a.thresh }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ a.thresh }}]{% endfor %} + +{% elif sm.thresholds_present %} + no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ sm.prefail_closest.thresh }}{% endif %} + +{% endif %} + vendor context (NOT used for the verdict — raw units are vendor-defined): + reallocation events(196)={{ sm.realloc_events if sm.realloc_events is not none else 'n/a' }} grown bad blocks={{ sm.grown_bad_blocks if sm.grown_bad_blocks is not none else 'n/a' }} erase fails={{ sm.erase_fail_count if sm.erase_fail_count is not none else 'n/a' }} program fails={{ sm.program_fail_count if sm.program_fail_count is not none else 'n/a' }} unexpected power loss={{ sm.unexpected_power_loss if sm.unexpected_power_loss is not none else 'n/a' }} +{% if sm.realloc_attr %} + attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ sm.realloc_attr.thresh }} (the signal that is judged) +{% endif %} {% for a in sm.attributes | default([]) %} - [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }} + [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %} <<< when_failed={{ a.when_failed }}{% endif %} + {% endfor %} {% else %} SMART NOT READ — {{ (node_health_root_skip_reason if not node_health_root_available else ('smartmontools not installed on this node (ships with roles/node_baseline; this role never installs packages)' if not node_health_facts.tools.smartctl else 'device exposes no SMART data')) | regex_replace('\s+', ' ') | trim }}