fix(node_health): SMART verdict was wrong by 4096x — judge on portable signals
The first real SMART run FAILED k3s-prod-1's drive on reallocated=61440. That verdict
was WRONG, and the failure mode is instructive: the check trusted a VENDOR-DEFINED raw
value as if it were an ATA-standard sector count.
Micron TN-FD-20 ("SMART Attributes for the C400 SSD") names attribute 5 "Reallocated
BLOCK Count" and defines its raw as grown-bad-blocks x Block_Sector_Count, where
Block_Sector_Count is 256 x 16 = 4096 for the 128Gb-NAND parts (256GB/512GB). This is
the 512GB C400-MTFDDAC512MAM: 61440 / 4096 = 15. Triply corroborated on the drive itself
— attr 170 Grown_Failing_Block_Ct = 15, attr 196 Reallocated_Event_Count = 15, attr 5
normalised value 100 against thresh 10, and `when_failed` empty on every attribute.
FIFTEEN grown bad blocks, not sixty-one thousand bad sectors.
smartctl(8) states it outright: since ATA/ATAPI-4 rev 4 the meaning of raw Attribute
fields is "entirely vendor-specific". So a raw value can never be a portable verdict.
FAIL now comes only from signals that mean the same thing on every drive: the drive's own
smart_status.passed; NORMALISED value vs the MANUFACTURER's own threshold via smartctl's
when_failed ("now"/"past"/""), split Pre-fail -> FAIL and Old_age-or-past -> WARN; the
ATA-standard counters 197/198/187 (199 CRC is WARN-only — it is link, not media); and for
NVMe critical_warning / media_errors / percentage_used. Wear reads from the NORMALISED
lifetime indicator (202/231/233, or NVMe percentage_used) — a percentage by definition.
Vendor raw counts are still collected and PRINTED WITH THEIR NUMBERS, and may raise a
WARN, but can never drive a FAIL. Attribute 5's raw is now shown beside the normalised
reading that contradicts it, labelled vendor-encoded, with a mechanically-derived
"(= 15 x 4096)". Endurance attributes are matched by THE NAME smartctl's drivedb resolved
for that model, not by bare ID — drivedb is the portability layer; on an unrecognised
drive the name is Unknown_Attribute, nothing matches, and the field stays null with no
claim made. The STORAGE section now prints 170/172/173/174/189/206, whose previous
omission is exactly what let the misread stand unchallenged.
Also documented from TN-FD-20: on this family attr 197 "will always be 0, as reallocation
will be done on-the-fly" — so pending=0 is NOT evidence of health here. The role reports
the number and infers nothing from it.
Branches real hardware cannot exercise were verified with a harness rendering the actual
role expressions against synthetic smartctl payloads: virtio/no-stdout -> SKIP; NVMe
healthy -> PASS; NVMe degraded -> FAIL; a genuinely dying ATA drive (attr 5 normalised 8
<= thresh 10, pending 48) -> FAIL driven by the threshold trip and standard counters, not
the raw; unknown-vendor attrs -> PASS with the unnamed counter ignored; failed-in-the-past
-> WARN.
NEW VERDICTS. k3s-prod-1: 0 FAIL / 4 WARN / 12 PASS / 1 SKIP — SMART is a WARN naming the
real findings (31% of rated life consumed, 15 grown bad blocks, 187 erase failures, 32
unexpected power losses on a drive with no power-loss protection). k3s-dev-1: unchanged,
all three virtio disks still SKIP with their stated reason.
BREAKING (JSON artifact): the fact `reallocated_sectors` is renamed and gone — that name
asserted a unit ATA does not guarantee. Replaced by realloc_attr (the whole attribute,
normalised + raw), realloc_events, and grown_bad_blocks. Nothing in the repo reads it.
Serves: R-ENV-03, R-OBS-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
ce1966acb0
commit
0b3db6e778
5 files changed
+398
-29
No files matched your search
@@ -57,6 +57,14 @@
|
||||
'detail': node_health_root_skip_reason }] }}"
|
||||
|
||||
# --- Storage ------------------------------------------------------------------------
|
||||
# SMART VERDICTS — READ defaults/main.yml's SMART threshold block BEFORE EDITING THIS.
|
||||
# Short version: this check judges a drive on the signals smartctl(8) itself defines as
|
||||
# vendor-independent (the drive's own overall-health verdict; NORMALISED value vs the
|
||||
# MANUFACTURER'S OWN threshold, via smartctl's `when_failed`; and the ATA-standard error
|
||||
# counters 197/198/187/199 whose raw meaning is a plain event count everywhere, plus NVMe's
|
||||
# structured health log). Vendor raw counts are reported with their numbers and can raise a
|
||||
# WARN, but they can never produce a FAIL — because their units are vendor-defined and
|
||||
# reading one as if it were a portable sector count is exactly the defect this replaces.
|
||||
- name: Assess — SMART per physical disk
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
@@ -71,22 +79,80 @@
|
||||
ansible_become: false
|
||||
th: "{{ node_health_thresholds }}"
|
||||
disk_smart: "{{ (node_health_smart | selectattr('path', 'equalto', item.path) | list | first) | default({}) }}"
|
||||
# Sub-verdicts. Each is SKIP unless the number was actually read.
|
||||
|
||||
# --- Sub-verdicts. Each is SKIP unless the number behind it was actually read; a SKIP is
|
||||
# an honest "unknown" and never contributes a PASS. ------------------------------------
|
||||
|
||||
# 1. The drive's own overall-health self-assessment. Vendor-independent by construction.
|
||||
s_health: >-
|
||||
{{ 'SKIP' if (disk_smart.passed is not defined or disk_smart.passed is none)
|
||||
else ('PASS' if disk_smart.passed else 'FAIL') }}
|
||||
s_realloc: >-
|
||||
{{ 'SKIP' if (disk_smart.reallocated_sectors is not defined or disk_smart.reallocated_sectors is not number)
|
||||
else ('FAIL' if disk_smart.reallocated_sectors | int >= th.smart_reallocated_fail
|
||||
else ('WARN' if disk_smart.reallocated_sectors | int >= th.smart_reallocated_warn else 'PASS')) }}
|
||||
# 2. NVMe critical warning bitfield (spare below threshold, media in read-only, etc.).
|
||||
s_nvme_crit: >-
|
||||
{{ 'SKIP' if (disk_smart.nvme_critical_warning is not defined or disk_smart.nvme_critical_warning is not number)
|
||||
else ('PASS' if disk_smart.nvme_critical_warning | int == 0 else 'FAIL') }}
|
||||
# 3. THE portable per-attribute failure signal: normalised value at or below the
|
||||
# threshold the MANUFACTURER set for that attribute on that model. Pre-fail failing
|
||||
# now = FAIL. Old_age failing now, or anything that failed in the past, = WARN.
|
||||
s_thresh: >-
|
||||
{{ 'SKIP' if not (disk_smart.thresholds_present | default(false))
|
||||
else ('FAIL' if (disk_smart.failing_now_prefail | default([]) | length) > 0
|
||||
else ('WARN' if ((disk_smart.failing_now | default([]) | length) > 0
|
||||
or (disk_smart.failed_past | default([]) | length) > 0)
|
||||
else 'PASS')) }}
|
||||
# 4. ATA-standard hard-error counters. 197 pending; 198/187 + NVMe media_errors are
|
||||
# uncorrectable reads — data the drive could NOT return. Same meaning on every drive.
|
||||
s_pending: >-
|
||||
{{ 'SKIP' if (disk_smart.pending_sectors is not defined or disk_smart.pending_sectors is not number)
|
||||
else ('FAIL' if disk_smart.pending_sectors | int >= th.smart_pending_fail
|
||||
else ('WARN' if disk_smart.pending_sectors | int >= th.smart_pending_warn else 'PASS')) }}
|
||||
uncorr_values: >-
|
||||
{{ [disk_smart.offline_uncorrectable | default(none),
|
||||
disk_smart.reported_uncorrect | default(none),
|
||||
disk_smart.nvme_media_errors | default(none)] | select('number') | list }}
|
||||
s_uncorr: >-
|
||||
{{ 'SKIP' if (uncorr_values | length) == 0
|
||||
else ('FAIL' if (uncorr_values | max) >= th.smart_uncorrectable_fail
|
||||
else ('WARN' if (uncorr_values | max) >= th.smart_uncorrectable_warn else 'PASS')) }}
|
||||
# 5. 199 CRC — SATA link integrity (cable/backplane), not media. WARN only: the counter
|
||||
# never resets, so one historic bad seating must not condemn the node forever.
|
||||
s_crc: >-
|
||||
{{ 'SKIP' if (disk_smart.crc_errors is not defined or disk_smart.crc_errors is not number)
|
||||
else ('WARN' if disk_smart.crc_errors | int >= th.smart_crc_warn else 'PASS') }}
|
||||
# 6. Wear, from the NORMALISED lifetime indicator — a percentage by definition.
|
||||
life_rem: "{{ disk_smart.lifetime_remaining_pct | default(none) }}"
|
||||
life_used: "{{ (100 - (life_rem | int)) if (life_rem is number) else none }}"
|
||||
s_life: >-
|
||||
{{ 'SKIP' if (disk_smart.lifetime_remaining_pct is not defined or disk_smart.lifetime_remaining_pct is not number)
|
||||
else ('FAIL' if disk_smart.lifetime_remaining_pct | int <= th.smart_lifetime_remaining_fail_pct
|
||||
else ('WARN' if disk_smart.lifetime_remaining_pct | int <= th.smart_lifetime_remaining_warn_pct else 'PASS')) }}
|
||||
{{ 'SKIP' if (life_rem is not number)
|
||||
else ('FAIL' if life_rem | int <= th.smart_lifetime_remaining_fail_pct
|
||||
else ('WARN' if (life_rem | int <= th.smart_lifetime_remaining_warn_pct
|
||||
or life_used | int >= th.smart_life_used_pct_warn)
|
||||
else 'PASS')) }}
|
||||
# 7. Reallocation ACTIVITY. Portable evidence first (196 event count, and the grown
|
||||
# bad-block counter smartctl's drivedb named for this model). Only if neither exists
|
||||
# do we fall back to attribute 5's raw — and then ONLY to say "nonzero, so something
|
||||
# was remapped", which is portable, never to interpret its magnitude, which is not.
|
||||
remap_evidence: >-
|
||||
{{ [disk_smart.realloc_events | default(none),
|
||||
disk_smart.grown_bad_blocks | default(none)] | select('number') | list }}
|
||||
realloc_raw: "{{ (disk_smart.realloc_attr | default({}, true)).get('raw', {}).get('value', none) }}"
|
||||
s_remap: >-
|
||||
{{ ('WARN' if (remap_evidence | max) >= th.smart_realloc_events_warn else 'PASS')
|
||||
if (remap_evidence | length) > 0
|
||||
else (('WARN' if realloc_raw | int > 0 else 'PASS') if realloc_raw is number else 'SKIP') }}
|
||||
# 8. Endurance stress — erase/program failures are NAND wearing out under the FTL.
|
||||
s_erase: >-
|
||||
{{ 'SKIP' if (disk_smart.erase_fail_count is not defined or disk_smart.erase_fail_count is not number)
|
||||
else ('WARN' if disk_smart.erase_fail_count | int >= th.smart_erase_fail_warn else 'PASS') }}
|
||||
s_program: >-
|
||||
{{ 'SKIP' if (disk_smart.program_fail_count is not defined or disk_smart.program_fail_count is not number)
|
||||
else ('WARN' if disk_smart.program_fail_count | int >= th.smart_program_fail_warn else 'PASS') }}
|
||||
# 9. Unexpected power loss. On a consumer SSD with no power-loss protection each one is
|
||||
# an opportunity for a torn or lost in-flight write — which is precisely the guarantee
|
||||
# etcd's fsync contract depends on. Directly relevant to PROD-PROVISIONING.md Section 2.
|
||||
s_ploss: >-
|
||||
{{ 'SKIP' if (disk_smart.unexpected_power_loss is not defined or disk_smart.unexpected_power_loss is not number)
|
||||
else ('WARN' if disk_smart.unexpected_power_loss | int >= th.smart_unexpected_power_loss_warn else 'PASS') }}
|
||||
s_temp: >-
|
||||
{{ 'SKIP' if (disk_smart.temperature_c is not defined or disk_smart.temperature_c is not number)
|
||||
else ('FAIL' if disk_smart.temperature_c | int >= th.smart_temp_fail_c
|
||||
@@ -94,8 +160,82 @@
|
||||
# Worst sub-verdict wins. `select('in', ...)` preserves the order of the left-hand
|
||||
# list, so the LAST survivor is the most severe present.
|
||||
smart_status: >-
|
||||
{{ (['PASS', 'WARN', 'FAIL'] | select('in', [s_health, s_realloc, s_pending, s_life, s_temp]) | list | last)
|
||||
{{ (['PASS', 'WARN', 'FAIL'] | select('in', [s_health, s_nvme_crit, s_thresh, s_pending,
|
||||
s_uncorr, s_crc, s_life, s_remap, s_erase, s_program, s_ploss, s_temp]) | list | last)
|
||||
| default('SKIP', true) }}
|
||||
|
||||
# --- Detail. Every WARN/FAIL must arrive WITH the number that caused it; a verdict the
|
||||
# operator cannot check is not much better than a wrong one. -------------------------
|
||||
smart_findings: >-
|
||||
{%- set n = namespace(l=[]) -%}
|
||||
{%- if s_health == 'FAIL' -%}
|
||||
{%- set _ = n.l.append('the drive reports its OWN overall-health self-assessment as FAILED') -%}
|
||||
{%- endif -%}
|
||||
{%- if s_nvme_crit == 'FAIL' -%}
|
||||
{%- set _ = n.l.append('NVMe critical_warning=' ~ disk_smart.nvme_critical_warning ~ ' (non-zero)') -%}
|
||||
{%- endif -%}
|
||||
{%- for a in (disk_smart.failing_now_prefail | default([])) -%}
|
||||
{%- set _ = n.l.append('Pre-fail attribute ' ~ a.id ~ ' ' ~ a.name ~ ' is AT OR BELOW its manufacturer threshold NOW (value='
|
||||
~ a.value ~ ', thresh=' ~ a.thresh ~ ')') -%}
|
||||
{%- endfor -%}
|
||||
{%- for a in (disk_smart.failed_past | default([])) -%}
|
||||
{%- set _ = n.l.append('attribute ' ~ a.id ~ ' ' ~ a.name ~ ' crossed its manufacturer threshold IN THE PAST (worst='
|
||||
~ a.worst ~ ', thresh=' ~ a.thresh ~ ')') -%}
|
||||
{%- endfor -%}
|
||||
{%- if s_pending in ['WARN', 'FAIL'] -%}
|
||||
{%- set _ = n.l.append('current pending sectors (197)=' ~ disk_smart.pending_sectors) -%}
|
||||
{%- endif -%}
|
||||
{%- if s_uncorr in ['WARN', 'FAIL'] -%}
|
||||
{%- set _ = n.l.append('UNCORRECTABLE read errors — offline(198)=' ~ (disk_smart.offline_uncorrectable if disk_smart.offline_uncorrectable is number else 'n/a')
|
||||
~ ', reported(187)=' ~ (disk_smart.reported_uncorrect if disk_smart.reported_uncorrect is number else 'n/a')
|
||||
~ ', nvme media_errors=' ~ (disk_smart.nvme_media_errors if disk_smart.nvme_media_errors is number else 'n/a')) -%}
|
||||
{%- endif -%}
|
||||
{%- if s_crc == 'WARN' -%}
|
||||
{%- set _ = n.l.append('SATA link CRC errors (199)=' ~ disk_smart.crc_errors ~ ' — cable/backplane, not media') -%}
|
||||
{%- endif -%}
|
||||
{%- if s_life in ['WARN', 'FAIL'] -%}
|
||||
{%- set _ = n.l.append(life_used ~ '% of rated life CONSUMED (' ~ life_rem
|
||||
~ '% remaining, from the normalised wear indicator)') -%}
|
||||
{%- endif -%}
|
||||
{%- if s_remap == 'WARN' and (remap_evidence | length) > 0 -%}
|
||||
{%- set _ = n.l.append((remap_evidence | max) ~ ' grown bad block(s)/reallocation event(s)'
|
||||
~ (' [196 events=' ~ disk_smart.realloc_events ~ ']' if disk_smart.realloc_events is number else '')
|
||||
~ (' [grown failing blocks=' ~ disk_smart.grown_bad_blocks ~ ']' if disk_smart.grown_bad_blocks is number else '')) -%}
|
||||
{%- elif s_remap == 'WARN' -%}
|
||||
{%- set _ = n.l.append('the drive reports a non-zero reallocation attribute (5 raw=' ~ realloc_raw
|
||||
~ '); its UNIT is vendor-defined, so the count is not interpreted here') -%}
|
||||
{%- endif -%}
|
||||
{%- if s_erase == 'WARN' -%}
|
||||
{%- set _ = n.l.append('NAND erase failures=' ~ disk_smart.erase_fail_count) -%}
|
||||
{%- endif -%}
|
||||
{%- if s_program == 'WARN' -%}
|
||||
{%- set _ = n.l.append('NAND program failures=' ~ disk_smart.program_fail_count) -%}
|
||||
{%- endif -%}
|
||||
{%- if s_ploss == 'WARN' -%}
|
||||
{%- set _ = n.l.append('unexpected power losses=' ~ disk_smart.unexpected_power_loss
|
||||
~ ' — a consumer SSD has no power-loss protection, so each one risks a torn or lost in-flight write') -%}
|
||||
{%- endif -%}
|
||||
{%- if s_temp in ['WARN', 'FAIL'] -%}
|
||||
{%- set _ = n.l.append('temperature=' ~ disk_smart.temperature_c ~ 'C') -%}
|
||||
{%- endif -%}
|
||||
{{ n.l | join('; ') }}
|
||||
# The raw value that started this. Shown ALWAYS when it is non-zero and contradicts the
|
||||
# normalised reading, explicitly labelled as vendor-encoded and explicitly excluded from
|
||||
# the verdict — so the next operator to see 61440 sees why it is not 61440 bad sectors.
|
||||
realloc_raw_note: >-
|
||||
{%- if realloc_raw is number and realloc_raw | int > 0 -%}
|
||||
{%- set a = disk_smart.realloc_attr -%}
|
||||
attribute 5 ({{ a.name }}) raw={{ realloc_raw }} normalised value={{ a.value }} vs thresh={{ a.thresh }}
|
||||
{%- if (remap_evidence | length) > 0 and (remap_evidence | max) > 0
|
||||
and realloc_raw | int != (remap_evidence | max) | int
|
||||
and realloc_raw | int % (remap_evidence | max) | int == 0 -%}
|
||||
{{ ' (= ' ~ (remap_evidence | max) ~ ' x ' ~ (realloc_raw | int // (remap_evidence | max) | int) ~ ')' }}
|
||||
{%- endif -%}
|
||||
. SMART raw values are VENDOR-DEFINED (smartctl(8): "entirely vendor-specific") and this
|
||||
one is NOT a portable bad-sector count — it is reported as context and is NOT used for
|
||||
the verdict. The normalised value/threshold pair beside it is the manufacturer's own
|
||||
trip point and IS.
|
||||
{%- endif -%}
|
||||
smart_detail: >-
|
||||
{{ ('not read — ' ~ node_health_root_skip_reason)
|
||||
if not (node_health_root_available | bool)
|
||||
@@ -108,10 +248,19 @@
|
||||
if not (disk_smart.available | default(false))
|
||||
else ('overall-health=' ~ ('PASSED' if disk_smart.passed else 'FAILED')
|
||||
~ ', power_on_hours=' ~ (disk_smart.power_on_hours | default('?'))
|
||||
~ ', lifetime_remaining=' ~ (disk_smart.lifetime_remaining_pct | default('?')) ~ '%'
|
||||
~ ', reallocated=' ~ (disk_smart.reallocated_sectors | default('?'))
|
||||
~ ', pending=' ~ (disk_smart.pending_sectors | default('?'))
|
||||
~ ', temp=' ~ (disk_smart.temperature_c | default('?')) ~ 'C') }}
|
||||
~ ', power_cycles=' ~ (disk_smart.power_cycles | default('?'))
|
||||
~ ', life_remaining=' ~ ((life_rem | string ~ '%') if life_rem is number else 'not reported')
|
||||
~ ', pending(197)=' ~ (disk_smart.pending_sectors if disk_smart.pending_sectors is number else 'n/a')
|
||||
~ ', offline_uncorrectable(198)=' ~ (disk_smart.offline_uncorrectable if disk_smart.offline_uncorrectable is number else 'n/a')
|
||||
~ ', reported_uncorrect(187)=' ~ (disk_smart.reported_uncorrect if disk_smart.reported_uncorrect is number else 'n/a')
|
||||
~ ', crc(199)=' ~ (disk_smart.crc_errors if disk_smart.crc_errors is number else 'n/a')
|
||||
~ (('. NO attribute is at or below its manufacturer threshold; closest is '
|
||||
~ disk_smart.prefail_closest.id ~ ' ' ~ disk_smart.prefail_closest.name
|
||||
~ ' value=' ~ disk_smart.prefail_closest.value
|
||||
~ ' vs thresh=' ~ disk_smart.prefail_closest.thresh)
|
||||
if (s_thresh == 'PASS' and (disk_smart.prefail_closest | default(none)) is mapping) else '')
|
||||
~ (('. FINDINGS: ' ~ smart_findings) if smart_findings | trim else '')
|
||||
~ (('. CONTEXT: ' ~ realloc_raw_note) if realloc_raw_note | trim else '')) }}
|
||||
|
||||
- name: Assess — kernel-visible I/O errors per disk (no root needed)
|
||||
ansible.builtin.set_fact:
|
||||
|
||||
@@ -43,12 +43,79 @@
|
||||
smart_json: "{{ (smart_result.stdout | default('') | trim) | default('{}', true) | from_json }}"
|
||||
smart_attrs: "{{ smart_json.ata_smart_attributes.table | default([]) }}"
|
||||
smart_nvme: "{{ smart_json.nvme_smart_health_information_log | default({}) }}"
|
||||
# Normalised ("value") wear indicators, in vendor-preference order. Micron/Crucial use
|
||||
# 202 Percent_Lifetime_Remaining; many others use 231 SSD_Life_Left or 233. The RAW
|
||||
# value is kept alongside in `attributes` so nothing is hidden by this normalisation.
|
||||
|
||||
# --- VENDOR-INDEPENDENT FAILURE SIGNAL: normalised value vs the drive's own threshold.
|
||||
# smartctl computes this for us and emits it as `when_failed`, per smartctl(8): "If the
|
||||
# Attribute's current Normalized value is less than or equal to the threshold value, then
|
||||
# the WHEN_FAILED column will display FAILING_NOW. If not, but the worst recorded value is
|
||||
# less than or equal to the threshold value, then this column will display In_the_past. If
|
||||
# the WHEN_FAILED column has no entry ... then this Attribute is OK now (not failing) and
|
||||
# has also never failed in the past." In JSON those become "now" / "past" / "". We match
|
||||
# case-insensitively on 'past' rather than on an exact token so the same code reads either
|
||||
# the JSON spelling or the plain-text one if a future smartctl changes it.
|
||||
#
|
||||
# This is the ONLY comparison in this role that can condemn a drive on an attribute the
|
||||
# role does not itself understand, and that is the point: the manufacturer picked the
|
||||
# threshold, for that attribute, on that model. It needs no per-vendor knowledge from us.
|
||||
smart_failed_attrs: >-
|
||||
{{ smart_attrs | selectattr('when_failed', 'defined')
|
||||
| rejectattr('when_failed', 'equalto', '') | list }}
|
||||
smart_failed_past: "{{ smart_failed_attrs | selectattr('when_failed', 'search', '(?i)past') | list }}"
|
||||
smart_failing_now: "{{ smart_failed_attrs | rejectattr('when_failed', 'search', '(?i)past') | list }}"
|
||||
# Of the failing ones, which are Pre-fail (imminent failure) vs Old_age (design life
|
||||
# exceeded)? smartctl decodes that flag from the drive too, so this stays vendor-neutral.
|
||||
smart_failing_now_prefail: "{{ smart_failing_now | selectattr('flags.prefailure', 'defined') | selectattr('flags.prefailure') | list }}"
|
||||
# How close is the drive to ANY of its own trip points? The smallest (value - thresh)
|
||||
# across all Pre-fail attributes that actually carry a threshold (thresh 0 = "cannot
|
||||
# trip", per ATA, so it is excluded rather than counted as a margin of `value`).
|
||||
# Reported as context; a shrinking margin is the early warning the raw values cannot give.
|
||||
smart_prefail_margin_json: >-
|
||||
{%- set ns = namespace(best=none) -%}
|
||||
{%- for a in smart_attrs -%}
|
||||
{%- if (a.flags.prefailure | default(false)) and ((a.thresh | default(0)) | int) > 0 -%}
|
||||
{%- set m = ((a.value | default(0)) | int) - ((a.thresh) | int) -%}
|
||||
{%- if ns.best is none or m < ns.best.margin -%}
|
||||
{%- set ns.best = {'id': a.id, 'name': a.name, 'value': (a.value | int),
|
||||
'worst': ((a.worst | default(a.value)) | int),
|
||||
'thresh': (a.thresh | int), 'margin': m} -%}
|
||||
{%- endif -%}
|
||||
{%- endif -%}
|
||||
{%- endfor -%}
|
||||
{{ ns.best | to_json }}
|
||||
|
||||
# --- Normalised ("value") wear indicators, in vendor-preference order. Micron/Crucial use
|
||||
# 202 Perc_Rated_Life_Used; many others use 231 SSD_Life_Left or 233. All three express
|
||||
# the SAME thing in the normalised field — a percentage of rated life REMAINING — which is
|
||||
# exactly why the normalised field is usable across vendors where their raws are not.
|
||||
smart_life_202: "{{ smart_attrs | selectattr('id', 'equalto', 202) | map(attribute='value') | first | default(none) }}"
|
||||
smart_life_231: "{{ smart_attrs | selectattr('id', 'equalto', 231) | map(attribute='value') | first | default(none) }}"
|
||||
smart_life_233: "{{ smart_attrs | selectattr('id', 'equalto', 233) | map(attribute='value') | first | default(none) }}"
|
||||
|
||||
# --- VENDOR CONTEXT. Everything below is collected and printed but NEVER produces a FAIL.
|
||||
# Attribute 5 is kept in full — id, drive's own name, normalised value/worst/thresh AND
|
||||
# raw — precisely so the report can show the raw next to the normalised reading that
|
||||
# contradicts it, instead of silently dropping the number that started this.
|
||||
smart_attr5: "{{ (smart_attrs | selectattr('id', 'equalto', 5) | list | first) | default(none) }}"
|
||||
# 196 Reallocated_Event_Count — an ATA-standard *event* count (one tick per remap
|
||||
# operation), the portable stand-in for attribute 5's unportable raw.
|
||||
smart_realloc_events: "{{ smart_attrs | selectattr('id', 'equalto', 196) | map(attribute='raw.value') | first | default(none) }}"
|
||||
# Endurance counters matched by the NAME smartctl's drive database resolved for this
|
||||
# model, not by a bare vendor attribute ID. IDs 170-174 are in the vendor-specific range
|
||||
# and mean different things on different controllers; drivedb is smartmontools' own
|
||||
# per-(model, firmware) translation of them, so matching its resolved name borrows that
|
||||
# knowledge instead of hardcoding one vendor's. On a drive drivedb does not recognise the
|
||||
# name is "Unknown_Attribute", nothing matches, and these stay null — no claim is made.
|
||||
smart_grown_blocks: >-
|
||||
{{ smart_attrs | selectattr('name', 'search', '(?i)(grown|runtime)[_ -]?(failing|bad)?[_ -]?block')
|
||||
| map(attribute='raw.value') | first | default(none) }}
|
||||
smart_erase_fail: "{{ smart_attrs | selectattr('name', 'search', '(?i)erase[_ -]?fail') | map(attribute='raw.value') | first | default(none) }}"
|
||||
smart_program_fail: "{{ smart_attrs | selectattr('name', 'search', '(?i)program[_ -]?fail|prog[_ -]?fail') | map(attribute='raw.value') | first | default(none) }}"
|
||||
# '(?i)power[_ -]?loss[_ -]?(ct|cnt|count)' deliberately does NOT match Intel's
|
||||
# Power_Loss_Cap_Test (a capacitor self-test result, not a count of power losses).
|
||||
smart_power_loss: >-
|
||||
{{ smart_attrs | selectattr('name', 'search', '(?i)(unexpect\w*|unsafe)[_ -]?(power|shutdown)|power[_ -]?loss[_ -]?(ct|cnt|count)')
|
||||
| map(attribute='raw.value') | first | default(none) }}
|
||||
|
||||
smart_disk:
|
||||
path: "{{ smart_result.item.path }}"
|
||||
# smartctl ran, but did it understand the device?
|
||||
@@ -64,11 +131,17 @@
|
||||
power_on_hours: "{{ smart_json.power_on_time.hours | default(none) }}"
|
||||
power_cycles: "{{ smart_json.power_cycle_count | default(none) }}"
|
||||
temperature_c: "{{ smart_json.temperature.current | default(none) }}"
|
||||
reallocated_sectors: "{{ smart_attrs | selectattr('id', 'equalto', 5) | map(attribute='raw.value') | first | default(none) }}"
|
||||
# Normalised-vs-threshold verdict inputs (see the long note above).
|
||||
failing_now: "{{ smart_failing_now }}"
|
||||
failing_now_prefail: "{{ smart_failing_now_prefail }}"
|
||||
failed_past: "{{ smart_failed_past }}"
|
||||
thresholds_present: "{{ (smart_attrs | selectattr('thresh', 'defined') | rejectattr('thresh', 'equalto', 0) | list | length) > 0 }}"
|
||||
prefail_closest: "{{ smart_prefail_margin_json | from_json }}"
|
||||
# ATA-standard error counters — same meaning on every drive that implements them.
|
||||
pending_sectors: "{{ smart_attrs | selectattr('id', 'equalto', 197) | map(attribute='raw.value') | first | default(none) }}"
|
||||
offline_uncorrectable: "{{ smart_attrs | selectattr('id', 'equalto', 198) | map(attribute='raw.value') | first | default(none) }}"
|
||||
crc_errors: "{{ smart_attrs | selectattr('id', 'equalto', 199) | map(attribute='raw.value') | first | default(none) }}"
|
||||
reported_uncorrect: "{{ smart_attrs | selectattr('id', 'equalto', 187) | map(attribute='raw.value') | first | default(none) }}"
|
||||
crc_errors: "{{ smart_attrs | selectattr('id', 'equalto', 199) | map(attribute='raw.value') | first | default(none) }}"
|
||||
host_writes_raw: "{{ smart_attrs | selectattr('id', 'equalto', 241) | map(attribute='raw.value') | first | default(none) }}"
|
||||
# NVMe reports wear directly as "percentage_used"; ATA SSDs report a countdown.
|
||||
lifetime_remaining_pct: >-
|
||||
@@ -77,6 +150,18 @@
|
||||
else (smart_life_231 if smart_life_231 is not none else smart_life_233)) }}
|
||||
nvme_media_errors: "{{ smart_nvme.media_errors | default(none) }}"
|
||||
nvme_critical_warning: "{{ smart_nvme.critical_warning | default(none) }}"
|
||||
nvme_unsafe_shutdowns: "{{ smart_nvme.unsafe_shutdowns | default(none) }}"
|
||||
# --- Vendor context. Never a verdict on its own; always printed with its own numbers.
|
||||
realloc_events: "{{ smart_realloc_events }}"
|
||||
grown_bad_blocks: "{{ smart_grown_blocks }}"
|
||||
erase_fail_count: "{{ smart_erase_fail }}"
|
||||
program_fail_count: "{{ smart_program_fail }}"
|
||||
unexpected_power_loss: "{{ smart_power_loss if smart_power_loss is not none else (smart_nvme.unsafe_shutdowns | default(none)) }}"
|
||||
# Attribute 5, whole and labelled, so the raw is visible BESIDE the normalised reading
|
||||
# rather than standing in for it. `reallocated_sectors` is deliberately gone: that name
|
||||
# asserted a unit the ATA spec does not guarantee, and asserting it is what produced the
|
||||
# false FAIL on k3s-prod-1 (see defaults/main.yml).
|
||||
realloc_attr: "{{ smart_attr5 }}"
|
||||
# Every attribute the operator asked to see, verbatim, names and all — vendors
|
||||
# disagree on naming, so the checks key off IDs and the report shows the drive's
|
||||
# own labels.
|
||||
|
||||
Reference in new issue
Block a user