jakman85andClaude Opus 5
0b3db6e778
The first real SMART run FAILED k3s-prod-1's drive on reallocated=61440. That verdict
was WRONG, and the failure mode is instructive: the check trusted a VENDOR-DEFINED raw
value as if it were an ATA-standard sector count.
Micron TN-FD-20 ("SMART Attributes for the C400 SSD") names attribute 5 "Reallocated
BLOCK Count" and defines its raw as grown-bad-blocks x Block_Sector_Count, where
Block_Sector_Count is 256 x 16 = 4096 for the 128Gb-NAND parts (256GB/512GB). This is
the 512GB C400-MTFDDAC512MAM: 61440 / 4096 = 15. Triply corroborated on the drive itself
— attr 170 Grown_Failing_Block_Ct = 15, attr 196 Reallocated_Event_Count = 15, attr 5
normalised value 100 against thresh 10, and `when_failed` empty on every attribute.
FIFTEEN grown bad blocks, not sixty-one thousand bad sectors.
smartctl(8) states it outright: since ATA/ATAPI-4 rev 4 the meaning of raw Attribute
fields is "entirely vendor-specific". So a raw value can never be a portable verdict.
FAIL now comes only from signals that mean the same thing on every drive: the drive's own
smart_status.passed; NORMALISED value vs the MANUFACTURER's own threshold via smartctl's
when_failed ("now"/"past"/""), split Pre-fail -> FAIL and Old_age-or-past -> WARN; the
ATA-standard counters 197/198/187 (199 CRC is WARN-only — it is link, not media); and for
NVMe critical_warning / media_errors / percentage_used. Wear reads from the NORMALISED
lifetime indicator (202/231/233, or NVMe percentage_used) — a percentage by definition.
Vendor raw counts are still collected and PRINTED WITH THEIR NUMBERS, and may raise a
WARN, but can never drive a FAIL. Attribute 5's raw is now shown beside the normalised
reading that contradicts it, labelled vendor-encoded, with a mechanically-derived
"(= 15 x 4096)". Endurance attributes are matched by THE NAME smartctl's drivedb resolved
for that model, not by bare ID — drivedb is the portability layer; on an unrecognised
drive the name is Unknown_Attribute, nothing matches, and the field stays null with no
claim made. The STORAGE section now prints 170/172/173/174/189/206, whose previous
omission is exactly what let the misread stand unchallenged.
Also documented from TN-FD-20: on this family attr 197 "will always be 0, as reallocation
will be done on-the-fly" — so pending=0 is NOT evidence of health here. The role reports
the number and infers nothing from it.
Branches real hardware cannot exercise were verified with a harness rendering the actual
role expressions against synthetic smartctl payloads: virtio/no-stdout -> SKIP; NVMe
healthy -> PASS; NVMe degraded -> FAIL; a genuinely dying ATA drive (attr 5 normalised 8
<= thresh 10, pending 48) -> FAIL driven by the threshold trip and standard counters, not
the raw; unknown-vendor attrs -> PASS with the unnamed counter ignored; failed-in-the-past
-> WARN.
NEW VERDICTS. k3s-prod-1: 0 FAIL / 4 WARN / 12 PASS / 1 SKIP — SMART is a WARN naming the
real findings (31% of rated life consumed, 15 grown bad blocks, 187 erase failures, 32
unexpected power losses on a drive with no power-loss protection). k3s-dev-1: unchanged,
all three virtio disks still SKIP with their stated reason.
BREAKING (JSON artifact): the fact `reallocated_sectors` is renamed and gone — that name
asserted a unit ATA does not guarantee. Replaced by realloc_attr (the whole attribute,
normalised + raw), realloc_events, and grown_bad_blocks. Nothing in the repo reads it.
Serves: R-ENV-03, R-OBS-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
139 lines
8.6 KiB
YAML
139 lines
8.6 KiB
YAML
---
|
|
# node_health — READ-ONLY hardware/health inspection of a node (bare metal or VM).
|
|
#
|
|
# Sibling of roles/cluster_verify: cluster_verify answers "is the CLUSTER healthy",
|
|
# node_health answers "is the HARDWARE under it healthy". Same posture — it inspects and
|
|
# reports, it never changes the node. Deliberately NOT part of node_baseline: baseline
|
|
# installs and configures, health only looks. That separation is why this role must never
|
|
# `apt install` anything (smartmontools is declared in node_baseline_packages; if it is
|
|
# missing here the SMART checks SKIP with a reason, they do not self-heal).
|
|
#
|
|
# Serves R-ENV-01 (reproducible node provisioning — you cannot certify a node you have not
|
|
# measured), R-ENV-03 (bare-metal parity) and R-OBS-04 (node-level observability).
|
|
|
|
# --- Reporting ---------------------------------------------------------------------
|
|
# Artifacts land under infra/ansible/_artifacts/node-health/, mirroring the repo's
|
|
# conformance/_artifacts/ convention: generated output, regenerated on every run, listed
|
|
# in .gitignore, never committed (it also carries drive serial numbers).
|
|
node_health_artifact_dir: "{{ playbook_dir }}/../_artifacts/node-health"
|
|
node_health_write_artifacts: true
|
|
# Print the human summary to the play output as well as writing it to the artifact dir.
|
|
node_health_show_summary: true
|
|
# Report-only by default: a health check that aborts the play hides the other 20 findings.
|
|
# Set true (e.g. in CI or a pre-build gate) to make any FAIL check end the play non-zero.
|
|
node_health_fail_on_fail: false
|
|
# Same, one step stricter: also fail on WARN.
|
|
node_health_fail_on_warn: false
|
|
|
|
# --- Root-only checks ---------------------------------------------------------------
|
|
# Everything this role can read WITHOUT root, it reads without root. Only SMART needs
|
|
# escalation (the ATA/NVMe passthrough ioctls are root-only). The role PROBES become
|
|
# first and, if escalation is unavailable (e.g. prod's vaulted become password is not
|
|
# present — infra/PROD-PROVISIONING.md §5), those checks report SKIPPED with the exact
|
|
# reason. Never a hard failure, and never a fake PASS: a skip is honest.
|
|
node_health_try_privileged: true
|
|
# --nocheck=standby: never spin up a sleeping disk just to read it. Read-only hygiene.
|
|
node_health_smartctl_args: "--json=c --all --nocheck=standby"
|
|
|
|
# --- Thresholds ---------------------------------------------------------------------
|
|
# All verdicts come from here, so a site can retune without touching task logic.
|
|
node_health_thresholds:
|
|
# Filesystems
|
|
fs_used_pct_warn: 80
|
|
fs_used_pct_fail: 90
|
|
inode_used_pct_warn: 80
|
|
# Memory (percentage of MemTotal still available)
|
|
mem_available_pct_warn: 15
|
|
mem_available_pct_fail: 5
|
|
# Temperatures (deg C, any sensor)
|
|
temp_warn_c: 70
|
|
temp_fail_c: 85
|
|
# Pressure-stall: "some" avg60, percent of wall time stalled. Dev learned to care about
|
|
# CPU PSI specifically (it shows saturation long before load average makes it obvious).
|
|
psi_some_avg60_warn: 10.0
|
|
psi_some_avg60_fail: 40.0
|
|
# Network
|
|
nic_min_speed_mbps: 1000
|
|
# A single upstream resolver is a known SPOF on this project — it broke containerd
|
|
# image pulls and CoreDNS forwarding on dev (DECISIONS-LOG 2026-08-31, finding 5).
|
|
# Counted against the node's REAL upstreams: where systemd-resolved is in use the
|
|
# collector looks behind the 127.0.0.53 stub rather than counting it as one server.
|
|
# The fix side of this is the `dns`-tagged block in roles/node_baseline.
|
|
min_nameservers: 2
|
|
# --- SMART (only evaluated when smartctl actually ran) -----------------------------
|
|
#
|
|
# WHY THESE THRESHOLDS AND NOT "reallocated sectors" ANY MORE.
|
|
# The first real run of this role FAILED k3s-prod-1's Crucial/Micron C400 on
|
|
# `reallocated=61440`, read as a plain bad-sector count. That verdict was WRONG, and it
|
|
# was wrong for a structural reason, not a tuning one: SMART *raw* values are vendor
|
|
# defined. smartctl(8) says so outright -- "starting with ATA/ATAPI-4, revision 4, the
|
|
# meaning of these Attribute fields has been made entirely vendor-specific". Micron's own
|
|
# TN-FD-20 (SMART Attributes for the C400 SSD) defines ID 5 not as a sector count but as
|
|
# "Reallocated Block Count", raw = "the grown NAND Flash bad-block count multiplied by the
|
|
# Block_Sector_Count, which is 256 x 8 for 64Gb NAND ... or 256 x 16 for 128Gb NAND (256GB
|
|
# and 512GB)". 256 x 16 = 4096, and 61440 / 4096 = 15 -- corroborated independently by
|
|
# ID 170 Grown_Failing_Block_Ct = 15 and ID 196 Reallocated_Event_Count = 15. Fifteen
|
|
# grown bad blocks, not sixty-one thousand bad sectors.
|
|
#
|
|
# So no threshold below is ever applied to attribute 5's raw value. The verdict comes from
|
|
# the signals smartctl itself defines as vendor-INDEPENDENT:
|
|
# * the drive's own overall-health self-assessment (smart_status.passed);
|
|
# * NORMALISED value vs THRESHOLD, i.e. smartctl's when_failed ("now"/"past"), which is
|
|
# the manufacturer's OWN trip point for that attribute on that drive;
|
|
# * the ATA-standard error counters whose raw meaning is a plain event count everywhere:
|
|
# 197 pending, 198 offline-uncorrectable, 187 reported-uncorrectable, 199 CRC;
|
|
# * NVMe's structured health log (critical_warning, media_errors, percentage_used).
|
|
# Vendor raw counts are still collected and printed -- they are real information -- but
|
|
# they can only ever raise a WARN with the number shown, never a FAIL on their own.
|
|
#
|
|
# Hard-error counters (197/198/187 raw, NVMe media_errors). A plain event count on every
|
|
# drive that implements them. NOTE the converse is not evidence: some drives (the C400
|
|
# among them, per TN-FD-20) hard-code 197 to 0 because reallocation happens on the fly, so
|
|
# "pending = 0" proves nothing. Nonzero is a finding; zero is not a clean bill of health.
|
|
smart_pending_warn: 1
|
|
smart_pending_fail: 10
|
|
smart_uncorrectable_warn: 1
|
|
smart_uncorrectable_fail: 10
|
|
# 199 UDMA_CRC_Error_Count -- SATA link/cable integrity, not media. WARN only: it is a
|
|
# lifetime count that never resets, so a historic cable reseat should not FAIL a node.
|
|
smart_crc_warn: 1
|
|
# Wear, from the NORMALISED lifetime indicator (ATA 202/231/233 value, or NVMe
|
|
# percentage_used) -- a percentage by definition, so portable in a way raw wear counts
|
|
# are not. ..._remaining_* are the "this drive is nearly used up" bands; ..._used_pct_warn
|
|
# is the softer "this is not a fresh drive, know that before you put etcd on it" notice
|
|
# that PROD-PROVISIONING.md Section 2 actually needs.
|
|
smart_lifetime_remaining_warn_pct: 20
|
|
smart_lifetime_remaining_fail_pct: 10
|
|
smart_life_used_pct_warn: 25
|
|
# Reallocation ACTIVITY. 196 Reallocated_Event_Count is an ATA-standard *event* count
|
|
# (one per remap operation) and is the portable stand-in for attribute 5's unportable raw.
|
|
# WARN-only by design: remapping is what a drive is supposed to do, and how many events
|
|
# are "too many" is drive-specific -- the manufacturer already encodes that answer in
|
|
# attribute 5's normalised threshold, which the when_failed check above honours.
|
|
smart_realloc_events_warn: 1
|
|
# Endurance-stress counters, matched by the NAME smartctl's drive database resolved for
|
|
# this model rather than by a bare vendor ID -- drivedb IS the portability layer. On a
|
|
# drive smartctl does not recognise these simply do not match and nothing is claimed.
|
|
smart_erase_fail_warn: 1
|
|
smart_program_fail_warn: 1
|
|
# Unexpected power loss on a consumer SSD with no power-loss protection: each one is a
|
|
# chance of a torn/lost in-flight write. Directly relevant to etcd's fsync contract.
|
|
smart_unexpected_power_loss_warn: 10
|
|
smart_temp_warn_c: 55
|
|
smart_temp_fail_c: 70
|
|
# Power-on hours past which a consumer SSD is simply old. Informational — age alone is
|
|
# not a failure, but it belongs in the operator's field of view when deciding whether a
|
|
# drive may hold production state (PROD-PROVISIONING.md §2).
|
|
smart_power_on_hours_notice: 40000
|
|
|
|
# ATA SMART attribute IDs surfaced verbatim in the report. Vendors disagree on names, so
|
|
# the report prints whatever the drive calls them and the checks key off the IDs.
|
|
# 170-174 and 189 are the Micron/Crucial (and, by convention, most SSD controllers')
|
|
# endurance and grown-bad-block counters; they are the CONTEXT that made the C400 verdict
|
|
# above provable, and leaving them out of the report is what let a single misread raw value
|
|
# stand unchallenged. 1 and 12 are included so the error-rate and power-cycle baselines are
|
|
# visible next to them.
|
|
node_health_smart_attribute_ids: [1, 5, 9, 12, 170, 171, 172, 173, 174, 177, 179, 181,
|
|
182, 183, 184, 187, 188, 189, 190, 194, 195, 196, 197,
|
|
198, 199, 202, 206, 231, 232, 233, 241, 242, 246]
|