From b19c5db9ae43cf007eb45a1d2c0fd04befa108fc Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 3 Sep 2026 22:32:25 -0500 Subject: [PATCH] =?UTF-8?q?feat(prod):=20node=203=20onboarded=20=E2=80=94?= =?UTF-8?q?=20and=20it=20is=20NOT=20immune;=20BIOS-setting=20hypothesis=20?= =?UTF-8?q?disproven?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit k3s-prod-3 online at the planned 192.168.2.12 (serial 8CC015307N, same chassis/BIOS/CPU), enabled in inventory, vaulted become confirmed, package floor installed. Health: 0 FAIL / 4 WARN / 14 PASS / 1 SKIP — the only node with zero FAILs. WD Blue WDS500G2B0A SATA SSD at 100% life, 3213 h, 0 grown bad blocks, 0 erase fails. Matched Crucial 2x16 GB pair — the only node with symmetric memory. *** NODE 3 IS NOT IMMUNE TO THE §0b BUG. *** Its firmware DOES publish a MAT (MEMATTR=0xbeece018); the ABSENCE of a memattr diagnostic line means the kernel read a VALID table, which is the opposite of immunity. There are THREE outcomes, not two: bail-out on a garbage version >2, a valid parse with a sane size, and the catastrophic path. Seven boots across three machines show the MAT address MOVES BOOT-TO-BOOT on every node with settings constant, and each node has been seen on more than one path. Blast radius widens from two nodes to ALL THREE; all three need the >= 6.13 kernel. THE CORRUPTING BYTES ARE THE OS BOOT CHAIN, not firmware and not DRAM: garbage version 1463964749 = 0x5742544D appeared on node 2 AND node 3 — two different machines, identical bytes. The overwriter is the shared shim/GRUB/EFI-stub/initrd placement writing into the page the firmware pointed at. So initrd size, module set and boot device are part of the causal chain. BIOS-SETTING HYPOTHESIS: DISPROVEN, and the three-way diff cannot support it even in principle. All 240 attributes read on all three; firmware identical fleet-wide. The discriminating set (both working nodes agree, broken differs) is seven attributes, of which: - Fast Boot is the ONLY live candidate, and only as a probability-shifter — HP documents it 'skips some preboot initialization steps', which changes the heap top and hence where the MAT lands. Different mechanism from the previously-killed MRC-caching theory; that stays dead. - Legacy Boot Options is AFFIRMATIVELY INERT: 'Configure Legacy Support and Secure Boot' = 'Legacy Support Disable and Secure Boot Enable' on ALL THREE, Option ROM Launch Policy = 'All UEFI' on all three. No CSM is built on node 2 either. A naive diff gets this wrong. - the rest are coincidence. After Power Loss is ELIMINATED by node 3, which works and shares node 2's Power Off — though it is now a §1 checklist violation on TWO nodes. FATAL CONFOUND: the 1=3!=2 partition is perfectly collinear with node 2 being the only single-DIMM AND the only NVMe-booting node. Adding node 3 did not break the confound, it DOUBLED it. No setting can be the cause anyway: node 2 booted good once and bad twice with settings held constant. EXPERIMENT DESIGN RECORDED because the obvious approach is statistically worthless: p(bad) is known only to n=3, 95% CI ~0.09-0.99, so even 8 clean boots prove nothing. Elimination needs ~20 boots/arm; an odds shift needs ~50. Zero-touch first step: 20 reboots of node 2 unchanged, logging MEMATTR address vs outcome — both failures were at exactly 0xbeeb0018 and all five successes elsewhere, so if outcome maps onto a narrow address range the mechanism is settled without touching anything. ROLE BUG FIXED: node 3's WD drive reports a SMART threshold for only 1 of 25 attributes, and smartctl OMITS the key rather than emitting 0 — summary.txt.j2 dereferenced a.thresh bare in five places and the report task died, so no artifact was written. Added a thr() macro rendering 'n/a' (deliberately not 0, which would read as a real ATA cannot-trip threshold). Would have hit any drive of this class. Also: three nodes, three different disk models, only one NVMe, and every node has a single disk with no unpartitioned spare — so §2's identical-layout requirement is unmet on all three. Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) --- infra/PROD-PROVISIONING.md | 165 +++++++++++++++--- infra/ansible/inventory/hosts.ini | 28 ++- .../node_health/templates/summary.txt.j2 | 19 +- 3 files changed, 179 insertions(+), 33 deletions(-) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 684d605..7bad2fe 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -25,15 +25,15 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31. |---|---|---|---| | Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` | | LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` | -| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | | -| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | | -| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | | -| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | | -| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | | -| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | | -| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | | -| `/dev/kvm` | present | present | | -| Serial | MXL1052NGN | MXL0073949 | | +| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | +| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | +| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | 31.1 GiB usable of 32 GB (2×16 GB Crucial `CT16G4SFD8266.C16FE`, **a matched pair**) | +| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) | +| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC | +| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12.107+deb13-amd64 | +| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | +| `/dev/kvm` | present | present | present | +| Serial | MXL1052NGN | MXL0073949 | 8CC015307N | **Node 1's DIMMs are also not a matched pair** (surfaced 2026-09-01 by the new health check; unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2**E1 (DDR4-**3200**), @@ -41,10 +41,21 @@ unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2 but they differ in rank organisation and rated speed, so the pair runs at the slower 2667. Recorded only because §2 requires the three nodes to end up symmetric, and "2×16 GB Micron" hid the mismatch. -**The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a -healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica -placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's -NVMe is the drive node 1 should have had. +**The nodes are NOT hardware-symmetric — and with node 3 in (2026-09-03) the asymmetry is +THREE-WAY, not a node-1 outlier.** Node 1 has a ~2011 SATA SSD with real wear; node 2 a healthy +modern NVMe; node 3 a third distinct model, a WD Blue 3D SATA SSD in excellent health (100% life, +0 grown bad blocks). **Three nodes, three different drive models, and only one is NVMe.** §2 requires +an identical disk layout across all three because Longhorn replica placement assumes symmetry — so +this is a live input to the disk decision, not a footnote. Node 2's NVMe is the drive node 1 should +have had; node 3 does not have it either. Node 3 is also the only node with **two matched DIMMs** +(2× Crucial 2Rx8 DDR4-2666) and the only one with its **full** RAM complement of 32 GB actually +usable. **That is luck, not immunity:** node 3's firmware publishes an EFI Memory Attributes Table +too (`MEMATTR=0xbeece018`), the kernel simply parsed it cleanly on this boot, so node 3 is exposed to +the §0b bug on exactly the same terms as node 1 and needs the same >= 6.13 kernel. Node 3 therefore +WIDENS §0b's blast radius from "both prod nodes" to **all three**; it is not a control machine. +Every node also carries **exactly one** physical disk with **no unpartitioned spare**, so +`node_baseline_longhorn_device` / `node_baseline_etcd_device` have nothing to claim on any of the +three without repartitioning or adding drives — the §2 decision cannot be satisfied by node 3 either. Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd). @@ -92,15 +103,42 @@ memblock_reserve(efi_mem_attr_table, tbl_size); That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` = DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero. -### BOTH prod nodes are affected — node 1 has only been lucky +### ALL THREE prod nodes are affected — none is immune + +Seven boots observed across the three machines. **The MAT address moves boot-to-boot on every +node with settings held constant**, and each node has been seen on more than one path: + +| Node | Boot | `MEMATTR=` | Kernel's read | Memory | +|---|---|---|---|---| +| 1 | −1 | `0xbeee6018` | *(no message — table read as VALID)* | 32.4 G ✅ | +| 1 | 0 | `0xbeec9298` | version **1030** (> 2 → bail) | 32.4 G ✅ | +| 2 | −2 | `0xbeadc018` | version **1463964749** (> 2 → bail) | 15.9 G ✅ | +| 2 | −1 | `0xbeeb0018` | **Failed to map** | 2.6 G ❌ | +| 2 | 0 | `0xbeeb0018` | **Failed to map** | 2.6 G ❌ | +| 3 | −1 | `0xbee8a018` | version **1463964749** (> 2 → bail) | 32.4 G ✅ | +| 3 | 0 | `0xbeece018` | *(no message — table read as VALID)* | 32.4 G ✅ | + +So there are **three** outcomes, not two: bail-out on a garbage version > 2, a *valid* parse with a +sane size, and the catastrophic path. `k3s-prod-3` publishes a MAT at `0xbeece018` exactly like the +others — the *absence* of a memattr diagnostic line on its current boot means the kernel read a +**valid** table, which is the opposite of immunity. Dev nodes publish no MAT at all (Proxmox +SeaBIOS/OVMF) and are genuinely unaffected. + +**All three prod nodes are on the same lottery and all three need the ≥ 6.13 kernel.** + +### The corrupting bytes are the OS boot chain, not the firmware or the DRAM + +The garbage version `1463964749` = `0x5742544D` appeared on **node 2 AND node 3** — two different +machines, identical corrupting bytes. That rules out both firmware-specific behaviour and a DRAM +fault: the overwriter is the **shared OS boot chain** (shim / GRUB / EFI stub / initrd placement), +writing the same data into the page the firmware pointed at. This is the documented failure mode — +the firmware places the MAT in memory the OS classifies as plain `System RAM`, and whoever allocates +there next overwrites it. + +It also means the boot chain's own allocation footprint — initrd size, module set, boot device — is +part of the causal chain, which matters for the experiment design below. -| Node | memattr version field | Outcome | -|---|---|---| -| `k3s-prod-1` | `1030` — garbage, but **> 2** so the kernel bails | full 32 GiB **by luck, not immunity** | -| `k3s-prod-2` | reads **≤ 2** so the kernel trusts it | 2.6 GiB | -| dev nodes | **no memattr table published** (Proxmox OVMF) | unaffected, full 24 GiB | -Node 1 is one unlucky boot away from the identical fault. **Fix both.** ### Mechanism, traced to the byte (second, independent confirmation) @@ -212,6 +250,93 @@ action beyond `apt upgrade`. heuristic that was rejecting *legitimate* tables (false kexec warnings in Meta's fleet); it is not required for this symptom and depends on the first commit anyway. +### Is it a BIOS setting? — three-way diff, researched 2026-09-03. **No.** + +The operator's hypothesis, tested properly now that there are two working nodes and one broken one. +All **240** attributes read on all three; identical key sets; identical firmware everywhere (same +BIOS version string, ME 12.1.1.1022, GOP 9.0.1085, Super I/O 8.9.22). So every difference is settings +or hardware, not firmware revision. + +**The discriminating set — both working nodes agree, the broken node differs.** 26 attributes differ +in total; after excluding per-machine noise (serials, UUIDs, MACs, asset tags, Born On Date, Feature +Byte, SKU, memory size, storage device names) and settings falling on a different partition, seven +remain: + +| Attribute | nodes 1 & 3 (work) | node 2 (broken) | Verdict | +|---|---|---|---| +| **Fast Boot** | Disable | **Enable** | **plausible probability-shifter — the only live candidate** | +| Legacy Boot Options | Disable | Enable | **coincidence — the setting is INERT, see below** | +| Microsoft / Windows / MS Option ROM UEFI CA 2023 | Enable | Disable | coincidence (sub-page `db` size at most) | +| TPM Activation Policy | F1 to Boot | Allow user to reject | coincidence | +| Recover OS after Boot Failure | Disable | Enable | coincidence (but a policy problem in its own right) | + +**`After Power Loss` is eliminated by the third node** — node 3 *works* and shares node 2's +`Power Off`; node 1 is the odd one out. It was never mechanistically credible (PCH/ME power policy, +applied before firmware executes). It remains a §1 checklist violation on **two** nodes now. + +**`Legacy Boot Options` is affirmatively inert**, which a naive diff gets wrong. Three attributes are +**identical on all three nodes** and settle it: `Configure Legacy Support and Secure Boot` = +*"Legacy Support Disable and Secure Boot Enable"*, `Configure Option ROM Launch Policy` = *"All +UEFI"*, `UEFI Boot Options` = Enable. `Legacy Boot Options` is only the boot-order checkbox; the +master switch that actually instantiates a CSM has legacy support disabled on node 2 too. No CSM is +built, no legacy option ROM is shadowed. The second-strongest candidate has **no boot-time +allocation footprint at all**. Secure Boot is likewise not a differential — all three log +`Secure boot enabled` and load the same HP DB cert. + +**`Fast Boot` is the one credible mechanism**, and for a *different reason* than the hypothesis this +document previously recorded and killed. HP documents it verbatim: *"reduces boot up time by +bypassing boot to USB, CD-ROM, and PXE. **This skips some preboot initialization steps.**"* Skipped +preboot init ⇒ fewer boot-services allocations before the MAT is installed ⇒ a different heap top ⇒ +**the MAT lands at a different address**. That is exactly the perturbation that flipped the +near-identical Dell case in [LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382), +where the reporter's "fix" was **unplugging a USB mouse**. (The old Fast-Boot-caches-MRC-training +theory stays dead — byte-identical `BIOS-e820` refuted it.) + +#### Verdict: the hypothesis is not supported, and this diff cannot support it even in principle + +1. **No setting can be the cause.** Node 2 booted good once and bad twice with settings constant, and + the MAT address moves every boot on all three machines. +2. **`Fast Boot` may shift the odds.** That is the honest ceiling. There is currently zero evidence + that it does. +3. **A fatal confound:** the `1=3 ≠ 2` partition is *perfectly collinear* with two hardware + differences that have at least as strong a mechanism — node 2 is the only **single-DIMM** node and + the only **NVMe-booting** node. Nodes 1 and 3 are both dual-DIMM and both SATA. **Adding node 3 + did not break the confound; it doubled it.** A different boot device means a different initrd + module set and size, hence a different EFI-stub allocation footprint — the very thing that decides + whether the MAT page gets overwritten. + +**No model-specific documentation exists.** HP publishes nothing mentioning the EFI Memory Attributes +Table, MAT placement, or an NX/DEP option on this generation. The only memory-adjacent options +(`Option ROM Launch Policy`, `DMA Protection`, `Pre-boot DMA protection`) are identical fleet-wide. No +HP advisory matches. The R21 changelog never mentions memory, and these nodes run **02.26.00 dated +05/05/2026 — newer than anything HP has indexed publicly**, so there is no BIOS update to chase. + +#### The experiment, if it is ever worth running + +Two statistics kill the obvious approach. Node 2's failure rate is known only to **n = 3** (2 bad of +3); the 95% Clopper–Pearson interval on p(bad) is ≈ **0.09–0.99**. Against a null that loose, even +8 clean boots after a change prove nothing. **A baseline arm is mandatory**, and: detecting +*elimination* needs ~**20 boots per arm**; detecting a mere odds shift (0.67 → 0.33) needs ~**50 per +arm** for 80% power. **Do not design for an odds shift — design for elimination and report +"inconclusive" otherwise.** + +**Zero-touch first step, and the one actually worth doing:** 20 scripted reboots of node 2 with +*nothing changed*, logging `MEMATTR=` and the `Memory: …K/…K available` line per boot. That +establishes the real p(bad) we lack, and tests the causal chain directly — **is the outcome predicted +by where the MAT landed?** Present data hint yes: both failures at exactly `0xbeeb0018`, all five +successes elsewhere. If the bad outcome maps onto a narrow address range, the mechanism is settled +without touching anything. + +**If more is wanted, swap hardware rather than BIOS settings** — it crosses settings against memory +topology in one move, needs no firmware write (writes are refused; a BIOS change means a physical F10 +visit per arm), and is work §2 wants anyway: move node 2's single Samsung DIMM into node 3 and node +3's matched pair into node 2, then 20 boots each. Fault follows the DIMM ⇒ topology, settings +exonerated. Fault stays on node 2 ⇒ node 2 implicated, and only then is Fast Boot worth testing. +Fault disappears ⇒ single-DIMM topology. ~1 hour unattended. + +**None of this changes the action.** The fix is already known, is not in dispute, and applies to all +three nodes: kernel ≥ 6.13. + ### What NOT to do - **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is diff --git a/infra/ansible/inventory/hosts.ini b/infra/ansible/inventory/hosts.ini index f13087c..562ec54 100644 --- a/infra/ansible/inventory/hosts.ini +++ b/infra/ansible/inventory/hosts.ini @@ -48,13 +48,14 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh # ============================================================================= # PROD (k3s-prod-*) — Phase 7 PREP ONLY. See infra/PROD-PROVISIONING.md (T-7.1, -# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). Node 1 -# (`k3s-prod-1`, HP EliteDesk 800 G5 Mini) is on the bench and SSH-reachable; nodes 2 -# and 3 have not arrived. This section is inventory/group_vars scaffolding only — -# nothing has been run against node 1, and the operator has decided the CLUSTER stays -# uninitialized (no k3s install, no --cluster-init) until all three nodes are present -# (decision #2 in the DECISIONS-LOG entry above) — no single-server-etcd window, no -# later member-add dance. +# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). ALL THREE +# nodes are now on the bench and SSH-reachable (`k3s-prod-1` 2026-08-31, `k3s-prod-2` +# 2026-09-01, `k3s-prod-3` 2026-09-03) — every HP EliteDesk 800 G5 Mini. This section +# is still inventory/group_vars scaffolding only: nothing beyond the read-only health +# check and the node-baseline PACKAGE floor has been run, and the operator has decided +# the CLUSTER stays uninitialized (no k3s install, no --cluster-init) until the §2 disk +# decision is closed (decision #2 in the DECISIONS-LOG entry above) — no +# single-server-etcd window, no later member-add dance. # # KEY DIFFERENCE FROM DEV: prod sits DIRECTLY on the 192.168.2.x LAN — no pfSense NAT # hop in front of it. Dev's `ansible_host` above is a NAT target (192.168.2.102:225x) @@ -92,9 +93,18 @@ k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10 # node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same # partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the # nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision. -# node 3 remains commented until it is racked. +# k3s-prod-3 came online 2026-09-03 at the planned 192.168.2.12 — same HP EliteDesk 800 G5 +# Mini, same BIOS R21 02.26.00, same i7-9700T, 2x16 GB (a MATCHED Crucial pair, unlike node +# 1's mismatched Microns), full 31.1 GiB usable. It is NOT structurally immune to the §0b +# memattr bug: its firmware DOES publish a MAT (MEMATTR=0xbeece018) and the kernel merely +# happened to parse it cleanly this boot, so node 3 is on the same lottery as node 1 and +# needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA +# SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now +# three-way, not a node-1 outlier. +# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately +# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2. k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11 -# k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12 +k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children] # pattern; named to match the group_vars directory so Ansible auto-loads it). diff --git a/infra/ansible/roles/node_health/templates/summary.txt.j2 b/infra/ansible/roles/node_health/templates/summary.txt.j2 index 732af45..41ebb69 100644 --- a/infra/ansible/roles/node_health/templates/summary.txt.j2 +++ b/infra/ansible/roles/node_health/templates/summary.txt.j2 @@ -12,6 +12,17 @@ {% macro gib(n) -%} {{ (n | int / 1073741824) | round(1) }} GiB {%- endmacro %} +{# smartctl OMITS `thresh` entirely for any attribute whose threshold the drive does not + report (surfaced 2026-09-03 by k3s-prod-3's WDC WDS500G2B0A, which reports one for 1 of + its 25 attributes) — an ABSENT key, not thresh=0, so a bare `a.thresh` raised + "object of type 'dict' has no attribute 'thresh'" and killed the whole report. Render it + as `n/a` (the same token this template already uses for unreported readings) rather than + defaulting to 0, which would falsely read as a real ATA "cannot trip" threshold. The + collector already guards this — tasks/collect_smart.yml uses selectattr('thresh', + 'defined') and `a.thresh | default(0)` — so only the template was exposed. #} +{% macro thr(a) -%} +{{ a.thresh | default('n/a') }} +{%- endmacro %} ================================================================================ NODE HEALTH — {{ inventory_hostname }} ({{ ansible_host | default('n/a') }}) collected {{ node_health_facts.collected_at }} as uid {{ node_health_facts.collector_uid }} @@ -73,19 +84,19 @@ lifetime_remaining={{ (sm.lifetime_remaining_pct | string ~ '%') if sm.lifetime_remaining_pct is not none else 'not reported' }} temp={{ (sm.temperature_c | string ~ 'C') if sm.temperature_c is not none else 'not reported by this drive' }} pending(197)={{ sm.pending_sectors | default('?') }} offline_uncorrectable(198)={{ sm.offline_uncorrectable | default('?') }} reported_uncorrect(187)={{ sm.reported_uncorrect | default('?') }} crc(199)={{ sm.crc_errors | default('?') }} {% if sm.failing_now or sm.failed_past %} - THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ a.thresh }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ a.thresh }}]{% endfor %} + THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ thr(a) }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ thr(a) }}]{% endfor %} {% elif sm.thresholds_present %} - no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ sm.prefail_closest.thresh }}{% endif %} + no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ thr(sm.prefail_closest) }}{% endif %} {% endif %} vendor context (NOT used for the verdict — raw units are vendor-defined): reallocation events(196)={{ sm.realloc_events if sm.realloc_events is not none else 'n/a' }} grown bad blocks={{ sm.grown_bad_blocks if sm.grown_bad_blocks is not none else 'n/a' }} erase fails={{ sm.erase_fail_count if sm.erase_fail_count is not none else 'n/a' }} program fails={{ sm.program_fail_count if sm.program_fail_count is not none else 'n/a' }} unexpected power loss={{ sm.unexpected_power_loss if sm.unexpected_power_loss is not none else 'n/a' }} {% if sm.realloc_attr %} - attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ sm.realloc_attr.thresh }} (the signal that is judged) + attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ thr(sm.realloc_attr) }} (the signal that is judged) {% endif %} {% for a in sm.attributes | default([]) %} - [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %} <<< when_failed={{ a.when_failed }}{% endif %} + [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ thr(a) }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %} <<< when_failed={{ a.when_failed }}{% endif %} {% endfor %} {% else %}