Files
home-network-k3s-impl/infra/ansible/roles/node_health/templates/summary.txt.j2
T
jakman85andClaude Opus 5 9f59c36000 feat(ansible): node_health role — read-only hardware health for prod + dev nodes
Operator ask: implement the roles needed to check prod node hardware health, and run
them to find out our state. This is that role; the run's findings are below.

roles/node_health is a read-only sibling to cluster_verify (cluster health vs hardware
health), styled after node_baseline. One unprivileged collector script does the work in
a single SSH round trip — reading /sys/class/dmi, thermal + hwmon, EDAC, cpufreq +
thermal_throttle, /proc/pressure, /sys/block incl. SCSI ioerr/iotmo counters, lsblk,
lscpu, mounts + statvfs, per-NIC stats, resolv.conf.

THE DESIGN POINT: an unreadable field is emitted as null WITH A RECORDED REASON, never
as zero. The summary ends with a "NOT READ (and why) — these are UNKNOWNS, not passes"
section. A fake zero on a reallocated-sector count is exactly the kind of false green
this project refuses. 22 checks report PASS/WARN/FAIL/SKIP/INFO; every SKIP says why.
Report-only by default; thresholds live in defaults/main.yml.

Portable across bare metal AND VMs — validated against k3s-prod-1 (bare metal, no root)
and k3s-dev-1 (VM, root). VM-absent things (SMART on virtio, coretemp, EDAC) skip
cleanly, and iSCSI/Longhorn volumes are split out of the physical-disk set so a live
replica is never mistaken for a spare device.

playbooks/node-health.yml takes its target group as a variable (-e node_health_hosts=
k3s_prod), deliberately unlike the other playbooks that hardcode k3s_servers — see the
group-name gap already flagged in hosts.ini.

Two become facts found the hard way and documented in the role: a group_vars
`ansible_become: true` OUTRANKS `become: false` as a play/task keyword, so unprivileged
tasks must override the variable; and `failed_when: false` does NOT catch "Missing sudo
password" — only `ignore_errors` does.

FIRST RUN, k3s-prod-1, 2026-08-31: 0 FAIL · 3 WARN · 12 PASS · 3 SKIP · 4 INFO.
Box is healthy, cool and unstressed: i7-9700T 8C/8T, VT-x + /dev/kvm (KubeVirt OK), zero
throttle events, hottest sensor 33C, PSI 0.00 across the board, 31.1 GiB RAM, eno1 up at
1000 Mb/s, BIOS R21 02.26.00, no reboot pending.

DRIVE HEALTH IS STILL UNKNOWN — the headline. SMART did not run, blocked TWICE over: no
vaulted ansible_become_password for prod, AND smartmontools is not on the node at all
(node_baseline has never run there). Power-on hours, wear, reallocated/pending sectors
and drive temperature all remain unmeasured on a ~2011 consumer SSD.

WARNs: kernel ioerr_cnt=6 on sda (read cautiously — four freshly attached iSCSI disks on
dev each showed a flat 7, suggesting a fixed per-device discovery cost, not media
trouble; attribution needs dmesg/root); 0.95 GiB swap still active (node_baseline does
the swapoff); one IPv4 resolver only (the other two upstreams are IPv6 Google), which
refines rather than resolves the DNS-SPOF finding.

ESCALATION — ECC status is UNKNOWABLE on this hardware: /sys/devices/system/edac/mc
exists but no controller is bound. Non-ECC consumer RAM. Silent memory corruption on a
prod node would be undetectable. Recorded as a SKIP with reason, not a pass.

Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-31 23:20:25 -05:00

148 lines
10 KiB
Django/Jinja
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{# ===========================================================================
node_health human summary. Rendered ONCE (tasks/report.yml) and used for both
the on-screen report and the .txt artifact, so the two can never drift.
Detail strings are normalised with regex_replace('\s+',' ') because the check
details are authored as YAML folded scalars in assess.yml and can carry stray
newlines/indentation from that folding.
=========================================================================== #}
{% macro fmt(text, width=74, pad=10) -%}
{{ text | regex_replace('\s+', ' ') | trim | wordwrap(width) | indent(pad, first=True) }}
{%- endmacro %}
{% macro gib(n) -%}
{{ (n | int / 1073741824) | round(1) }} GiB
{%- endmacro %}
================================================================================
NODE HEALTH — {{ inventory_hostname }} ({{ ansible_host | default('n/a') }})
collected {{ node_health_facts.collected_at }} as uid {{ node_health_facts.collector_uid }}
================================================================================
VERDICT {{ node_health_counts.FAIL }} FAIL · {{ node_health_counts.WARN }} WARN · {{ node_health_counts.PASS }} PASS · {{ node_health_counts.SKIP }} SKIP · {{ node_health_counts.INFO }} INFO
{{ 'privileged checks RAN' if node_health_root_available | bool else 'privileged checks SKIPPED — root not available' }}
--------------------------------------------------------------------------------
CHECKS
--------------------------------------------------------------------------------
{% for c in node_health_checks %}
[{{ '%-4s' | format(c.status) }}] {{ c.id }} — {{ c.title }}
{{ fmt(c.detail) }}
{% endfor %}
--------------------------------------------------------------------------------
MACHINE — {{ node_health_facts.dmi.sys_vendor | default('?') }} {{ node_health_facts.dmi.product_name | default('?') }}
--------------------------------------------------------------------------------
OS {{ node_health_facts.os.pretty_name | default('?') }}
Kernel {{ node_health_facts.os.kernel | default('?') }}
Uptime {{ (node_health_facts.os.uptime_seconds | float / 86400) | round(2) }} days
BIOS {{ node_health_facts.dmi.bios_vendor | default('?') }} {{ node_health_facts.dmi.bios_version | default('?') }} ({{ node_health_facts.dmi.bios_date | default('?') }}){% if node_health_facts.dmi.ec_firmware_release %}, EC {{ node_health_facts.dmi.ec_firmware_release }}{% endif %}
Board {{ node_health_facts.dmi.board_vendor | default('?') }} {{ node_health_facts.dmi.board_name | default('?') }}
CPU {{ node_health_facts.cpu.model | default('?') }}
{{ node_health_facts.cpu.cpus | default('?') }} logical / {{ node_health_facts.cpu.sockets | default('?') }} socket(s) x {{ node_health_facts.cpu.cores_per_socket | default('?') }} core(s) x {{ node_health_facts.cpu.threads_per_core | default('?') }} thread(s)
virtualisation: {{ node_health_facts.cpu.virtualization | default('none reported') }}, /dev/kvm={{ node_health_facts.cpu.kvm_device }}
{% if node_health_facts.cpu.scaling %}
scaling: {{ node_health_facts.cpu.scaling.driver }}/{{ node_health_facts.cpu.scaling.governor }}, {{ (node_health_facts.cpu.scaling.min_khz | int / 1000) | round | int }}–{{ (node_health_facts.cpu.scaling.max_khz | int / 1000) | round | int }} MHz (now {{ (node_health_facts.cpu.scaling.cur_khz | int / 1000) | round | int }} MHz)
{% endif %}
Memory {{ (node_health_facts.memory.mem_total_kb | int / 1048576) | round(1) }} GiB total, {{ (node_health_facts.memory.mem_available_kb | int / 1048576) | round(1) }} GiB available, swap {{ (node_health_facts.memory.swap_total_kb | int / 1048576) | round(2) }} GiB
ECC/EDAC {{ 'controllers: ' ~ (node_health_facts.memory.edac.controllers | map(attribute='name') | join(', ')) ~ ' CE=' ~ node_health_facts.memory.edac.ce_total | string ~ ' UE=' ~ node_health_facts.memory.edac.ue_total | string if node_health_facts.memory.edac.controllers_present else 'not exposed (no EDAC controller bound — ECC status UNKNOWN on this platform)' }}
Load {{ node_health_facts.cpu.loadavg | default([]) | join(' ') }}
--------------------------------------------------------------------------------
STORAGE
--------------------------------------------------------------------------------
{% for d in node_health_facts.storage.disks %}
{{ d.path }} {{ gib(d.size_bytes) }} {{ 'rotational' if d.rotational else 'non-rotating' }} {{ d.transport | default('?', true) }}{% if d.fstype %} whole-device {{ d.fstype }}{% endif %}{% if d.mountpoints %} mounted at {{ d.mountpoints | join(',') }}{% endif %}
model {{ d.model | default('(none reported)', true) }}{% if d.firmware_rev %} fw {{ d.firmware_rev }}{% endif %}{% if d.serial %} s/n {{ d.serial }}{% endif %}
{% if d.scsi_counters %}
kernel I/O completed={{ d.scsi_counters.iodone_cnt }} errors={{ d.scsi_counters.ioerr_cnt }} timeouts={{ d.scsi_counters.iotmo_cnt }}
{% endif %}
{% set sm = (node_health_smart | selectattr('path', 'equalto', d.path) | list | first) | default(none) %}
{% if sm and sm.available %}
SMART overall-health={{ 'PASSED' if sm.passed else 'FAILED' }} power_on_hours={{ sm.power_on_hours | default('?') }} power_cycles={{ sm.power_cycles | default('?') }}
lifetime_remaining={{ sm.lifetime_remaining_pct | default('?') }}% temp={{ sm.temperature_c | default('?') }}C
reallocated={{ sm.reallocated_sectors | default('?') }} pending={{ sm.pending_sectors | default('?') }} offline_uncorrectable={{ sm.offline_uncorrectable | default('?') }} crc_errors={{ sm.crc_errors | default('?') }}
{% for a in sm.attributes | default([]) %}
[{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}
{% endfor %}
{% else %}
SMART NOT READ — {{ (node_health_root_skip_reason if not node_health_root_available else ('smartmontools not installed on this node (ships with roles/node_baseline; this role never installs packages)' if not node_health_facts.tools.smartctl else 'device exposes no SMART data')) | regex_replace('\s+', ' ') | trim }}
{% endif %}
{% for p in d.partitions %}
{{ '%-12s' | format(p.kname) }} {{ '%9s' | format(gib(p.size_bytes)) }} {{ '%-6s' | format(p.fstype | default('-', true)) }} {{ (p.mountpoints[:2] | join(', ') ~ (' (+%d more)' | format(p.mountpoints | length - 2) if (p.mountpoints | length) > 2 else '')) if p.mountpoints else '(unmounted)' }}
{% endfor %}
{% endfor %}
{% if node_health_facts.storage.attached_volumes | default([]) %}
ATTACHED VOLUMES (iSCSI/FC — Longhorn replicas etc., not hardware in this box)
{% for a in node_health_facts.storage.attached_volumes %}
{{ '%-10s' | format(a.path) }} {{ '%9s' | format(gib(a.size_bytes)) }} {{ a.transport }} {{ a.model | default('-', true) }}
{% endfor %}
{% endif %}
MOUNTED FILESYSTEMS
{% for m in node_health_facts.storage.mounts | sort(attribute='mountpoint') %}
{{ '%-14s' | format(m.mountpoint) }} {{ '%-6s' | format(m.fstype) }} {{ '%9s' | format(gib(m.size_bytes)) }} total, {{ '%9s' | format(gib(m.avail_bytes)) }} free ({{ m.used_pct }}% used, inodes {{ (m.inodes_used_pct | string ~ '%') if m.inodes_used_pct is not none else 'n/a' }})
{% endfor %}
--------------------------------------------------------------------------------
THERMAL
--------------------------------------------------------------------------------
{% for z in node_health_facts.thermal.zones %}
zone {{ '%-18s' | format(z.type | default(z.zone)) }} {{ (z.temp_c | string ~ 'C') if z.temp_c is not none else 'unreadable (sensor returned no data)' }}{% if z.critical_c %} (critical {{ z.critical_c }}C){% endif %}
{% endfor %}
{% for chip in node_health_facts.thermal.hwmon %}
{% for t in chip.temps %}
{{ '%-16s' | format(chip.name | default('hwmon')) }} {{ '%-22s' | format(t.label) }} {{ (t.temp_c | string ~ 'C') if t.temp_c is not none else 'unreadable (sensor returned no data)' }}{% if t.crit_c %} (crit {{ t.crit_c }}C){% endif %}
{% endfor %}
{% for f in chip.fans %}
{{ '%-16s' | format(chip.name | default('hwmon')) }} {{ '%-22s' | format(f.label) }} {{ f.rpm }} rpm
{% endfor %}
{% endfor %}
--------------------------------------------------------------------------------
PRESSURE STALL (/proc/pressure — % of wall time stalled)
--------------------------------------------------------------------------------
{% for res in ['cpu', 'memory', 'io'] %}
{% set p = node_health_facts.pressure[res] %}
{% if p %}
{{ '%-7s' | format(res) }} some avg10={{ p.some.avg10 }} avg60={{ p.some.avg60 }} avg300={{ p.some.avg300 }}{% if p.full is defined %} | full avg60={{ p.full.avg60 }}{% endif %}
{% else %}
{{ '%-7s' | format(res) }} unavailable
{% endif %}
{% endfor %}
--------------------------------------------------------------------------------
NETWORK
--------------------------------------------------------------------------------
{% for n in node_health_facts.network.interfaces %}
{{ '%-10s' | format(n.name) }} {{ '%-6s' | format(n.operstate) }} {{ '%-8s' | format((n.speed_mbps | string ~ 'Mb/s') if n.speed_mbps else 'no-link') }} {{ '%-9s' | format(n.duplex | default('-')) }} mtu {{ n.mtu }} driver {{ n.driver | default('-') }}
rx {{ n.statistics.rx_packets | default('?') }} pkts / errors {{ n.statistics.rx_errors | default('?') }} / dropped {{ n.statistics.rx_dropped | default('?') }}
tx {{ n.statistics.tx_packets | default('?') }} pkts / errors {{ n.statistics.tx_errors | default('?') }} / dropped {{ n.statistics.tx_dropped | default('?') }}
{% endfor %}
nameservers {{ node_health_facts.network.nameservers | join(', ') | default('none configured', true) }}
--------------------------------------------------------------------------------
NOT READ (and why) — these are UNKNOWNS, not passes
--------------------------------------------------------------------------------
{% for c in node_health_checks if c.status == 'SKIP' %}
- {{ c.id }}: {{ c.detail | regex_replace('\s+', ' ') | trim | wordwrap(70) | indent(3) }}
{% else %}
(nothing skipped)
{% endfor %}
{% if node_health_facts.unreadable %}
Unreadable paths reported by the collector:
{% for u in node_health_facts.unreadable %}
- {{ u.path }} — {{ u.reason }}
{% endfor %}
{% endif %}
tools present: {% for t, p in node_health_facts.tools.items() %}{{ t }}={{ p | default('MISSING', true) }}{{ ', ' if not loop.last }}{% endfor %}
================================================================================