jakman85andClaude Opus 5
9f59c36000
Operator ask: implement the roles needed to check prod node hardware health, and run them to find out our state. This is that role; the run's findings are below. roles/node_health is a read-only sibling to cluster_verify (cluster health vs hardware health), styled after node_baseline. One unprivileged collector script does the work in a single SSH round trip — reading /sys/class/dmi, thermal + hwmon, EDAC, cpufreq + thermal_throttle, /proc/pressure, /sys/block incl. SCSI ioerr/iotmo counters, lsblk, lscpu, mounts + statvfs, per-NIC stats, resolv.conf. THE DESIGN POINT: an unreadable field is emitted as null WITH A RECORDED REASON, never as zero. The summary ends with a "NOT READ (and why) — these are UNKNOWNS, not passes" section. A fake zero on a reallocated-sector count is exactly the kind of false green this project refuses. 22 checks report PASS/WARN/FAIL/SKIP/INFO; every SKIP says why. Report-only by default; thresholds live in defaults/main.yml. Portable across bare metal AND VMs — validated against k3s-prod-1 (bare metal, no root) and k3s-dev-1 (VM, root). VM-absent things (SMART on virtio, coretemp, EDAC) skip cleanly, and iSCSI/Longhorn volumes are split out of the physical-disk set so a live replica is never mistaken for a spare device. playbooks/node-health.yml takes its target group as a variable (-e node_health_hosts= k3s_prod), deliberately unlike the other playbooks that hardcode k3s_servers — see the group-name gap already flagged in hosts.ini. Two become facts found the hard way and documented in the role: a group_vars `ansible_become: true` OUTRANKS `become: false` as a play/task keyword, so unprivileged tasks must override the variable; and `failed_when: false` does NOT catch "Missing sudo password" — only `ignore_errors` does. FIRST RUN, k3s-prod-1, 2026-08-31: 0 FAIL · 3 WARN · 12 PASS · 3 SKIP · 4 INFO. Box is healthy, cool and unstressed: i7-9700T 8C/8T, VT-x + /dev/kvm (KubeVirt OK), zero throttle events, hottest sensor 33C, PSI 0.00 across the board, 31.1 GiB RAM, eno1 up at 1000 Mb/s, BIOS R21 02.26.00, no reboot pending. DRIVE HEALTH IS STILL UNKNOWN — the headline. SMART did not run, blocked TWICE over: no vaulted ansible_become_password for prod, AND smartmontools is not on the node at all (node_baseline has never run there). Power-on hours, wear, reallocated/pending sectors and drive temperature all remain unmeasured on a ~2011 consumer SSD. WARNs: kernel ioerr_cnt=6 on sda (read cautiously — four freshly attached iSCSI disks on dev each showed a flat 7, suggesting a fixed per-device discovery cost, not media trouble; attribution needs dmesg/root); 0.95 GiB swap still active (node_baseline does the swapoff); one IPv4 resolver only (the other two upstreams are IPv6 Google), which refines rather than resolves the DNS-SPOF finding. ESCALATION — ECC status is UNKNOWABLE on this hardware: /sys/devices/system/edac/mc exists but no controller is bound. Non-ECC consumer RAM. Silent memory corruption on a prod node would be undetectable. Recorded as a SKIP with reason, not a pass. Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
79 lines
4.1 KiB
YAML
79 lines
4.1 KiB
YAML
---
|
|
# node_health — READ-ONLY hardware/health inspection of a node (bare metal or VM).
|
|
#
|
|
# Sibling of roles/cluster_verify: cluster_verify answers "is the CLUSTER healthy",
|
|
# node_health answers "is the HARDWARE under it healthy". Same posture — it inspects and
|
|
# reports, it never changes the node. Deliberately NOT part of node_baseline: baseline
|
|
# installs and configures, health only looks. That separation is why this role must never
|
|
# `apt install` anything (smartmontools is declared in node_baseline_packages; if it is
|
|
# missing here the SMART checks SKIP with a reason, they do not self-heal).
|
|
#
|
|
# Serves R-ENV-01 (reproducible node provisioning — you cannot certify a node you have not
|
|
# measured), R-ENV-03 (bare-metal parity) and R-OBS-04 (node-level observability).
|
|
|
|
# --- Reporting ---------------------------------------------------------------------
|
|
# Artifacts land under infra/ansible/_artifacts/node-health/, mirroring the repo's
|
|
# conformance/_artifacts/ convention: generated output, regenerated on every run, listed
|
|
# in .gitignore, never committed (it also carries drive serial numbers).
|
|
node_health_artifact_dir: "{{ playbook_dir }}/../_artifacts/node-health"
|
|
node_health_write_artifacts: true
|
|
# Print the human summary to the play output as well as writing it to the artifact dir.
|
|
node_health_show_summary: true
|
|
# Report-only by default: a health check that aborts the play hides the other 20 findings.
|
|
# Set true (e.g. in CI or a pre-build gate) to make any FAIL check end the play non-zero.
|
|
node_health_fail_on_fail: false
|
|
# Same, one step stricter: also fail on WARN.
|
|
node_health_fail_on_warn: false
|
|
|
|
# --- Root-only checks ---------------------------------------------------------------
|
|
# Everything this role can read WITHOUT root, it reads without root. Only SMART needs
|
|
# escalation (the ATA/NVMe passthrough ioctls are root-only). The role PROBES become
|
|
# first and, if escalation is unavailable (e.g. prod's vaulted become password is not
|
|
# present — infra/PROD-PROVISIONING.md §5), those checks report SKIPPED with the exact
|
|
# reason. Never a hard failure, and never a fake PASS: a skip is honest.
|
|
node_health_try_privileged: true
|
|
# --nocheck=standby: never spin up a sleeping disk just to read it. Read-only hygiene.
|
|
node_health_smartctl_args: "--json=c --all --nocheck=standby"
|
|
|
|
# --- Thresholds ---------------------------------------------------------------------
|
|
# All verdicts come from here, so a site can retune without touching task logic.
|
|
node_health_thresholds:
|
|
# Filesystems
|
|
fs_used_pct_warn: 80
|
|
fs_used_pct_fail: 90
|
|
inode_used_pct_warn: 80
|
|
# Memory (percentage of MemTotal still available)
|
|
mem_available_pct_warn: 15
|
|
mem_available_pct_fail: 5
|
|
# Temperatures (deg C, any sensor)
|
|
temp_warn_c: 70
|
|
temp_fail_c: 85
|
|
# Pressure-stall: "some" avg60, percent of wall time stalled. Dev learned to care about
|
|
# CPU PSI specifically (it shows saturation long before load average makes it obvious).
|
|
psi_some_avg60_warn: 10.0
|
|
psi_some_avg60_fail: 40.0
|
|
# Network
|
|
nic_min_speed_mbps: 1000
|
|
# A single upstream resolver is a known SPOF on this project — it broke containerd
|
|
# image pulls and CoreDNS forwarding on dev (DECISIONS-LOG 2026-08-31, finding 5).
|
|
min_nameservers: 2
|
|
# SMART (only evaluated when smartctl actually ran)
|
|
smart_reallocated_warn: 1
|
|
smart_reallocated_fail: 50
|
|
smart_pending_warn: 1
|
|
smart_pending_fail: 10
|
|
smart_lifetime_remaining_warn_pct: 20
|
|
smart_lifetime_remaining_fail_pct: 10
|
|
smart_temp_warn_c: 55
|
|
smart_temp_fail_c: 70
|
|
# Power-on hours past which a consumer SSD is simply old. Informational — age alone is
|
|
# not a failure, but it belongs in the operator's field of view when deciding whether a
|
|
# drive may hold production state (PROD-PROVISIONING.md §2).
|
|
smart_power_on_hours_notice: 40000
|
|
|
|
# ATA SMART attribute IDs surfaced verbatim in the report. Vendors disagree on names, so
|
|
# the report prints whatever the drive calls them and the checks key off the IDs.
|
|
node_health_smart_attribute_ids: [5, 9, 177, 179, 181, 182, 183, 184, 187, 188, 190,
|
|
194, 195, 196, 197, 198, 199, 202, 231, 232, 233,
|
|
241, 242, 246]
|