jakman85andClaude Opus 5
9f59c36000
Operator ask: implement the roles needed to check prod node hardware health, and run them to find out our state. This is that role; the run's findings are below. roles/node_health is a read-only sibling to cluster_verify (cluster health vs hardware health), styled after node_baseline. One unprivileged collector script does the work in a single SSH round trip — reading /sys/class/dmi, thermal + hwmon, EDAC, cpufreq + thermal_throttle, /proc/pressure, /sys/block incl. SCSI ioerr/iotmo counters, lsblk, lscpu, mounts + statvfs, per-NIC stats, resolv.conf. THE DESIGN POINT: an unreadable field is emitted as null WITH A RECORDED REASON, never as zero. The summary ends with a "NOT READ (and why) — these are UNKNOWNS, not passes" section. A fake zero on a reallocated-sector count is exactly the kind of false green this project refuses. 22 checks report PASS/WARN/FAIL/SKIP/INFO; every SKIP says why. Report-only by default; thresholds live in defaults/main.yml. Portable across bare metal AND VMs — validated against k3s-prod-1 (bare metal, no root) and k3s-dev-1 (VM, root). VM-absent things (SMART on virtio, coretemp, EDAC) skip cleanly, and iSCSI/Longhorn volumes are split out of the physical-disk set so a live replica is never mistaken for a spare device. playbooks/node-health.yml takes its target group as a variable (-e node_health_hosts= k3s_prod), deliberately unlike the other playbooks that hardcode k3s_servers — see the group-name gap already flagged in hosts.ini. Two become facts found the hard way and documented in the role: a group_vars `ansible_become: true` OUTRANKS `become: false` as a play/task keyword, so unprivileged tasks must override the variable; and `failed_when: false` does NOT catch "Missing sudo password" — only `ignore_errors` does. FIRST RUN, k3s-prod-1, 2026-08-31: 0 FAIL · 3 WARN · 12 PASS · 3 SKIP · 4 INFO. Box is healthy, cool and unstressed: i7-9700T 8C/8T, VT-x + /dev/kvm (KubeVirt OK), zero throttle events, hottest sensor 33C, PSI 0.00 across the board, 31.1 GiB RAM, eno1 up at 1000 Mb/s, BIOS R21 02.26.00, no reboot pending. DRIVE HEALTH IS STILL UNKNOWN — the headline. SMART did not run, blocked TWICE over: no vaulted ansible_become_password for prod, AND smartmontools is not on the node at all (node_baseline has never run there). Power-on hours, wear, reallocated/pending sectors and drive temperature all remain unmeasured on a ~2011 consumer SSD. WARNs: kernel ioerr_cnt=6 on sda (read cautiously — four freshly attached iSCSI disks on dev each showed a flat 7, suggesting a fixed per-device discovery cost, not media trouble; attribution needs dmesg/root); 0.95 GiB swap still active (node_baseline does the swapoff); one IPv4 resolver only (the other two upstreams are IPv6 Google), which refines rather than resolves the DNS-SPOF finding. ESCALATION — ECC status is UNKNOWABLE on this hardware: /sys/devices/system/edac/mc exists but no controller is bound. Non-ECC consumer RAM. Silent memory corruption on a prod node would be undetectable. Recorded as a SKIP with reason, not a pass. Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
61 lines
2.6 KiB
YAML
61 lines
2.6 KiB
YAML
---
|
|
# node_health — read-only hardware health inspection. See defaults/main.yml for the why.
|
|
#
|
|
# ORDER OF OPERATIONS
|
|
# 1. Collect everything readable WITHOUT root (one script, one round trip).
|
|
# 2. Probe whether privilege escalation actually works on this host.
|
|
# 3. Only if it does AND smartmontools is installed: read SMART.
|
|
# 4. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
|
|
# 5. Render the human summary + write the JSON artifact.
|
|
#
|
|
# BECOME HANDLING — the subtle bit, read before editing.
|
|
# `ansible_become` set as an inventory/group var (which is exactly what
|
|
# inventory/group_vars/k3s_prod/vars.yml does for prod) is a CONNECTION VARIABLE, and
|
|
# connection variables OUTRANK the `become:` play/task keyword. Verified on this repo's
|
|
# ansible-core 2.20: a play or task saying `become: false` still tries to sudo, and on
|
|
# prod (no NOPASSWD, vaulted password) dies with "Missing sudo password" before the
|
|
# module ever runs. So the unprivileged tasks here must override the VARIABLE, not the
|
|
# keyword — hence `vars: ansible_become: false` on the include below. That makes the role
|
|
# self-contained: it behaves identically whatever the calling play sets.
|
|
#
|
|
# Equally verified: `failed_when: false` does NOT catch a missing-sudo-password error
|
|
# (it is raised by the become plugin before the task's failure expression is evaluated).
|
|
# `ignore_errors: true` DOES, and still populates the registered variable. That is why
|
|
# the escalation probe below uses ignore_errors — anything else turns "no vault yet" into
|
|
# a hard, misleading play failure.
|
|
|
|
- name: Initialise the health record
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: []
|
|
node_health_smart: []
|
|
node_health_root_available: false
|
|
node_health_root_skip_reason: "privileged checks not attempted (node_health_try_privileged=false)"
|
|
vars:
|
|
ansible_become: false
|
|
|
|
- name: Collect unprivileged hardware facts
|
|
ansible.builtin.include_tasks: collect_unprivileged.yml
|
|
# Applies to every task in the included file (dynamic includes pass their vars down).
|
|
vars:
|
|
ansible_become: false
|
|
|
|
- name: Probe privilege escalation
|
|
ansible.builtin.include_tasks: probe_privilege.yml
|
|
when: node_health_try_privileged | bool
|
|
|
|
- name: Read SMART data (privileged)
|
|
ansible.builtin.include_tasks: collect_smart.yml
|
|
when:
|
|
- node_health_root_available | bool
|
|
- node_health_facts.tools.smartctl is truthy
|
|
|
|
- name: Assess collected facts into explicit checks
|
|
ansible.builtin.include_tasks: assess.yml
|
|
vars:
|
|
ansible_become: false
|
|
|
|
- name: Report (human summary + JSON artifact)
|
|
ansible.builtin.include_tasks: report.yml
|
|
vars:
|
|
ansible_become: false
|