jakman85andClaude Opus 5
cf33338448
The fault is BOOT-TO-BOOT NON-DETERMINISTIC — the same machine booted correctly two boots
earlier — so it has to be detected on every health run, not diagnosed once. And node 1 is
exposed to the identical lottery on every reboot while currently looking healthy.
firmware.efi_memattr — reads the kernel ring buffer and distinguishes six states:
Failed to map EFI Memory Attributes table @ 0x.. -> FAIL (memory lost NOW)
Unexpected EFI Memory Attributes table version <N> -> WARN (corrupt table still published;
intact only by luck)
[Firmware Bug]: Corrupted EFI ... detected! -> PASS (the >=6.13 fix working)
no memattr line, log reaches early boot -> PASS (no table / accepted / not UEFI)
no memattr line, log does NOT reach early boot -> SKIP
dmesg unreadable -> SKIP
memory.physical_total — deliberately a SEPARATE check, not folded in: it measures the SYMPTOM
with no kernel log at all, so an unreadable-dmesg node is still not unmeasured. Two sub-verdicts,
worst wins: /proc/zoneinfo present-vs-managed (world-readable, works on metal and VMs — sum of
managed IS MemTotal) and MemTotal vs the DIMM total from dmidecode. Thresholds 10% WARN / 25%
FAIL, set from measured baselines: 2.0% prod-1, 2.1% workstation, 2.2% dev VM, vs 83.4% prod-2.
THREE GUARDS AGAINST A FALSE PASS, because "no memattr line" must never silently mean "fine":
1. dmesg failure yields available:false with the exact reason (rc, uid, kernel.dmesg_restrict) -> SKIP
2. the ring buffer WRAPS and every memattr line prints in the first fraction of a second, so the
collector records a boot_covered anchor (Linux version / BIOS-provided physical RAM map); no
memattr line WITHOUT that anchor is a SKIP, not a PASS
3. memory.physical_total needs no kernel log at all
Verified by forcing the unprivileged path: SKIP with the EPERM reason while zoneinfo still PASSed.
VERDICTS, confirmed live:
k3s-prod-1 WARN "EXPOSED, not immune ... survived only because its garbage version field read
1030, which is > 2 ... one unlucky boot from the failure state" + PASS 2.0%
k3s-prod-2 FAIL "THIS NODE HAS LOST MEMORY TO THE BUG ... at 0xbeeb0018" + FAIL 83.4%/83.6%
k3s-dev-* PASS both, NO false finding: "no UEFI config-table line ... did not boot via UEFI.
Log reaches back to early boot, so that absence is measured, not assumed."
Both failure texts carry the remediation verbatim, including that there is NO kernel cmdline
workaround because efi_memattr_init() runs before any efi= bail-out.
Root-gating follows the role's existing pattern: --privileged re-runs THE SAME collector under
become, only when the unprivileged pass actually failed to read something. Reimplementing the log
grep and SMBIOS parse in Jinja would have been a second source of truth for the same fact.
LATENT BUG FIXED in passing: the collector's run() used bare shutil.which(), so dmidecode (in
/usr/sbin) was invisible under the unprivileged k3s PATH and produced a FALSE "not installed"
skip reason. Now shares the sbin-aware which() collect_tools() already had.
INCIDENTAL FINDING recorded in §0b: node 1's DIMMs are NOT a matched pair — DIMM1 Micron
8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity
is right, but different rank organisation and rated speed, so the pair runs at 2667. Not a fault;
recorded because §2 requires the three nodes to end up symmetric and "2×16 GB Micron" hid it.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
73 lines
3.4 KiB
YAML
73 lines
3.4 KiB
YAML
---
|
|
# node_health — read-only hardware health inspection. See defaults/main.yml for the why.
|
|
#
|
|
# ORDER OF OPERATIONS
|
|
# 1. Collect everything readable WITHOUT root (one script, one round trip).
|
|
# 2. Probe whether privilege escalation actually works on this host.
|
|
# 3. Only if it does, and only for what the first pass could NOT read: re-run the same
|
|
# collector with --privileged (kernel ring buffer, SMBIOS DIMM inventory).
|
|
# 4. Only if it does AND smartmontools is installed: read SMART.
|
|
# 5. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
|
|
# 6. Render the human summary + write the JSON artifact.
|
|
#
|
|
# BECOME HANDLING — the subtle bit, read before editing.
|
|
# `ansible_become` set as an inventory/group var (which is exactly what
|
|
# inventory/group_vars/k3s_prod/vars.yml does for prod) is a CONNECTION VARIABLE, and
|
|
# connection variables OUTRANK the `become:` play/task keyword. Verified on this repo's
|
|
# ansible-core 2.20: a play or task saying `become: false` still tries to sudo, and on
|
|
# prod (no NOPASSWD, vaulted password) dies with "Missing sudo password" before the
|
|
# module ever runs. So the unprivileged tasks here must override the VARIABLE, not the
|
|
# keyword — hence `vars: ansible_become: false` on the include below. That makes the role
|
|
# self-contained: it behaves identically whatever the calling play sets.
|
|
#
|
|
# Equally verified: `failed_when: false` does NOT catch a missing-sudo-password error
|
|
# (it is raised by the become plugin before the task's failure expression is evaluated).
|
|
# `ignore_errors: true` DOES, and still populates the registered variable. That is why
|
|
# the escalation probe below uses ignore_errors — anything else turns "no vault yet" into
|
|
# a hard, misleading play failure.
|
|
|
|
- name: Initialise the health record
|
|
ansible.builtin.set_fact:
|
|
node_health_checks: []
|
|
node_health_smart: []
|
|
node_health_root_available: false
|
|
node_health_root_skip_reason: "privileged checks not attempted (node_health_try_privileged=false)"
|
|
vars:
|
|
ansible_become: false
|
|
|
|
- name: Collect unprivileged hardware facts
|
|
ansible.builtin.include_tasks: collect_unprivileged.yml
|
|
# Applies to every task in the included file (dynamic includes pass their vars down).
|
|
vars:
|
|
ansible_become: false
|
|
|
|
- name: Probe privilege escalation
|
|
ansible.builtin.include_tasks: probe_privilege.yml
|
|
when: node_health_try_privileged | bool
|
|
|
|
# Only when the unprivileged pass actually failed to read one of them. On a node with
|
|
# kernel.dmesg_restrict=0 the first pass already has the kernel log and this is skipped
|
|
# entirely — "everything readable WITHOUT root is read without root" (defaults/main.yml).
|
|
- name: Re-read the root-only facts with escalation (kernel log, SMBIOS DIMMs)
|
|
ansible.builtin.include_tasks: collect_privileged.yml
|
|
when:
|
|
- node_health_root_available | bool
|
|
- (not (node_health_facts.klog.available | bool))
|
|
or (not (node_health_facts.memory.devices.available | bool))
|
|
|
|
- name: Read SMART data (privileged)
|
|
ansible.builtin.include_tasks: collect_smart.yml
|
|
when:
|
|
- node_health_root_available | bool
|
|
- node_health_facts.tools.smartctl is truthy
|
|
|
|
- name: Assess collected facts into explicit checks
|
|
ansible.builtin.include_tasks: assess.yml
|
|
vars:
|
|
ansible_become: false
|
|
|
|
- name: Report (human summary + JSON artifact)
|
|
ansible.builtin.include_tasks: report.yml
|
|
vars:
|
|
ansible_become: false
|