Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

feat(prod): node 3 onboarded — and it is NOT immune; BIOS-setting hypothesis disproven

Browse Source 
k3s-prod-3 online at the planned 192.168.2.12 (serial 8CC015307N, same chassis/BIOS/CPU),
enabled in inventory, vaulted become confirmed, package floor installed. Health: 0 FAIL /
4 WARN / 14 PASS / 1 SKIP — the only node with zero FAILs. WD Blue WDS500G2B0A SATA SSD at
100% life, 3213 h, 0 grown bad blocks, 0 erase fails. Matched Crucial 2x16 GB pair — the only
node with symmetric memory.

*** NODE 3 IS NOT IMMUNE TO THE §0b BUG. *** Its firmware DOES publish a MAT
(MEMATTR=0xbeece018); the ABSENCE of a memattr diagnostic line means the kernel read a VALID
table, which is the opposite of immunity. There are THREE outcomes, not two: bail-out on a
garbage version >2, a valid parse with a sane size, and the catastrophic path. Seven boots across
three machines show the MAT address MOVES BOOT-TO-BOOT on every node with settings constant, and
each node has been seen on more than one path. Blast radius widens from two nodes to ALL THREE;
all three need the >= 6.13 kernel.

THE CORRUPTING BYTES ARE THE OS BOOT CHAIN, not firmware and not DRAM: garbage version
1463964749 = 0x5742544D appeared on node 2 AND node 3 — two different machines, identical bytes.
The overwriter is the shared shim/GRUB/EFI-stub/initrd placement writing into the page the
firmware pointed at. So initrd size, module set and boot device are part of the causal chain.

BIOS-SETTING HYPOTHESIS: DISPROVEN, and the three-way diff cannot support it even in principle.
All 240 attributes read on all three; firmware identical fleet-wide. The discriminating set (both
working nodes agree, broken differs) is seven attributes, of which:
  - Fast Boot is the ONLY live candidate, and only as a probability-shifter — HP documents it
    'skips some preboot initialization steps', which changes the heap top and hence where the MAT
    lands. Different mechanism from the previously-killed MRC-caching theory; that stays dead.
  - Legacy Boot Options is AFFIRMATIVELY INERT: 'Configure Legacy Support and Secure Boot' =
    'Legacy Support Disable and Secure Boot Enable' on ALL THREE, Option ROM Launch Policy =
    'All UEFI' on all three. No CSM is built on node 2 either. A naive diff gets this wrong.
  - the rest are coincidence. After Power Loss is ELIMINATED by node 3, which works and shares
    node 2's Power Off — though it is now a §1 checklist violation on TWO nodes.
FATAL CONFOUND: the 1=3!=2 partition is perfectly collinear with node 2 being the only
single-DIMM AND the only NVMe-booting node. Adding node 3 did not break the confound, it DOUBLED
it. No setting can be the cause anyway: node 2 booted good once and bad twice with settings held
constant.

EXPERIMENT DESIGN RECORDED because the obvious approach is statistically worthless: p(bad) is
known only to n=3, 95% CI ~0.09-0.99, so even 8 clean boots prove nothing. Elimination needs ~20
boots/arm; an odds shift needs ~50. Zero-touch first step: 20 reboots of node 2 unchanged, logging
MEMATTR address vs outcome — both failures were at exactly 0xbeeb0018 and all five successes
elsewhere, so if outcome maps onto a narrow address range the mechanism is settled without
touching anything.

ROLE BUG FIXED: node 3's WD drive reports a SMART threshold for only 1 of 25 attributes, and
smartctl OMITS the key rather than emitting 0 — summary.txt.j2 dereferenced a.thresh bare in five
places and the report task died, so no artifact was written. Added a thr() macro rendering 'n/a'
(deliberately not 0, which would read as a real ATA cannot-trip threshold). Would have hit any
drive of this class.

Also: three nodes, three different disk models, only one NVMe, and every node has a single disk
with no unpartitioned spare — so §2's identical-layout requirement is unmet on all three.

Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-03 22:32:25 -05:00
1 parent 6e4d311545
commit b19c5db9ae
3 files changed
+179 -33
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

infra/ansible/inventory/hosts.ini
+19 -9
View File
@@ -48,13 +48,14 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
# =============================================================================
# PROD (k3s-prod-*) — Phase 7 PREP ONLY. See infra/PROD-PROVISIONING.md (T-7.1,
# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). Node 1
# (`k3s-prod-1`, HP EliteDesk 800 G5 Mini) is on the bench and SSH-reachable; nodes 2
# and 3 have not arrived. This section is inventory/group_vars scaffolding only —
# nothing has been run against node 1, and the operator has decided the CLUSTER stays
# uninitialized (no k3s install, no --cluster-init) until all three nodes are present
# (decision #2 in the DECISIONS-LOG entry above) — no single-server-etcd window, no
# later member-add dance.
# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). ALL THREE
# nodes are now on the bench and SSH-reachable (`k3s-prod-1` 2026-08-31, `k3s-prod-2`
# 2026-09-01, `k3s-prod-3` 2026-09-03) — every HP EliteDesk 800 G5 Mini. This section
# is still inventory/group_vars scaffolding only: nothing beyond the read-only health
# check and the node-baseline PACKAGE floor has been run, and the operator has decided
# the CLUSTER stays uninitialized (no k3s install, no --cluster-init) until the §2 disk
# decision is closed (decision #2 in the DECISIONS-LOG entry above) — no
# single-server-etcd window, no later member-add dance.
#
# KEY DIFFERENCE FROM DEV: prod sits DIRECTLY on the 192.168.2.x LAN — no pfSense NAT
# hop in front of it. Dev's `ansible_host` above is a NAT target (192.168.2.102:225x)
@@ -92,9 +93,18 @@ k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
# partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the
# nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision.
# node 3 remains commented until it is racked.
# k3s-prod-3 came online 2026-09-03 at the planned 192.168.2.12 — same HP EliteDesk 800 G5
# Mini, same BIOS R21 02.26.00, same i7-9700T, 2x16 GB (a MATCHED Crucial pair, unlike node
# 1's mismatched Microns), full 31.1 GiB usable. It is NOT structurally immune to the §0b
# memattr bug: its firmware DOES publish a MAT (MEMATTR=0xbeece018) and the kernel merely
# happened to parse it cleanly this boot, so node 3 is on the same lottery as node 1 and
# needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA
# SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now
# three-way, not a node-1 outlier.
# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately
# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2.
k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
# k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
# Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
# pattern; named to match the group_vars directory so Ansible auto-loads it).
infra/ansible/roles/node_health/templates/summary.txt.j2
+15 -4
View File
@@ -12,6 +12,17 @@
{% macro gib(n) -%}
{{ (n | int / 1073741824) | round(1) }} GiB
{%- endmacro %}
{# smartctl OMITS `thresh` entirely for any attribute whose threshold the drive does not
report (surfaced 2026-09-03 by k3s-prod-3's WDC WDS500G2B0A, which reports one for 1 of
its 25 attributes) — an ABSENT key, not thresh=0, so a bare `a.thresh` raised
"object of type 'dict' has no attribute 'thresh'" and killed the whole report. Render it
as `n/a` (the same token this template already uses for unreported readings) rather than
defaulting to 0, which would falsely read as a real ATA "cannot trip" threshold. The
collector already guards this — tasks/collect_smart.yml uses selectattr('thresh',
'defined') and `a.thresh | default(0)` — so only the template was exposed. #}
{% macro thr(a) -%}
{{ a.thresh | default('n/a') }}
{%- endmacro %}
================================================================================
NODE HEALTH — {{ inventory_hostname }} ({{ ansible_host | default('n/a') }})
collected {{ node_health_facts.collected_at }} as uid {{ node_health_facts.collector_uid }}
@@ -73,19 +84,19 @@
lifetime_remaining={{ (sm.lifetime_remaining_pct | string ~ '%') if sm.lifetime_remaining_pct is not none else 'not reported' }} temp={{ (sm.temperature_c | string ~ 'C') if sm.temperature_c is not none else 'not reported by this drive' }}
pending(197)={{ sm.pending_sectors | default('?') }} offline_uncorrectable(198)={{ sm.offline_uncorrectable | default('?') }} reported_uncorrect(187)={{ sm.reported_uncorrect | default('?') }} crc(199)={{ sm.crc_errors | default('?') }}
{% if sm.failing_now or sm.failed_past %}
THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ a.thresh }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ a.thresh }}]{% endfor %}
THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ thr(a) }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ thr(a) }}]{% endfor %}
{% elif sm.thresholds_present %}
no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ sm.prefail_closest.thresh }}{% endif %}
no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ thr(sm.prefail_closest) }}{% endif %}
{% endif %}
vendor context (NOT used for the verdict — raw units are vendor-defined):
reallocation events(196)={{ sm.realloc_events if sm.realloc_events is not none else 'n/a' }} grown bad blocks={{ sm.grown_bad_blocks if sm.grown_bad_blocks is not none else 'n/a' }} erase fails={{ sm.erase_fail_count if sm.erase_fail_count is not none else 'n/a' }} program fails={{ sm.program_fail_count if sm.program_fail_count is not none else 'n/a' }} unexpected power loss={{ sm.unexpected_power_loss if sm.unexpected_power_loss is not none else 'n/a' }}
{% if sm.realloc_attr %}
attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ sm.realloc_attr.thresh }} (the signal that is judged)
attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ thr(sm.realloc_attr) }} (the signal that is judged)
{% endif %}
{% for a in sm.attributes | default([]) %}
[{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %} <<< when_failed={{ a.when_failed }}{% endif %}
[{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ thr(a) }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %} <<< when_failed={{ a.when_failed }}{% endif %}
{% endfor %}
{% else %}