Files
home-network-k3s-impl/infra/ansible/roles/node_health/templates/summary.txt.j2
T
jakman85andClaude Opus 5 c1f24e4a6d fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03)
Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.

The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.

MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
  * systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
    symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
    loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
    FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
    silently bypass the local zone and ad-blocking.
  * resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
    lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
    would destroy PiHole-first ordering.
  * contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
    and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.

SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.

PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.

tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.

ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.

Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.

Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 00:11:49 -05:00

155 lines
11 KiB
Django/Jinja
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{# ===========================================================================
node_health human summary. Rendered ONCE (tasks/report.yml) and used for both
the on-screen report and the .txt artifact, so the two can never drift.
Detail strings are normalised with regex_replace('\s+',' ') because the check
details are authored as YAML folded scalars in assess.yml and can carry stray
newlines/indentation from that folding.
=========================================================================== #}
{% macro fmt(text, width=74, pad=10) -%}
{{ text | regex_replace('\s+', ' ') | trim | wordwrap(width) | indent(pad, first=True) }}
{%- endmacro %}
{% macro gib(n) -%}
{{ (n | int / 1073741824) | round(1) }} GiB
{%- endmacro %}
================================================================================
NODE HEALTH — {{ inventory_hostname }} ({{ ansible_host | default('n/a') }})
collected {{ node_health_facts.collected_at }} as uid {{ node_health_facts.collector_uid }}
================================================================================
VERDICT {{ node_health_counts.FAIL }} FAIL · {{ node_health_counts.WARN }} WARN · {{ node_health_counts.PASS }} PASS · {{ node_health_counts.SKIP }} SKIP · {{ node_health_counts.INFO }} INFO
{{ 'privileged checks RAN' if node_health_root_available | bool else 'privileged checks SKIPPED — root not available' }}
--------------------------------------------------------------------------------
CHECKS
--------------------------------------------------------------------------------
{% for c in node_health_checks %}
[{{ '%-4s' | format(c.status) }}] {{ c.id }} — {{ c.title }}
{{ fmt(c.detail) }}
{% endfor %}
--------------------------------------------------------------------------------
MACHINE — {{ node_health_facts.dmi.sys_vendor | default('?') }} {{ node_health_facts.dmi.product_name | default('?') }}
--------------------------------------------------------------------------------
OS {{ node_health_facts.os.pretty_name | default('?') }}
Kernel {{ node_health_facts.os.kernel | default('?') }}
Uptime {{ (node_health_facts.os.uptime_seconds | float / 86400) | round(2) }} days
BIOS {{ node_health_facts.dmi.bios_vendor | default('?') }} {{ node_health_facts.dmi.bios_version | default('?') }} ({{ node_health_facts.dmi.bios_date | default('?') }}){% if node_health_facts.dmi.ec_firmware_release %}, EC {{ node_health_facts.dmi.ec_firmware_release }}{% endif %}
Board {{ node_health_facts.dmi.board_vendor | default('?') }} {{ node_health_facts.dmi.board_name | default('?') }}
CPU {{ node_health_facts.cpu.model | default('?') }}
{{ node_health_facts.cpu.cpus | default('?') }} logical / {{ node_health_facts.cpu.sockets | default('?') }} socket(s) x {{ node_health_facts.cpu.cores_per_socket | default('?') }} core(s) x {{ node_health_facts.cpu.threads_per_core | default('?') }} thread(s)
virtualisation: {{ node_health_facts.cpu.virtualization | default('none reported') }}, /dev/kvm={{ node_health_facts.cpu.kvm_device }}
{% if node_health_facts.cpu.scaling %}
scaling: {{ node_health_facts.cpu.scaling.driver }}/{{ node_health_facts.cpu.scaling.governor }}, {{ (node_health_facts.cpu.scaling.min_khz | int / 1000) | round | int }}–{{ (node_health_facts.cpu.scaling.max_khz | int / 1000) | round | int }} MHz (now {{ (node_health_facts.cpu.scaling.cur_khz | int / 1000) | round | int }} MHz)
{% endif %}
Memory {{ (node_health_facts.memory.mem_total_kb | int / 1048576) | round(1) }} GiB total, {{ (node_health_facts.memory.mem_available_kb | int / 1048576) | round(1) }} GiB available, swap {{ (node_health_facts.memory.swap_total_kb | int / 1048576) | round(2) }} GiB
ECC/EDAC {{ 'controllers: ' ~ (node_health_facts.memory.edac.controllers | map(attribute='name') | join(', ')) ~ ' CE=' ~ node_health_facts.memory.edac.ce_total | string ~ ' UE=' ~ node_health_facts.memory.edac.ue_total | string if node_health_facts.memory.edac.controllers_present else 'not exposed (no EDAC controller bound — ECC status UNKNOWN on this platform)' }}
Load {{ node_health_facts.cpu.loadavg | default([]) | join(' ') }}
--------------------------------------------------------------------------------
STORAGE
--------------------------------------------------------------------------------
{% for d in node_health_facts.storage.disks %}
{{ d.path }} {{ gib(d.size_bytes) }} {{ 'rotational' if d.rotational else 'non-rotating' }} {{ d.transport | default('?', true) }}{% if d.fstype %} whole-device {{ d.fstype }}{% endif %}{% if d.mountpoints %} mounted at {{ d.mountpoints | join(',') }}{% endif %}
model {{ d.model | default('(none reported)', true) }}{% if d.firmware_rev %} fw {{ d.firmware_rev }}{% endif %}{% if d.serial %} s/n {{ d.serial }}{% endif %}
{% if d.scsi_counters %}
kernel I/O completed={{ d.scsi_counters.iodone_cnt }} errors={{ d.scsi_counters.ioerr_cnt }} timeouts={{ d.scsi_counters.iotmo_cnt }}
{% endif %}
{% set sm = (node_health_smart | selectattr('path', 'equalto', d.path) | list | first) | default(none) %}
{% if sm and sm.available %}
SMART overall-health={{ 'PASSED' if sm.passed else 'FAILED' }} power_on_hours={{ sm.power_on_hours | default('?') }} power_cycles={{ sm.power_cycles | default('?') }}
lifetime_remaining={{ sm.lifetime_remaining_pct | default('?') }}% temp={{ sm.temperature_c | default('?') }}C
reallocated={{ sm.reallocated_sectors | default('?') }} pending={{ sm.pending_sectors | default('?') }} offline_uncorrectable={{ sm.offline_uncorrectable | default('?') }} crc_errors={{ sm.crc_errors | default('?') }}
{% for a in sm.attributes | default([]) %}
[{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}
{% endfor %}
{% else %}
SMART NOT READ — {{ (node_health_root_skip_reason if not node_health_root_available else ('smartmontools not installed on this node (ships with roles/node_baseline; this role never installs packages)' if not node_health_facts.tools.smartctl else 'device exposes no SMART data')) | regex_replace('\s+', ' ') | trim }}
{% endif %}
{% for p in d.partitions %}
{{ '%-12s' | format(p.kname) }} {{ '%9s' | format(gib(p.size_bytes)) }} {{ '%-6s' | format(p.fstype | default('-', true)) }} {{ (p.mountpoints[:2] | join(', ') ~ (' (+%d more)' | format(p.mountpoints | length - 2) if (p.mountpoints | length) > 2 else '')) if p.mountpoints else '(unmounted)' }}
{% endfor %}
{% endfor %}
{% if node_health_facts.storage.attached_volumes | default([]) %}
ATTACHED VOLUMES (iSCSI/FC — Longhorn replicas etc., not hardware in this box)
{% for a in node_health_facts.storage.attached_volumes %}
{{ '%-10s' | format(a.path) }} {{ '%9s' | format(gib(a.size_bytes)) }} {{ a.transport }} {{ a.model | default('-', true) }}
{% endfor %}
{% endif %}
MOUNTED FILESYSTEMS
{% for m in node_health_facts.storage.mounts | sort(attribute='mountpoint') %}
{{ '%-14s' | format(m.mountpoint) }} {{ '%-6s' | format(m.fstype) }} {{ '%9s' | format(gib(m.size_bytes)) }} total, {{ '%9s' | format(gib(m.avail_bytes)) }} free ({{ m.used_pct }}% used, inodes {{ (m.inodes_used_pct | string ~ '%') if m.inodes_used_pct is not none else 'n/a' }})
{% endfor %}
--------------------------------------------------------------------------------
THERMAL
--------------------------------------------------------------------------------
{% for z in node_health_facts.thermal.zones %}
zone {{ '%-18s' | format(z.type | default(z.zone)) }} {{ (z.temp_c | string ~ 'C') if z.temp_c is not none else 'unreadable (sensor returned no data)' }}{% if z.critical_c %} (critical {{ z.critical_c }}C){% endif %}
{% endfor %}
{% for chip in node_health_facts.thermal.hwmon %}
{% for t in chip.temps %}
{{ '%-16s' | format(chip.name | default('hwmon')) }} {{ '%-22s' | format(t.label) }} {{ (t.temp_c | string ~ 'C') if t.temp_c is not none else 'unreadable (sensor returned no data)' }}{% if t.crit_c %} (crit {{ t.crit_c }}C){% endif %}
{% endfor %}
{% for f in chip.fans %}
{{ '%-16s' | format(chip.name | default('hwmon')) }} {{ '%-22s' | format(f.label) }} {{ f.rpm }} rpm
{% endfor %}
{% endfor %}
--------------------------------------------------------------------------------
PRESSURE STALL (/proc/pressure — % of wall time stalled)
--------------------------------------------------------------------------------
{% for res in ['cpu', 'memory', 'io'] %}
{% set p = node_health_facts.pressure[res] %}
{% if p %}
{{ '%-7s' | format(res) }} some avg10={{ p.some.avg10 }} avg60={{ p.some.avg60 }} avg300={{ p.some.avg300 }}{% if p.full is defined %} | full avg60={{ p.full.avg60 }}{% endif %}
{% else %}
{{ '%-7s' | format(res) }} unavailable
{% endif %}
{% endfor %}
--------------------------------------------------------------------------------
NETWORK
--------------------------------------------------------------------------------
{% for n in node_health_facts.network.interfaces %}
{{ '%-10s' | format(n.name) }} {{ '%-6s' | format(n.operstate) }} {{ '%-8s' | format((n.speed_mbps | string ~ 'Mb/s') if n.speed_mbps else 'no-link') }} {{ '%-9s' | format(n.duplex | default('-')) }} mtu {{ n.mtu }} driver {{ n.driver | default('-') }}
rx {{ n.statistics.rx_packets | default('?') }} pkts / errors {{ n.statistics.rx_errors | default('?') }} / dropped {{ n.statistics.rx_dropped | default('?') }}
tx {{ n.statistics.tx_packets | default('?') }} pkts / errors {{ n.statistics.tx_errors | default('?') }} / dropped {{ n.statistics.tx_dropped | default('?') }}
{% endfor %}
{% if node_health_facts.network.nameservers is none %}
nameservers NOT READ — {{ node_health_facts.network.resolver_unavailable_reason | default('the collector recorded no reason', true) | regex_replace('\s+', ' ') | trim | wordwrap(66) | indent(13) }}
{% else %}
nameservers {{ node_health_facts.network.nameservers | join(', ') | default('none configured', true) }}
{% if node_health_facts.network.resolver_stub | default([], true) | length > 0 %}
(/etc/resolv.conf holds only the systemd-resolved stub {{ node_health_facts.network.resolver_stub | join(', ') }}; the servers above are the real upstreams behind it)
{% endif %}
{% endif %}
--------------------------------------------------------------------------------
NOT READ (and why) — these are UNKNOWNS, not passes
--------------------------------------------------------------------------------
{% for c in node_health_checks if c.status == 'SKIP' %}
- {{ c.id }}: {{ c.detail | regex_replace('\s+', ' ') | trim | wordwrap(70) | indent(3) }}
{% else %}
(nothing skipped)
{% endfor %}
{% if node_health_facts.unreadable %}
Unreadable paths reported by the collector:
{% for u in node_health_facts.unreadable %}
- {{ u.path }} — {{ u.reason }}
{% endfor %}
{% endif %}
tools present: {% for t, p in node_health_facts.tools.items() %}{{ t }}={{ p | default('MISSING', true) }}{{ ', ' if not loop.last }}{% endfor %}
================================================================================