Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.
The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.
MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
* systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
silently bypass the local zone and ad-blocking.
* resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
would destroy PiHole-first ordering.
* contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.
SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.
PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.
tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.
ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.
Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
155 lines
11 KiB
Django/Jinja
155 lines
11 KiB
Django/Jinja
{# ===========================================================================
|
||
node_health human summary. Rendered ONCE (tasks/report.yml) and used for both
|
||
the on-screen report and the .txt artifact, so the two can never drift.
|
||
|
||
Detail strings are normalised with regex_replace('\s+',' ') because the check
|
||
details are authored as YAML folded scalars in assess.yml and can carry stray
|
||
newlines/indentation from that folding.
|
||
=========================================================================== #}
|
||
{% macro fmt(text, width=74, pad=10) -%}
|
||
{{ text | regex_replace('\s+', ' ') | trim | wordwrap(width) | indent(pad, first=True) }}
|
||
{%- endmacro %}
|
||
{% macro gib(n) -%}
|
||
{{ (n | int / 1073741824) | round(1) }} GiB
|
||
{%- endmacro %}
|
||
================================================================================
|
||
NODE HEALTH — {{ inventory_hostname }} ({{ ansible_host | default('n/a') }})
|
||
collected {{ node_health_facts.collected_at }} as uid {{ node_health_facts.collector_uid }}
|
||
================================================================================
|
||
|
||
VERDICT {{ node_health_counts.FAIL }} FAIL · {{ node_health_counts.WARN }} WARN · {{ node_health_counts.PASS }} PASS · {{ node_health_counts.SKIP }} SKIP · {{ node_health_counts.INFO }} INFO
|
||
{{ 'privileged checks RAN' if node_health_root_available | bool else 'privileged checks SKIPPED — root not available' }}
|
||
|
||
--------------------------------------------------------------------------------
|
||
CHECKS
|
||
--------------------------------------------------------------------------------
|
||
{% for c in node_health_checks %}
|
||
[{{ '%-4s' | format(c.status) }}] {{ c.id }} — {{ c.title }}
|
||
{{ fmt(c.detail) }}
|
||
{% endfor %}
|
||
|
||
--------------------------------------------------------------------------------
|
||
MACHINE — {{ node_health_facts.dmi.sys_vendor | default('?') }} {{ node_health_facts.dmi.product_name | default('?') }}
|
||
--------------------------------------------------------------------------------
|
||
OS {{ node_health_facts.os.pretty_name | default('?') }}
|
||
Kernel {{ node_health_facts.os.kernel | default('?') }}
|
||
Uptime {{ (node_health_facts.os.uptime_seconds | float / 86400) | round(2) }} days
|
||
BIOS {{ node_health_facts.dmi.bios_vendor | default('?') }} {{ node_health_facts.dmi.bios_version | default('?') }} ({{ node_health_facts.dmi.bios_date | default('?') }}){% if node_health_facts.dmi.ec_firmware_release %}, EC {{ node_health_facts.dmi.ec_firmware_release }}{% endif %}
|
||
|
||
Board {{ node_health_facts.dmi.board_vendor | default('?') }} {{ node_health_facts.dmi.board_name | default('?') }}
|
||
CPU {{ node_health_facts.cpu.model | default('?') }}
|
||
{{ node_health_facts.cpu.cpus | default('?') }} logical / {{ node_health_facts.cpu.sockets | default('?') }} socket(s) x {{ node_health_facts.cpu.cores_per_socket | default('?') }} core(s) x {{ node_health_facts.cpu.threads_per_core | default('?') }} thread(s)
|
||
virtualisation: {{ node_health_facts.cpu.virtualization | default('none reported') }}, /dev/kvm={{ node_health_facts.cpu.kvm_device }}
|
||
{% if node_health_facts.cpu.scaling %}
|
||
scaling: {{ node_health_facts.cpu.scaling.driver }}/{{ node_health_facts.cpu.scaling.governor }}, {{ (node_health_facts.cpu.scaling.min_khz | int / 1000) | round | int }}–{{ (node_health_facts.cpu.scaling.max_khz | int / 1000) | round | int }} MHz (now {{ (node_health_facts.cpu.scaling.cur_khz | int / 1000) | round | int }} MHz)
|
||
{% endif %}
|
||
Memory {{ (node_health_facts.memory.mem_total_kb | int / 1048576) | round(1) }} GiB total, {{ (node_health_facts.memory.mem_available_kb | int / 1048576) | round(1) }} GiB available, swap {{ (node_health_facts.memory.swap_total_kb | int / 1048576) | round(2) }} GiB
|
||
ECC/EDAC {{ 'controllers: ' ~ (node_health_facts.memory.edac.controllers | map(attribute='name') | join(', ')) ~ ' CE=' ~ node_health_facts.memory.edac.ce_total | string ~ ' UE=' ~ node_health_facts.memory.edac.ue_total | string if node_health_facts.memory.edac.controllers_present else 'not exposed (no EDAC controller bound — ECC status UNKNOWN on this platform)' }}
|
||
Load {{ node_health_facts.cpu.loadavg | default([]) | join(' ') }}
|
||
|
||
--------------------------------------------------------------------------------
|
||
STORAGE
|
||
--------------------------------------------------------------------------------
|
||
{% for d in node_health_facts.storage.disks %}
|
||
{{ d.path }} {{ gib(d.size_bytes) }} {{ 'rotational' if d.rotational else 'non-rotating' }} {{ d.transport | default('?', true) }}{% if d.fstype %} whole-device {{ d.fstype }}{% endif %}{% if d.mountpoints %} mounted at {{ d.mountpoints | join(',') }}{% endif %}
|
||
|
||
model {{ d.model | default('(none reported)', true) }}{% if d.firmware_rev %} fw {{ d.firmware_rev }}{% endif %}{% if d.serial %} s/n {{ d.serial }}{% endif %}
|
||
|
||
{% if d.scsi_counters %}
|
||
kernel I/O completed={{ d.scsi_counters.iodone_cnt }} errors={{ d.scsi_counters.ioerr_cnt }} timeouts={{ d.scsi_counters.iotmo_cnt }}
|
||
{% endif %}
|
||
{% set sm = (node_health_smart | selectattr('path', 'equalto', d.path) | list | first) | default(none) %}
|
||
{% if sm and sm.available %}
|
||
SMART overall-health={{ 'PASSED' if sm.passed else 'FAILED' }} power_on_hours={{ sm.power_on_hours | default('?') }} power_cycles={{ sm.power_cycles | default('?') }}
|
||
lifetime_remaining={{ sm.lifetime_remaining_pct | default('?') }}% temp={{ sm.temperature_c | default('?') }}C
|
||
reallocated={{ sm.reallocated_sectors | default('?') }} pending={{ sm.pending_sectors | default('?') }} offline_uncorrectable={{ sm.offline_uncorrectable | default('?') }} crc_errors={{ sm.crc_errors | default('?') }}
|
||
{% for a in sm.attributes | default([]) %}
|
||
[{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}
|
||
{% endfor %}
|
||
{% else %}
|
||
SMART NOT READ — {{ (node_health_root_skip_reason if not node_health_root_available else ('smartmontools not installed on this node (ships with roles/node_baseline; this role never installs packages)' if not node_health_facts.tools.smartctl else 'device exposes no SMART data')) | regex_replace('\s+', ' ') | trim }}
|
||
{% endif %}
|
||
{% for p in d.partitions %}
|
||
{{ '%-12s' | format(p.kname) }} {{ '%9s' | format(gib(p.size_bytes)) }} {{ '%-6s' | format(p.fstype | default('-', true)) }} {{ (p.mountpoints[:2] | join(', ') ~ (' (+%d more)' | format(p.mountpoints | length - 2) if (p.mountpoints | length) > 2 else '')) if p.mountpoints else '(unmounted)' }}
|
||
{% endfor %}
|
||
|
||
{% endfor %}
|
||
{% if node_health_facts.storage.attached_volumes | default([]) %}
|
||
ATTACHED VOLUMES (iSCSI/FC — Longhorn replicas etc., not hardware in this box)
|
||
{% for a in node_health_facts.storage.attached_volumes %}
|
||
{{ '%-10s' | format(a.path) }} {{ '%9s' | format(gib(a.size_bytes)) }} {{ a.transport }} {{ a.model | default('-', true) }}
|
||
{% endfor %}
|
||
|
||
{% endif %}
|
||
MOUNTED FILESYSTEMS
|
||
{% for m in node_health_facts.storage.mounts | sort(attribute='mountpoint') %}
|
||
{{ '%-14s' | format(m.mountpoint) }} {{ '%-6s' | format(m.fstype) }} {{ '%9s' | format(gib(m.size_bytes)) }} total, {{ '%9s' | format(gib(m.avail_bytes)) }} free ({{ m.used_pct }}% used, inodes {{ (m.inodes_used_pct | string ~ '%') if m.inodes_used_pct is not none else 'n/a' }})
|
||
{% endfor %}
|
||
|
||
--------------------------------------------------------------------------------
|
||
THERMAL
|
||
--------------------------------------------------------------------------------
|
||
{% for z in node_health_facts.thermal.zones %}
|
||
zone {{ '%-18s' | format(z.type | default(z.zone)) }} {{ (z.temp_c | string ~ 'C') if z.temp_c is not none else 'unreadable (sensor returned no data)' }}{% if z.critical_c %} (critical {{ z.critical_c }}C){% endif %}
|
||
|
||
{% endfor %}
|
||
{% for chip in node_health_facts.thermal.hwmon %}
|
||
{% for t in chip.temps %}
|
||
{{ '%-16s' | format(chip.name | default('hwmon')) }} {{ '%-22s' | format(t.label) }} {{ (t.temp_c | string ~ 'C') if t.temp_c is not none else 'unreadable (sensor returned no data)' }}{% if t.crit_c %} (crit {{ t.crit_c }}C){% endif %}
|
||
|
||
{% endfor %}
|
||
{% for f in chip.fans %}
|
||
{{ '%-16s' | format(chip.name | default('hwmon')) }} {{ '%-22s' | format(f.label) }} {{ f.rpm }} rpm
|
||
{% endfor %}
|
||
{% endfor %}
|
||
|
||
--------------------------------------------------------------------------------
|
||
PRESSURE STALL (/proc/pressure — % of wall time stalled)
|
||
--------------------------------------------------------------------------------
|
||
{% for res in ['cpu', 'memory', 'io'] %}
|
||
{% set p = node_health_facts.pressure[res] %}
|
||
{% if p %}
|
||
{{ '%-7s' | format(res) }} some avg10={{ p.some.avg10 }} avg60={{ p.some.avg60 }} avg300={{ p.some.avg300 }}{% if p.full is defined %} | full avg60={{ p.full.avg60 }}{% endif %}
|
||
|
||
{% else %}
|
||
{{ '%-7s' | format(res) }} unavailable
|
||
{% endif %}
|
||
{% endfor %}
|
||
|
||
--------------------------------------------------------------------------------
|
||
NETWORK
|
||
--------------------------------------------------------------------------------
|
||
{% for n in node_health_facts.network.interfaces %}
|
||
{{ '%-10s' | format(n.name) }} {{ '%-6s' | format(n.operstate) }} {{ '%-8s' | format((n.speed_mbps | string ~ 'Mb/s') if n.speed_mbps else 'no-link') }} {{ '%-9s' | format(n.duplex | default('-')) }} mtu {{ n.mtu }} driver {{ n.driver | default('-') }}
|
||
rx {{ n.statistics.rx_packets | default('?') }} pkts / errors {{ n.statistics.rx_errors | default('?') }} / dropped {{ n.statistics.rx_dropped | default('?') }}
|
||
tx {{ n.statistics.tx_packets | default('?') }} pkts / errors {{ n.statistics.tx_errors | default('?') }} / dropped {{ n.statistics.tx_dropped | default('?') }}
|
||
{% endfor %}
|
||
{% if node_health_facts.network.nameservers is none %}
|
||
nameservers NOT READ — {{ node_health_facts.network.resolver_unavailable_reason | default('the collector recorded no reason', true) | regex_replace('\s+', ' ') | trim | wordwrap(66) | indent(13) }}
|
||
{% else %}
|
||
nameservers {{ node_health_facts.network.nameservers | join(', ') | default('none configured', true) }}
|
||
{% if node_health_facts.network.resolver_stub | default([], true) | length > 0 %}
|
||
(/etc/resolv.conf holds only the systemd-resolved stub {{ node_health_facts.network.resolver_stub | join(', ') }}; the servers above are the real upstreams behind it)
|
||
{% endif %}
|
||
{% endif %}
|
||
|
||
--------------------------------------------------------------------------------
|
||
NOT READ (and why) — these are UNKNOWNS, not passes
|
||
--------------------------------------------------------------------------------
|
||
{% for c in node_health_checks if c.status == 'SKIP' %}
|
||
- {{ c.id }}: {{ c.detail | regex_replace('\s+', ' ') | trim | wordwrap(70) | indent(3) }}
|
||
{% else %}
|
||
(nothing skipped)
|
||
{% endfor %}
|
||
{% if node_health_facts.unreadable %}
|
||
|
||
Unreadable paths reported by the collector:
|
||
{% for u in node_health_facts.unreadable %}
|
||
- {{ u.path }} — {{ u.reason }}
|
||
{% endfor %}
|
||
{% endif %}
|
||
|
||
tools present: {% for t, p in node_health_facts.tools.items() %}{{ t }}={{ p | default('MISSING', true) }}{{ ', ' if not loop.last }}{% endfor %}
|
||
|
||
================================================================================
|