jakman85andClaude Opus 5
c1f24e4a6d
Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.
The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.
MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
* systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
silently bypass the local zone and ad-blocking.
* resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
would destroy PiHole-first ordering.
* contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.
SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.
PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.
tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.
ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.
Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
82 lines
4.4 KiB
YAML
82 lines
4.4 KiB
YAML
---
|
|
# node_health — READ-ONLY hardware/health inspection of a node (bare metal or VM).
|
|
#
|
|
# Sibling of roles/cluster_verify: cluster_verify answers "is the CLUSTER healthy",
|
|
# node_health answers "is the HARDWARE under it healthy". Same posture — it inspects and
|
|
# reports, it never changes the node. Deliberately NOT part of node_baseline: baseline
|
|
# installs and configures, health only looks. That separation is why this role must never
|
|
# `apt install` anything (smartmontools is declared in node_baseline_packages; if it is
|
|
# missing here the SMART checks SKIP with a reason, they do not self-heal).
|
|
#
|
|
# Serves R-ENV-01 (reproducible node provisioning — you cannot certify a node you have not
|
|
# measured), R-ENV-03 (bare-metal parity) and R-OBS-04 (node-level observability).
|
|
|
|
# --- Reporting ---------------------------------------------------------------------
|
|
# Artifacts land under infra/ansible/_artifacts/node-health/, mirroring the repo's
|
|
# conformance/_artifacts/ convention: generated output, regenerated on every run, listed
|
|
# in .gitignore, never committed (it also carries drive serial numbers).
|
|
node_health_artifact_dir: "{{ playbook_dir }}/../_artifacts/node-health"
|
|
node_health_write_artifacts: true
|
|
# Print the human summary to the play output as well as writing it to the artifact dir.
|
|
node_health_show_summary: true
|
|
# Report-only by default: a health check that aborts the play hides the other 20 findings.
|
|
# Set true (e.g. in CI or a pre-build gate) to make any FAIL check end the play non-zero.
|
|
node_health_fail_on_fail: false
|
|
# Same, one step stricter: also fail on WARN.
|
|
node_health_fail_on_warn: false
|
|
|
|
# --- Root-only checks ---------------------------------------------------------------
|
|
# Everything this role can read WITHOUT root, it reads without root. Only SMART needs
|
|
# escalation (the ATA/NVMe passthrough ioctls are root-only). The role PROBES become
|
|
# first and, if escalation is unavailable (e.g. prod's vaulted become password is not
|
|
# present — infra/PROD-PROVISIONING.md §5), those checks report SKIPPED with the exact
|
|
# reason. Never a hard failure, and never a fake PASS: a skip is honest.
|
|
node_health_try_privileged: true
|
|
# --nocheck=standby: never spin up a sleeping disk just to read it. Read-only hygiene.
|
|
node_health_smartctl_args: "--json=c --all --nocheck=standby"
|
|
|
|
# --- Thresholds ---------------------------------------------------------------------
|
|
# All verdicts come from here, so a site can retune without touching task logic.
|
|
node_health_thresholds:
|
|
# Filesystems
|
|
fs_used_pct_warn: 80
|
|
fs_used_pct_fail: 90
|
|
inode_used_pct_warn: 80
|
|
# Memory (percentage of MemTotal still available)
|
|
mem_available_pct_warn: 15
|
|
mem_available_pct_fail: 5
|
|
# Temperatures (deg C, any sensor)
|
|
temp_warn_c: 70
|
|
temp_fail_c: 85
|
|
# Pressure-stall: "some" avg60, percent of wall time stalled. Dev learned to care about
|
|
# CPU PSI specifically (it shows saturation long before load average makes it obvious).
|
|
psi_some_avg60_warn: 10.0
|
|
psi_some_avg60_fail: 40.0
|
|
# Network
|
|
nic_min_speed_mbps: 1000
|
|
# A single upstream resolver is a known SPOF on this project — it broke containerd
|
|
# image pulls and CoreDNS forwarding on dev (DECISIONS-LOG 2026-08-31, finding 5).
|
|
# Counted against the node's REAL upstreams: where systemd-resolved is in use the
|
|
# collector looks behind the 127.0.0.53 stub rather than counting it as one server.
|
|
# The fix side of this is the `dns`-tagged block in roles/node_baseline.
|
|
min_nameservers: 2
|
|
# SMART (only evaluated when smartctl actually ran)
|
|
smart_reallocated_warn: 1
|
|
smart_reallocated_fail: 50
|
|
smart_pending_warn: 1
|
|
smart_pending_fail: 10
|
|
smart_lifetime_remaining_warn_pct: 20
|
|
smart_lifetime_remaining_fail_pct: 10
|
|
smart_temp_warn_c: 55
|
|
smart_temp_fail_c: 70
|
|
# Power-on hours past which a consumer SSD is simply old. Informational — age alone is
|
|
# not a failure, but it belongs in the operator's field of view when deciding whether a
|
|
# drive may hold production state (PROD-PROVISIONING.md §2).
|
|
smart_power_on_hours_notice: 40000
|
|
|
|
# ATA SMART attribute IDs surfaced verbatim in the report. Vendors disagree on names, so
|
|
# the report prints whatever the drive calls them and the checks key off the IDs.
|
|
node_health_smart_attribute_ids: [5, 9, 177, 179, 181, 182, 183, 184, 187, 188, 190,
|
|
194, 195, 196, 197, 198, 199, 202, 231, 232, 233,
|
|
241, 242, 246]
|