fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03)

Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.

The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.

MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
  * systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
    symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
    loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
    FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
    silently bypass the local zone and ad-blocking.
  * resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
    lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
    would destroy PiHole-first ordering.
  * contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
    and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.

SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.

PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.

tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.

ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.

Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.

Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-01 00:11:49 -05:00
1 parent 14be9b68f6
commit c1f24e4a6d
10 files changed
+505 -23

No files matched your search

@@ -56,6 +56,9 @@ node_health_thresholds:
nic_min_speed_mbps: 1000
# A single upstream resolver is a known SPOF on this project — it broke containerd
# image pulls and CoreDNS forwarding on dev (DECISIONS-LOG 2026-08-31, finding 5).
# Counted against the node's REAL upstreams: where systemd-resolved is in use the
# collector looks behind the 127.0.0.53 stub rather than counting it as one server.
# The fix side of this is the `dns`-tagged block in roles/node_baseline.
min_nameservers: 2
# SMART (only evaluated when smartctl actually ran)
smart_reallocated_warn: 1
@@ -4,8 +4,10 @@
#
# Run by roles/node_health via ansible.builtin.script. Prints ONE JSON document on
# stdout and nothing else. It is strictly READ-ONLY: it opens files under /sys, /proc
# and /etc, and shells out only to `lsblk` (a read-only query). It never writes to the
# node, never installs anything, and never needs root.
# and /etc, and shells out only to read-only queries (`lsblk`, `lscpu`, and `resolvectl dns`
# — the last only when /etc/resolv.conf turns out to be the systemd-resolved stub and the
# real upstreams have to be asked of resolved). It never writes to the node, never installs
# anything, and never needs root.
#
# WHY A COLLECTOR SCRIPT AND NOT 40 ANSIBLE TASKS
# Everything here is "read a sysfs file that may or may not exist and may or may not
@@ -563,9 +565,113 @@ def collect_network():
# Resolver config is a known SPOF class on this project (a single upstream broke
# containerd pulls and CoreDNS forwarding on dev). Cheap to read here, so the health
# report can flag it on prod too.
resolv = read_text("/etc/resolv.conf") or ""
nameservers = re.findall(r"^\s*nameserver\s+(\S+)", resolv, re.M)
return {"interfaces": interfaces, "nameservers": nameservers}
resolvers = collect_resolvers()
return {
"interfaces": interfaces,
# List of REAL upstreams, or None when they could not be determined. NEVER the
# loopback stub, and never [] standing in for "unknown".
"nameservers": resolvers["nameservers"],
"resolver_source": resolvers["source"],
"resolver_stub": resolvers["stub"],
"resolver_unavailable_reason": resolvers["unavailable_reason"],
}
# systemd-resolved's loopback listeners. 127.0.0.53 is the stub (the address written into
# /etc/resolv.conf); 127.0.0.54 is the "bypass"/proxy listener. Neither is an UPSTREAM —
# seeing one in /etc/resolv.conf means the real servers live inside resolved.
RESOLVED_STUB_IPS = ("127.0.0.53", "127.0.0.54")
def _resolved_upstreams():
"""Real upstreams from systemd-resolved. Returns (servers, reason_unreadable).
`resolvectl dns` prints one line per scope ("Global:", "Link 2 (eth0): 1.2.3.4 ..."),
which is far easier to parse reliably than the wrapped columns of `resolvectl status`.
"""
rc, out, err = run(["resolvectl", "dns", "--no-pager"])
if rc is None:
return None, (
"/etc/resolv.conf holds only the systemd-resolved loopback stub, and the real "
"upstreams behind it could not be read: resolvectl is not installed on this node"
)
if rc != 0:
detail = (err.strip() or out.strip() or "no output").splitlines()[0]
return None, (
"/etc/resolv.conf holds only the systemd-resolved loopback stub, and "
"`resolvectl dns` exited %s so the real upstreams could not be read: %s"
% (rc, detail)
)
servers = []
for line in out.splitlines():
# Split on the FIRST colon only: everything after it is the server list, and IPv6
# addresses contain colons of their own.
_, sep, rest = line.partition(":")
if not sep:
continue
for token in rest.split():
if token not in servers:
servers.append(token)
if not servers:
return None, (
"/etc/resolv.conf holds only the systemd-resolved loopback stub and "
"`resolvectl dns` reported NO upstream servers on any scope — resolved is "
"running with an empty server list, so this node has no usable upstream"
)
return servers, None
def collect_resolvers():
"""The node's real DNS upstreams, plus how they were determined.
/etc/resolv.conf is NOT always the answer. Where systemd-resolved is in use (dev's
nodes today) the file contains only resolved's loopback stub, 127.0.0.53 — reading it
literally reports "1 nameserver" forever, which is a permanent FALSE single-upstream
WARN that no amount of correct configuration can clear. When the stub is detected the
real list is asked of resolved instead.
HONESTY DISCIPLINE, unchanged from the rest of this collector: if the stub is present
and the real list cannot be read, "nameservers" is None with the reason recorded — NOT
the stub address, NOT an empty list, and NOT a guess. An unknown must stay an unknown;
the assess step turns it into a SKIP, never a PASS or a WARN.
LIMIT worth knowing: only systemd-resolved's stub is special-cased. Some OTHER local
forwarder on 127.0.0.1 (dnsmasq, unbound) is reported exactly as it appears in
resolv.conf, because nothing generic can enumerate an arbitrary forwarder's upstreams.
"""
path = "/etc/resolv.conf"
# optional=False: a MISSING /etc/resolv.conf is itself a finding worth naming.
resolv = read_text(path, optional=False)
if resolv is None:
return {
"nameservers": None,
"source": None,
"stub": [],
"unavailable_reason": "/etc/resolv.conf could not be read (see the unreadable list)",
}
listed = re.findall(r"^\s*nameserver\s+(\S+)", resolv, re.M)
stub = [n for n in listed if n in RESOLVED_STUB_IPS]
# Only when EVERY listed server is a stub is the file a pure indirection. A mixed file
# (stub plus a real address) is reported as written — that is what the resolver library
# will actually use.
if listed and len(stub) == len(listed):
servers, reason = _resolved_upstreams()
if servers is None:
UNREADABLE.append({"path": "resolvectl dns", "reason": reason})
return {
"nameservers": None,
"source": None,
"stub": listed,
"unavailable_reason": reason,
}
return {
"nameservers": servers,
"source": "resolvectl dns (/etc/resolv.conf holds only the systemd-resolved "
"stub %s)" % ", ".join(listed),
"stub": listed,
"unavailable_reason": None,
}
return {"nameservers": listed, "source": path, "stub": stub, "unavailable_reason": None}
# --- OS / kernel / uptime -----------------------------------------------------------
@@ -435,32 +435,50 @@
{{ err_total | string ~ ' error(s), ' ~ drop_total | string ~ ' drop(s) across '
~ (up | map(attribute='name') | join(', ')) }}
# The collector reports the node's REAL upstreams: where /etc/resolv.conf holds only
# systemd-resolved's 127.0.0.53 stub (dev's shape) it asks resolved for the servers behind
# it, because counting the stub would be a permanent FALSE single-upstream WARN that no
# correct configuration could ever clear. When that lookup fails it reports null WITH a
# reason rather than a number — so the check SKIPs. A skip is honest; a WARN derived from
# an unknown is a fabricated verdict, and a PASS would be worse.
- name: Assess — DNS resolver redundancy
ansible.builtin.set_fact:
node_health_checks: "{{ node_health_checks + [{
'id': 'network.resolvers',
'title': 'Configured DNS upstreams (single upstream is a known SPOF here)',
'status': ('PASS' if (ns4 | length) >= (node_health_thresholds.min_nameservers | int) else 'WARN'),
'status': ('SKIP' if node_health_facts.network.nameservers is none
else ('PASS' if (ns4 | length) >= (node_health_thresholds.min_nameservers | int)
else 'WARN')),
'detail': ns_detail }] }}"
vars:
ansible_become: false
ns: "{{ node_health_facts.network.nameservers }}"
# `| default([], true)` only ever fires on the null (unreadable) path, where the
# counts below are unused because the status is already SKIP.
ns: "{{ node_health_facts.network.nameservers | default([], true) }}"
# Counted PER FAMILY on purpose. The outage this check exists for was a single IPv4
# upstream; IPv6 upstreams do not cover for it unless the node actually has working
# IPv6 egress, which this role cannot prove from sysfs alone. So the verdict keys off
# the IPv4 count and the detail shows the full picture.
ns4: "{{ ns | select('search', '^[0-9.]+$') | list }}"
ns6: "{{ ns | reject('search', '^[0-9.]+$') | list }}"
# Where the numbers came from — resolv.conf directly, or resolved behind the stub.
ns_src: "{{ node_health_facts.network.resolver_source | default('', true) }}"
ns_detail: >-
{{ (ns | length) | string ~ ' nameserver(s) — IPv4: '
{{ ('not read — ' ~ (node_health_facts.network.resolver_unavailable_reason
| default('the collector recorded no reason', true))
~ '. The upstream count is UNKNOWN here — not zero, and not one.')
if node_health_facts.network.nameservers is none
else ((ns | length) | string ~ ' nameserver(s) — IPv4: '
~ (ns4 | join(', ') | default('none', true))
~ '; IPv6: ' ~ (ns6 | join(', ') | default('none', true))
~ (' (read via ' ~ ns_src ~ ')' if ns_src else '')
~ ('. Only ' ~ (ns4 | length) | string ~ ' IPv4 upstream: a single IPv4 upstream
already broke containerd image pulls and CoreDNS forwarding on dev, and the
fix is stranded on fix/node-dns-resilient-upstream (DECISIONS-LOG
2026-08-31, finding 5). IPv6 upstreams only compensate if the node has
working IPv6 egress, which this check cannot verify.'
if (ns4 | length) < (node_health_thresholds.min_nameservers | int) else '') }}
already broke containerd image pulls and CoreDNS forwarding on dev. The fix
is the `dns`-tagged block in roles/node_baseline — set this environment
node_baseline_dns_servers (role defaults for dev, group_vars/k3s_prod for
prod) and re-run node-baseline.yml --tags dns. IPv6 upstreams only compensate
if the node has working IPv6 egress, which this check cannot verify.'
if (ns4 | length) < (node_health_thresholds.min_nameservers | int) else '')) }}
# --- OS / firmware ------------------------------------------------------------------
- name: Assess — pending reboot
@@ -124,7 +124,14 @@
rx {{ n.statistics.rx_packets | default('?') }} pkts / errors {{ n.statistics.rx_errors | default('?') }} / dropped {{ n.statistics.rx_dropped | default('?') }}
tx {{ n.statistics.tx_packets | default('?') }} pkts / errors {{ n.statistics.tx_errors | default('?') }} / dropped {{ n.statistics.tx_dropped | default('?') }}
{% endfor %}
{% if node_health_facts.network.nameservers is none %}
nameservers NOT READ — {{ node_health_facts.network.resolver_unavailable_reason | default('the collector recorded no reason', true) | regex_replace('\s+', ' ') | trim | wordwrap(66) | indent(13) }}
{% else %}
nameservers {{ node_health_facts.network.nameservers | join(', ') | default('none configured', true) }}
{% if node_health_facts.network.resolver_stub | default([], true) | length > 0 %}
(/etc/resolv.conf holds only the systemd-resolved stub {{ node_health_facts.network.resolver_stub | join(', ') }}; the servers above are the real upstreams behind it)
{% endif %}
{% endif %}
--------------------------------------------------------------------------------
NOT READ (and why) — these are UNKNOWNS, not passes