fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03)

Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.

The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.

MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
  * systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
    symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
    loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
    FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
    silently bypass the local zone and ad-blocking.
  * resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
    lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
    would destroy PiHole-first ordering.
  * contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
    and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.

SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.

PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.

tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.

ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.

Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.

Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-01 00:11:49 -05:00
1 parent 14be9b68f6
commit c1f24e4a6d
10 files changed
+505 -23

No files matched your search

@@ -4,8 +4,10 @@
#
# Run by roles/node_health via ansible.builtin.script. Prints ONE JSON document on
# stdout and nothing else. It is strictly READ-ONLY: it opens files under /sys, /proc
# and /etc, and shells out only to `lsblk` (a read-only query). It never writes to the
# node, never installs anything, and never needs root.
# and /etc, and shells out only to read-only queries (`lsblk`, `lscpu`, and `resolvectl dns`
# — the last only when /etc/resolv.conf turns out to be the systemd-resolved stub and the
# real upstreams have to be asked of resolved). It never writes to the node, never installs
# anything, and never needs root.
#
# WHY A COLLECTOR SCRIPT AND NOT 40 ANSIBLE TASKS
# Everything here is "read a sysfs file that may or may not exist and may or may not
@@ -563,9 +565,113 @@ def collect_network():
# Resolver config is a known SPOF class on this project (a single upstream broke
# containerd pulls and CoreDNS forwarding on dev). Cheap to read here, so the health
# report can flag it on prod too.
resolv = read_text("/etc/resolv.conf") or ""
nameservers = re.findall(r"^\s*nameserver\s+(\S+)", resolv, re.M)
return {"interfaces": interfaces, "nameservers": nameservers}
resolvers = collect_resolvers()
return {
"interfaces": interfaces,
# List of REAL upstreams, or None when they could not be determined. NEVER the
# loopback stub, and never [] standing in for "unknown".
"nameservers": resolvers["nameservers"],
"resolver_source": resolvers["source"],
"resolver_stub": resolvers["stub"],
"resolver_unavailable_reason": resolvers["unavailable_reason"],
}
# systemd-resolved's loopback listeners. 127.0.0.53 is the stub (the address written into
# /etc/resolv.conf); 127.0.0.54 is the "bypass"/proxy listener. Neither is an UPSTREAM —
# seeing one in /etc/resolv.conf means the real servers live inside resolved.
RESOLVED_STUB_IPS = ("127.0.0.53", "127.0.0.54")
def _resolved_upstreams():
"""Real upstreams from systemd-resolved. Returns (servers, reason_unreadable).
`resolvectl dns` prints one line per scope ("Global:", "Link 2 (eth0): 1.2.3.4 ..."),
which is far easier to parse reliably than the wrapped columns of `resolvectl status`.
"""
rc, out, err = run(["resolvectl", "dns", "--no-pager"])
if rc is None:
return None, (
"/etc/resolv.conf holds only the systemd-resolved loopback stub, and the real "
"upstreams behind it could not be read: resolvectl is not installed on this node"
)
if rc != 0:
detail = (err.strip() or out.strip() or "no output").splitlines()[0]
return None, (
"/etc/resolv.conf holds only the systemd-resolved loopback stub, and "
"`resolvectl dns` exited %s so the real upstreams could not be read: %s"
% (rc, detail)
)
servers = []
for line in out.splitlines():
# Split on the FIRST colon only: everything after it is the server list, and IPv6
# addresses contain colons of their own.
_, sep, rest = line.partition(":")
if not sep:
continue
for token in rest.split():
if token not in servers:
servers.append(token)
if not servers:
return None, (
"/etc/resolv.conf holds only the systemd-resolved loopback stub and "
"`resolvectl dns` reported NO upstream servers on any scope — resolved is "
"running with an empty server list, so this node has no usable upstream"
)
return servers, None
def collect_resolvers():
"""The node's real DNS upstreams, plus how they were determined.
/etc/resolv.conf is NOT always the answer. Where systemd-resolved is in use (dev's
nodes today) the file contains only resolved's loopback stub, 127.0.0.53 — reading it
literally reports "1 nameserver" forever, which is a permanent FALSE single-upstream
WARN that no amount of correct configuration can clear. When the stub is detected the
real list is asked of resolved instead.
HONESTY DISCIPLINE, unchanged from the rest of this collector: if the stub is present
and the real list cannot be read, "nameservers" is None with the reason recorded — NOT
the stub address, NOT an empty list, and NOT a guess. An unknown must stay an unknown;
the assess step turns it into a SKIP, never a PASS or a WARN.
LIMIT worth knowing: only systemd-resolved's stub is special-cased. Some OTHER local
forwarder on 127.0.0.1 (dnsmasq, unbound) is reported exactly as it appears in
resolv.conf, because nothing generic can enumerate an arbitrary forwarder's upstreams.
"""
path = "/etc/resolv.conf"
# optional=False: a MISSING /etc/resolv.conf is itself a finding worth naming.
resolv = read_text(path, optional=False)
if resolv is None:
return {
"nameservers": None,
"source": None,
"stub": [],
"unavailable_reason": "/etc/resolv.conf could not be read (see the unreadable list)",
}
listed = re.findall(r"^\s*nameserver\s+(\S+)", resolv, re.M)
stub = [n for n in listed if n in RESOLVED_STUB_IPS]
# Only when EVERY listed server is a stub is the file a pure indirection. A mixed file
# (stub plus a real address) is reported as written — that is what the resolver library
# will actually use.
if listed and len(stub) == len(listed):
servers, reason = _resolved_upstreams()
if servers is None:
UNREADABLE.append({"path": "resolvectl dns", "reason": reason})
return {
"nameservers": None,
"source": None,
"stub": listed,
"unavailable_reason": reason,
}
return {
"nameservers": servers,
"source": "resolvectl dns (/etc/resolv.conf holds only the systemd-resolved "
"stub %s)" % ", ".join(listed),
"stub": listed,
"unavailable_reason": None,
}
return {"nameservers": listed, "source": path, "stub": stub, "unavailable_reason": None}
# --- OS / kernel / uptime -----------------------------------------------------------