fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03)
Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.
The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.
MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
* systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
silently bypass the local zone and ad-blocking.
* resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
would destroy PiHole-first ordering.
* contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.
SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.
PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.
tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.
ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.
Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
14be9b68f6
commit
c1f24e4a6d
10 files changed
+505
-23
No files matched your search
@@ -56,6 +56,9 @@ node_health_thresholds:
|
||||
nic_min_speed_mbps: 1000
|
||||
# A single upstream resolver is a known SPOF on this project — it broke containerd
|
||||
# image pulls and CoreDNS forwarding on dev (DECISIONS-LOG 2026-08-31, finding 5).
|
||||
# Counted against the node's REAL upstreams: where systemd-resolved is in use the
|
||||
# collector looks behind the 127.0.0.53 stub rather than counting it as one server.
|
||||
# The fix side of this is the `dns`-tagged block in roles/node_baseline.
|
||||
min_nameservers: 2
|
||||
# SMART (only evaluated when smartctl actually ran)
|
||||
smart_reallocated_warn: 1
|
||||
|
||||
@@ -4,8 +4,10 @@
|
||||
#
|
||||
# Run by roles/node_health via ansible.builtin.script. Prints ONE JSON document on
|
||||
# stdout and nothing else. It is strictly READ-ONLY: it opens files under /sys, /proc
|
||||
# and /etc, and shells out only to `lsblk` (a read-only query). It never writes to the
|
||||
# node, never installs anything, and never needs root.
|
||||
# and /etc, and shells out only to read-only queries (`lsblk`, `lscpu`, and `resolvectl dns`
|
||||
# — the last only when /etc/resolv.conf turns out to be the systemd-resolved stub and the
|
||||
# real upstreams have to be asked of resolved). It never writes to the node, never installs
|
||||
# anything, and never needs root.
|
||||
#
|
||||
# WHY A COLLECTOR SCRIPT AND NOT 40 ANSIBLE TASKS
|
||||
# Everything here is "read a sysfs file that may or may not exist and may or may not
|
||||
@@ -563,9 +565,113 @@ def collect_network():
|
||||
# Resolver config is a known SPOF class on this project (a single upstream broke
|
||||
# containerd pulls and CoreDNS forwarding on dev). Cheap to read here, so the health
|
||||
# report can flag it on prod too.
|
||||
resolv = read_text("/etc/resolv.conf") or ""
|
||||
nameservers = re.findall(r"^\s*nameserver\s+(\S+)", resolv, re.M)
|
||||
return {"interfaces": interfaces, "nameservers": nameservers}
|
||||
resolvers = collect_resolvers()
|
||||
return {
|
||||
"interfaces": interfaces,
|
||||
# List of REAL upstreams, or None when they could not be determined. NEVER the
|
||||
# loopback stub, and never [] standing in for "unknown".
|
||||
"nameservers": resolvers["nameservers"],
|
||||
"resolver_source": resolvers["source"],
|
||||
"resolver_stub": resolvers["stub"],
|
||||
"resolver_unavailable_reason": resolvers["unavailable_reason"],
|
||||
}
|
||||
|
||||
|
||||
# systemd-resolved's loopback listeners. 127.0.0.53 is the stub (the address written into
|
||||
# /etc/resolv.conf); 127.0.0.54 is the "bypass"/proxy listener. Neither is an UPSTREAM —
|
||||
# seeing one in /etc/resolv.conf means the real servers live inside resolved.
|
||||
RESOLVED_STUB_IPS = ("127.0.0.53", "127.0.0.54")
|
||||
|
||||
|
||||
def _resolved_upstreams():
|
||||
"""Real upstreams from systemd-resolved. Returns (servers, reason_unreadable).
|
||||
|
||||
`resolvectl dns` prints one line per scope ("Global:", "Link 2 (eth0): 1.2.3.4 ..."),
|
||||
which is far easier to parse reliably than the wrapped columns of `resolvectl status`.
|
||||
"""
|
||||
rc, out, err = run(["resolvectl", "dns", "--no-pager"])
|
||||
if rc is None:
|
||||
return None, (
|
||||
"/etc/resolv.conf holds only the systemd-resolved loopback stub, and the real "
|
||||
"upstreams behind it could not be read: resolvectl is not installed on this node"
|
||||
)
|
||||
if rc != 0:
|
||||
detail = (err.strip() or out.strip() or "no output").splitlines()[0]
|
||||
return None, (
|
||||
"/etc/resolv.conf holds only the systemd-resolved loopback stub, and "
|
||||
"`resolvectl dns` exited %s so the real upstreams could not be read: %s"
|
||||
% (rc, detail)
|
||||
)
|
||||
servers = []
|
||||
for line in out.splitlines():
|
||||
# Split on the FIRST colon only: everything after it is the server list, and IPv6
|
||||
# addresses contain colons of their own.
|
||||
_, sep, rest = line.partition(":")
|
||||
if not sep:
|
||||
continue
|
||||
for token in rest.split():
|
||||
if token not in servers:
|
||||
servers.append(token)
|
||||
if not servers:
|
||||
return None, (
|
||||
"/etc/resolv.conf holds only the systemd-resolved loopback stub and "
|
||||
"`resolvectl dns` reported NO upstream servers on any scope — resolved is "
|
||||
"running with an empty server list, so this node has no usable upstream"
|
||||
)
|
||||
return servers, None
|
||||
|
||||
|
||||
def collect_resolvers():
|
||||
"""The node's real DNS upstreams, plus how they were determined.
|
||||
|
||||
/etc/resolv.conf is NOT always the answer. Where systemd-resolved is in use (dev's
|
||||
nodes today) the file contains only resolved's loopback stub, 127.0.0.53 — reading it
|
||||
literally reports "1 nameserver" forever, which is a permanent FALSE single-upstream
|
||||
WARN that no amount of correct configuration can clear. When the stub is detected the
|
||||
real list is asked of resolved instead.
|
||||
|
||||
HONESTY DISCIPLINE, unchanged from the rest of this collector: if the stub is present
|
||||
and the real list cannot be read, "nameservers" is None with the reason recorded — NOT
|
||||
the stub address, NOT an empty list, and NOT a guess. An unknown must stay an unknown;
|
||||
the assess step turns it into a SKIP, never a PASS or a WARN.
|
||||
|
||||
LIMIT worth knowing: only systemd-resolved's stub is special-cased. Some OTHER local
|
||||
forwarder on 127.0.0.1 (dnsmasq, unbound) is reported exactly as it appears in
|
||||
resolv.conf, because nothing generic can enumerate an arbitrary forwarder's upstreams.
|
||||
"""
|
||||
path = "/etc/resolv.conf"
|
||||
# optional=False: a MISSING /etc/resolv.conf is itself a finding worth naming.
|
||||
resolv = read_text(path, optional=False)
|
||||
if resolv is None:
|
||||
return {
|
||||
"nameservers": None,
|
||||
"source": None,
|
||||
"stub": [],
|
||||
"unavailable_reason": "/etc/resolv.conf could not be read (see the unreadable list)",
|
||||
}
|
||||
listed = re.findall(r"^\s*nameserver\s+(\S+)", resolv, re.M)
|
||||
stub = [n for n in listed if n in RESOLVED_STUB_IPS]
|
||||
# Only when EVERY listed server is a stub is the file a pure indirection. A mixed file
|
||||
# (stub plus a real address) is reported as written — that is what the resolver library
|
||||
# will actually use.
|
||||
if listed and len(stub) == len(listed):
|
||||
servers, reason = _resolved_upstreams()
|
||||
if servers is None:
|
||||
UNREADABLE.append({"path": "resolvectl dns", "reason": reason})
|
||||
return {
|
||||
"nameservers": None,
|
||||
"source": None,
|
||||
"stub": listed,
|
||||
"unavailable_reason": reason,
|
||||
}
|
||||
return {
|
||||
"nameservers": servers,
|
||||
"source": "resolvectl dns (/etc/resolv.conf holds only the systemd-resolved "
|
||||
"stub %s)" % ", ".join(listed),
|
||||
"stub": listed,
|
||||
"unavailable_reason": None,
|
||||
}
|
||||
return {"nameservers": listed, "source": path, "stub": stub, "unavailable_reason": None}
|
||||
|
||||
|
||||
# --- OS / kernel / uptime -----------------------------------------------------------
|
||||
|
||||
@@ -435,32 +435,50 @@
|
||||
{{ err_total | string ~ ' error(s), ' ~ drop_total | string ~ ' drop(s) across '
|
||||
~ (up | map(attribute='name') | join(', ')) }}
|
||||
|
||||
# The collector reports the node's REAL upstreams: where /etc/resolv.conf holds only
|
||||
# systemd-resolved's 127.0.0.53 stub (dev's shape) it asks resolved for the servers behind
|
||||
# it, because counting the stub would be a permanent FALSE single-upstream WARN that no
|
||||
# correct configuration could ever clear. When that lookup fails it reports null WITH a
|
||||
# reason rather than a number — so the check SKIPs. A skip is honest; a WARN derived from
|
||||
# an unknown is a fabricated verdict, and a PASS would be worse.
|
||||
- name: Assess — DNS resolver redundancy
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
'id': 'network.resolvers',
|
||||
'title': 'Configured DNS upstreams (single upstream is a known SPOF here)',
|
||||
'status': ('PASS' if (ns4 | length) >= (node_health_thresholds.min_nameservers | int) else 'WARN'),
|
||||
'status': ('SKIP' if node_health_facts.network.nameservers is none
|
||||
else ('PASS' if (ns4 | length) >= (node_health_thresholds.min_nameservers | int)
|
||||
else 'WARN')),
|
||||
'detail': ns_detail }] }}"
|
||||
vars:
|
||||
ansible_become: false
|
||||
ns: "{{ node_health_facts.network.nameservers }}"
|
||||
# `| default([], true)` only ever fires on the null (unreadable) path, where the
|
||||
# counts below are unused because the status is already SKIP.
|
||||
ns: "{{ node_health_facts.network.nameservers | default([], true) }}"
|
||||
# Counted PER FAMILY on purpose. The outage this check exists for was a single IPv4
|
||||
# upstream; IPv6 upstreams do not cover for it unless the node actually has working
|
||||
# IPv6 egress, which this role cannot prove from sysfs alone. So the verdict keys off
|
||||
# the IPv4 count and the detail shows the full picture.
|
||||
ns4: "{{ ns | select('search', '^[0-9.]+$') | list }}"
|
||||
ns6: "{{ ns | reject('search', '^[0-9.]+$') | list }}"
|
||||
# Where the numbers came from — resolv.conf directly, or resolved behind the stub.
|
||||
ns_src: "{{ node_health_facts.network.resolver_source | default('', true) }}"
|
||||
ns_detail: >-
|
||||
{{ (ns | length) | string ~ ' nameserver(s) — IPv4: '
|
||||
{{ ('not read — ' ~ (node_health_facts.network.resolver_unavailable_reason
|
||||
| default('the collector recorded no reason', true))
|
||||
~ '. The upstream count is UNKNOWN here — not zero, and not one.')
|
||||
if node_health_facts.network.nameservers is none
|
||||
else ((ns | length) | string ~ ' nameserver(s) — IPv4: '
|
||||
~ (ns4 | join(', ') | default('none', true))
|
||||
~ '; IPv6: ' ~ (ns6 | join(', ') | default('none', true))
|
||||
~ (' (read via ' ~ ns_src ~ ')' if ns_src else '')
|
||||
~ ('. Only ' ~ (ns4 | length) | string ~ ' IPv4 upstream: a single IPv4 upstream
|
||||
already broke containerd image pulls and CoreDNS forwarding on dev, and the
|
||||
fix is stranded on fix/node-dns-resilient-upstream (DECISIONS-LOG
|
||||
2026-08-31, finding 5). IPv6 upstreams only compensate if the node has
|
||||
working IPv6 egress, which this check cannot verify.'
|
||||
if (ns4 | length) < (node_health_thresholds.min_nameservers | int) else '') }}
|
||||
already broke containerd image pulls and CoreDNS forwarding on dev. The fix
|
||||
is the `dns`-tagged block in roles/node_baseline — set this environment
|
||||
node_baseline_dns_servers (role defaults for dev, group_vars/k3s_prod for
|
||||
prod) and re-run node-baseline.yml --tags dns. IPv6 upstreams only compensate
|
||||
if the node has working IPv6 egress, which this check cannot verify.'
|
||||
if (ns4 | length) < (node_health_thresholds.min_nameservers | int) else '')) }}
|
||||
|
||||
# --- OS / firmware ------------------------------------------------------------------
|
||||
- name: Assess — pending reboot
|
||||
|
||||
@@ -124,7 +124,14 @@
|
||||
rx {{ n.statistics.rx_packets | default('?') }} pkts / errors {{ n.statistics.rx_errors | default('?') }} / dropped {{ n.statistics.rx_dropped | default('?') }}
|
||||
tx {{ n.statistics.tx_packets | default('?') }} pkts / errors {{ n.statistics.tx_errors | default('?') }} / dropped {{ n.statistics.tx_dropped | default('?') }}
|
||||
{% endfor %}
|
||||
{% if node_health_facts.network.nameservers is none %}
|
||||
nameservers NOT READ — {{ node_health_facts.network.resolver_unavailable_reason | default('the collector recorded no reason', true) | regex_replace('\s+', ' ') | trim | wordwrap(66) | indent(13) }}
|
||||
{% else %}
|
||||
nameservers {{ node_health_facts.network.nameservers | join(', ') | default('none configured', true) }}
|
||||
{% if node_health_facts.network.resolver_stub | default([], true) | length > 0 %}
|
||||
(/etc/resolv.conf holds only the systemd-resolved stub {{ node_health_facts.network.resolver_stub | join(', ') }}; the servers above are the real upstreams behind it)
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
NOT READ (and why) — these are UNKNOWNS, not passes
|
||||
|
||||
Reference in new issue
Block a user