fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03)

Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.

The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.

MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
  * systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
    symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
    loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
    FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
    silently bypass the local zone and ad-blocking.
  * resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
    lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
    would destroy PiHole-first ordering.
  * contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
    and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.

SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.

PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.

tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.

ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.

Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.

Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-01 00:11:49 -05:00
1 parent 14be9b68f6
commit c1f24e4a6d
10 files changed
+505 -23

No files matched your search

@@ -435,32 +435,50 @@
{{ err_total | string ~ ' error(s), ' ~ drop_total | string ~ ' drop(s) across '
~ (up | map(attribute='name') | join(', ')) }}
# The collector reports the node's REAL upstreams: where /etc/resolv.conf holds only
# systemd-resolved's 127.0.0.53 stub (dev's shape) it asks resolved for the servers behind
# it, because counting the stub would be a permanent FALSE single-upstream WARN that no
# correct configuration could ever clear. When that lookup fails it reports null WITH a
# reason rather than a number — so the check SKIPs. A skip is honest; a WARN derived from
# an unknown is a fabricated verdict, and a PASS would be worse.
- name: Assess — DNS resolver redundancy
ansible.builtin.set_fact:
node_health_checks: "{{ node_health_checks + [{
'id': 'network.resolvers',
'title': 'Configured DNS upstreams (single upstream is a known SPOF here)',
'status': ('PASS' if (ns4 | length) >= (node_health_thresholds.min_nameservers | int) else 'WARN'),
'status': ('SKIP' if node_health_facts.network.nameservers is none
else ('PASS' if (ns4 | length) >= (node_health_thresholds.min_nameservers | int)
else 'WARN')),
'detail': ns_detail }] }}"
vars:
ansible_become: false
ns: "{{ node_health_facts.network.nameservers }}"
# `| default([], true)` only ever fires on the null (unreadable) path, where the
# counts below are unused because the status is already SKIP.
ns: "{{ node_health_facts.network.nameservers | default([], true) }}"
# Counted PER FAMILY on purpose. The outage this check exists for was a single IPv4
# upstream; IPv6 upstreams do not cover for it unless the node actually has working
# IPv6 egress, which this role cannot prove from sysfs alone. So the verdict keys off
# the IPv4 count and the detail shows the full picture.
ns4: "{{ ns | select('search', '^[0-9.]+$') | list }}"
ns6: "{{ ns | reject('search', '^[0-9.]+$') | list }}"
# Where the numbers came from — resolv.conf directly, or resolved behind the stub.
ns_src: "{{ node_health_facts.network.resolver_source | default('', true) }}"
ns_detail: >-
{{ (ns | length) | string ~ ' nameserver(s) — IPv4: '
{{ ('not read — ' ~ (node_health_facts.network.resolver_unavailable_reason
| default('the collector recorded no reason', true))
~ '. The upstream count is UNKNOWN here — not zero, and not one.')
if node_health_facts.network.nameservers is none
else ((ns | length) | string ~ ' nameserver(s) — IPv4: '
~ (ns4 | join(', ') | default('none', true))
~ '; IPv6: ' ~ (ns6 | join(', ') | default('none', true))
~ (' (read via ' ~ ns_src ~ ')' if ns_src else '')
~ ('. Only ' ~ (ns4 | length) | string ~ ' IPv4 upstream: a single IPv4 upstream
already broke containerd image pulls and CoreDNS forwarding on dev, and the
fix is stranded on fix/node-dns-resilient-upstream (DECISIONS-LOG
2026-08-31, finding 5). IPv6 upstreams only compensate if the node has
working IPv6 egress, which this check cannot verify.'
if (ns4 | length) < (node_health_thresholds.min_nameservers | int) else '') }}
already broke containerd image pulls and CoreDNS forwarding on dev. The fix
is the `dns`-tagged block in roles/node_baseline — set this environment
node_baseline_dns_servers (role defaults for dev, group_vars/k3s_prod for
prod) and re-run node-baseline.yml --tags dns. IPv6 upstreams only compensate
if the node has working IPv6 egress, which this check cannot verify.'
if (ns4 | length) < (node_health_thresholds.min_nameservers | int) else '')) }}
# --- OS / firmware ------------------------------------------------------------------
- name: Assess — pending reboot