fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03)
Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.
The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.
MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
* systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
silently bypass the local zone and ad-blocking.
* resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
would destroy PiHole-first ordering.
* contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.
SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.
PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.
tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.
ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.
Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
14be9b68f6
commit
c1f24e4a6d
10 files changed
+505
-23
No files matched your search
@@ -34,3 +34,40 @@ node_baseline_longhorn_fs_label: longhorn
|
||||
node_baseline_etcd_device: /dev/vdc
|
||||
node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db
|
||||
node_baseline_etcd_fs_label: k3s-etcd
|
||||
|
||||
# --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
|
||||
# These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml
|
||||
# (the same seam kube_vip_address / k3s_api_host / cluster_kubeconfig_name use) — NOT in
|
||||
# clusters/base/cluster-vars.yaml, which is a Flux postBuild ConfigMap that no Ansible
|
||||
# code reads.
|
||||
#
|
||||
# WHY: the nodes shipped with ONE upstream resolver. PiHole answers the local jackson.lan
|
||||
# zone instantly but its external recursion flaps (measured 0/25 for registry-1.docker.io /
|
||||
# quay.io / ghcr.io while gitea.jackson.lan resolved 5/5), and one upstream is a hard SPOF:
|
||||
# it broke containerd image pulls ("lookup registry-1.docker.io: Try again") AND CoreDNS,
|
||||
# which forwards to /etc/resolv.conf and flooded its log with i/o timeouts to 192.168.1.18.
|
||||
#
|
||||
# ORDER IS THE POLICY. This is a RESILIENCE fix, not a bypass: the first entry stays PiHole,
|
||||
# so *.jackson.lan and network-wide ad-blocking keep working exactly as before, and the
|
||||
# second is only ever consulted when the first fails. Both write paths preserve that order
|
||||
# (systemd-resolved tries the servers of one scope in order; glibc walks `nameserver` lines
|
||||
# in order). Never put a public resolver first, and never enable `rotate` — either would
|
||||
# silently take the local zone and the ad-block policy out of the path.
|
||||
#
|
||||
# Keep in parity with infra/tofu (var.dns_server, var.network_gateway), which sets the same
|
||||
# two servers via cloud-init so a freshly rebuilt dev node boots with this list already.
|
||||
node_baseline_dns_servers:
|
||||
- 192.168.1.18 # PiHole — local jackson.lan zone + ad-blocking. PRIMARY, keep first.
|
||||
- 192.168.1.1 # pfSense gateway — reliable external recursion. FALLBACK only.
|
||||
# Search domain(s). An EMPTY list means "configure no search domain" and is a legitimate
|
||||
# value (prod uses it — its local domain is not a decided thing), not a missing one.
|
||||
node_baseline_dns_search:
|
||||
- jackson.lan
|
||||
# resolv.conf `options` for the DIRECT-file mechanism only (systemd-resolved does its own
|
||||
# per-server failover timing and ignores these). glibc's defaults are timeout:5 attempts:2,
|
||||
# i.e. up to 20s of stalled lookups before it reaches the second server — long enough for
|
||||
# containerd to give up on a pull. These shorten the failover, and deliberately do NOT
|
||||
# include `rotate`, which would round-robin the servers and break PiHole-first ordering.
|
||||
node_baseline_dns_options:
|
||||
- timeout:2
|
||||
- attempts:2
|
||||
@@ -2,3 +2,12 @@
|
||||
- name: Reload sysctl
|
||||
ansible.builtin.command: sysctl --system
|
||||
changed_when: true
|
||||
|
||||
# Restarting systemd-resolved re-reads /etc/systemd/resolved.conf.d/ and re-points the
|
||||
# 127.0.0.53 stub at the new upstream list. It does NOT touch link addressing or routes,
|
||||
# so the SSH/NAT session Ansible is running over is unaffected. Only fires when the
|
||||
# `dns`-tagged drop-in above actually changed, so a re-run of the role is a no-op.
|
||||
- name: Restart systemd-resolved
|
||||
ansible.builtin.systemd:
|
||||
name: systemd-resolved
|
||||
state: restarted
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
# TAGS — every task here carries one of `packages`, `system` or `disks`. An untagged run
|
||||
# is unchanged (all three run, in this order). The split exists because the PACKAGE floor
|
||||
# TAGS — every task here carries one of `packages`, `system`, `disks` or `dns`. An untagged
|
||||
# run is unchanged (all four run, in this order). The split exists because the PACKAGE floor
|
||||
# and the DATA-DISK work have different readiness: packages are safe on any node from
|
||||
# first contact, while the disk tasks need block devices that a new environment may not
|
||||
# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
|
||||
@@ -9,7 +9,9 @@
|
||||
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
|
||||
# Skipping is the guard: with `--tags packages` no `disks`-tagged task is selected, so a
|
||||
# BLOCKED-pending-disk-decision device path is never even templated, let alone handed to
|
||||
# mkfs. A FULL run against such a node stops at the assert below instead.
|
||||
# mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
|
||||
# at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
|
||||
# ansible-playbook playbooks/node-baseline.yml --tags dns
|
||||
|
||||
- name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
|
||||
ansible.builtin.apt:
|
||||
@@ -154,3 +156,225 @@
|
||||
regexp: '^([^#].*\sswap\s.*)$'
|
||||
replace: '# \1 # disabled for k3s/kubelet'
|
||||
tags: [system]
|
||||
|
||||
# =====================================================================================
|
||||
# DNS RESILIENCE — tag `dns` (R-DNS-01 / R-DNS-03)
|
||||
# =====================================================================================
|
||||
# Root cause (verified on dev): the nodes carried a SINGLE upstream resolver. PiHole
|
||||
# (192.168.1.18) answers `*.jackson.lan` instantly but its external recursion flaps —
|
||||
# 0/25 for registry-1.docker.io / quay.io / ghcr.io while gitea.jackson.lan resolved 5/5.
|
||||
# One upstream is a hard SPOF: containerd image pulls died with "lookup
|
||||
# registry-1.docker.io: Try again", and CoreDNS (`forward . /etc/resolv.conf`) flooded its
|
||||
# log with "i/o timeout" to 192.168.1.18. The fix is ADDITIVE — PiHole stays first so the
|
||||
# local zone and the ad-blocking policy are untouched; a second upstream is only consulted
|
||||
# when the first fails. See defaults/main.yml for the ordering rule.
|
||||
#
|
||||
# WHY THIS DETECTS INSTEAD OF ASSUMING. An earlier attempt (fix/node-dns-resilient-upstream,
|
||||
# dropped) wrote /etc/netplan/99-k3s-dns.yaml and ran `netplan apply`. netplan is NOT
|
||||
# installed on these nodes — they run the Debian 13 trixie generic cloud image
|
||||
# (infra/tofu/variables.tf) and netplan is not in node_baseline_packages — so that handler
|
||||
# would have failed command-not-found and broken node-baseline.yml on EVERY node. Worse,
|
||||
# the two environments do not even share a resolver stack: dev's /etc/resolv.conf is the
|
||||
# systemd-resolved stub (127.0.0.53) while prod's is a directly-managed file listing
|
||||
# 192.168.2.1 (it has no cloud-init at all — bare metal). So the mechanism is DETECTED per
|
||||
# node and the matching seam is written. Adding a third mechanism means adding a branch
|
||||
# here, not rewriting the role.
|
||||
#
|
||||
# All of it is `dns`-tagged, so `--tags packages` (the prod package-floor escape hatch
|
||||
# documented at the top of this file) still installs nothing but packages.
|
||||
|
||||
- name: Determine whether systemd-resolved is running on this node
|
||||
ansible.builtin.command: systemctl is-active systemd-resolved
|
||||
register: node_baseline_resolved_active
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
tags: [dns]
|
||||
|
||||
- name: Determine whether NetworkManager is running (it rewrites /etc/resolv.conf)
|
||||
ansible.builtin.command: systemctl is-active NetworkManager
|
||||
register: node_baseline_nm_active
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
tags: [dns]
|
||||
|
||||
# follow: false — we need to know whether /etc/resolv.conf IS a symlink and where it
|
||||
# points, which is the strongest single signal of who owns the file.
|
||||
- name: Inspect /etc/resolv.conf itself (symlink or real file, and its target)
|
||||
ansible.builtin.stat:
|
||||
path: /etc/resolv.conf
|
||||
follow: false
|
||||
register: node_baseline_resolv_stat
|
||||
tags: [dns]
|
||||
|
||||
- name: Read the current /etc/resolv.conf contents
|
||||
ansible.builtin.slurp:
|
||||
src: /etc/resolv.conf
|
||||
register: node_baseline_resolv_slurp
|
||||
failed_when: false
|
||||
tags: [dns]
|
||||
|
||||
# resolvconf/openresolv keeps the authoritative copy under here and regenerates
|
||||
# /etc/resolv.conf from it, so writing /etc/resolv.conf directly would be silently undone.
|
||||
- name: Detect a resolvconf/openresolv installation that would clobber /etc/resolv.conf
|
||||
ansible.builtin.stat:
|
||||
path: /etc/resolvconf/resolv.conf.d
|
||||
register: node_baseline_resolvconf_dir
|
||||
tags: [dns]
|
||||
|
||||
# THE DETECTION. Three outcomes, in priority order:
|
||||
# systemd-resolved — resolved is running AND it owns /etc/resolv.conf, proven either by
|
||||
# the symlink pointing into /run/systemd/resolve/ or by the file containing only
|
||||
# resolved's loopback stub (127.0.0.53 / 127.0.0.54). Writing a resolved drop-in is
|
||||
# then the ONLY thing that changes real behaviour; writing resolv.conf would be
|
||||
# overwritten by resolved (or, worse, land in /run and vanish at reboot).
|
||||
# contended — nobody's stub, but resolvconf or NetworkManager is present/running and
|
||||
# would regenerate the file underneath us. We do NOT write. The role fails loudly
|
||||
# below rather than leaving an operator believing a change stuck when it did not.
|
||||
# resolv-conf-file — a plain, directly-managed /etc/resolv.conf (prod's shape today).
|
||||
# Safe to own outright.
|
||||
# resolved running but NOT owning resolv.conf correctly falls through to the file branch:
|
||||
# in that state the drop-in would change nothing glibc can see.
|
||||
- name: Detect which resolver mechanism this node actually uses
|
||||
ansible.builtin.set_fact:
|
||||
node_baseline_dns_mechanism: >-
|
||||
{{
|
||||
'systemd-resolved' if (node_baseline_resolved_running and node_baseline_resolved_owns_resolv)
|
||||
else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present)
|
||||
else 'resolv-conf-file')
|
||||
}}
|
||||
vars:
|
||||
node_baseline_resolved_running: "{{ (node_baseline_resolved_active.stdout | default('') | trim) == 'active' }}"
|
||||
node_baseline_nm_running: "{{ (node_baseline_nm_active.stdout | default('') | trim) == 'active' }}"
|
||||
node_baseline_resolvconf_present: "{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}"
|
||||
node_baseline_resolv_text: "{{ (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
|
||||
node_baseline_resolv_ns: "{{ node_baseline_resolv_text | regex_findall('(?m)^\\s*nameserver\\s+(\\S+)') }}"
|
||||
node_baseline_resolved_owns_resolv: >-
|
||||
{{ ('/run/systemd/resolve/' in (node_baseline_resolv_stat.stat.lnk_source | default('')))
|
||||
or ((node_baseline_resolv_ns | length) > 0
|
||||
and (node_baseline_resolv_ns | reject('in', ['127.0.0.53', '127.0.0.54']) | list | length) == 0) }}
|
||||
tags: [dns]
|
||||
|
||||
- name: Report the detected resolver mechanism
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
resolver mechanism = {{ node_baseline_dns_mechanism }}
|
||||
(systemd-resolved is-active={{ node_baseline_resolved_active.stdout | default('') | trim }},
|
||||
NetworkManager is-active={{ node_baseline_nm_active.stdout | default('') | trim }},
|
||||
/etc/resolv.conf islnk={{ node_baseline_resolv_stat.stat.islnk | default(false) }}
|
||||
target={{ node_baseline_resolv_stat.stat.lnk_source | default('(not a symlink)') }},
|
||||
resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }})
|
||||
tags: [dns]
|
||||
|
||||
# Same shape, and the same reason, as the data-disk assert above: this environment's
|
||||
# upstream list is an OPERATOR DECISION, and a placeholder that fails LOUDLY beats both a
|
||||
# guess and a silent inheritance of dev's PiHole. group_vars/k3s_prod/vars.yml deliberately
|
||||
# carries a BLOCKED second entry (see its comments and PROD-PROVISIONING.md §4): prod's
|
||||
# 192.168.2.1 is currently gateway AND only upstream, and choosing what sits behind it is a
|
||||
# network decision this role must not make. Running against prod stops HERE, before
|
||||
# anything is written.
|
||||
- name: Assert this environment's DNS upstream list has actually been decided
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- node_baseline_dns_servers | length >= 2
|
||||
- "'BLOCKED' not in (node_baseline_dns_servers | join(' '))"
|
||||
fail_msg: >-
|
||||
node_baseline_dns_servers is not a decided, redundant upstream list for this host's
|
||||
environment (got: {{ node_baseline_dns_servers | join(', ') }}). A single upstream is
|
||||
the exact SPOF this task exists to remove — it already broke containerd image pulls
|
||||
and CoreDNS forwarding on dev — so the role will not configure one, and it will NOT
|
||||
guess a second server: a public resolver would bypass the local zone and the
|
||||
ad-blocking policy, which is a decision, not a default. Decide the second upstream,
|
||||
record it in infra/PROD-PROVISIONING.md §4, and set node_baseline_dns_servers in
|
||||
inventory/group_vars/k3s_prod/vars.yml in the SAME change. To install only the
|
||||
package floor meanwhile, re-run with `--tags packages`, which skips every DNS task.
|
||||
quiet: true
|
||||
tags: [dns]
|
||||
|
||||
# The `contended` outcome. Not a warning: an operator who is told "changed" while another
|
||||
# daemon quietly reverts the file is worse off than one who is told nothing happened.
|
||||
- name: Refuse to manage /etc/resolv.conf while another resolver manager owns it
|
||||
ansible.builtin.fail:
|
||||
msg: >-
|
||||
/etc/resolv.conf on this node is not the systemd-resolved stub, but
|
||||
resolvconf/openresolv (/etc/resolvconf/resolv.conf.d present:
|
||||
{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}) or NetworkManager
|
||||
(is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}) is in play and
|
||||
would regenerate the file underneath this role. Writing it here would look like it
|
||||
worked and silently revert. Add the matching branch to roles/node_baseline (the
|
||||
`dns`-tagged block) for whichever manager this node really uses — do not force it.
|
||||
when: node_baseline_dns_mechanism == 'contended'
|
||||
tags: [dns]
|
||||
|
||||
# --- Mechanism A: systemd-resolved (dev's shape today) -------------------------------
|
||||
# A drop-in, never an edit of /etc/systemd/resolved.conf, so a package upgrade of that file
|
||||
# never conflicts. `DNS=` is one ORDERED list inside a single resolved scope: resolved uses
|
||||
# the first server and only rotates to the next when it fails, which is exactly the
|
||||
# PiHole-primary/fallback behaviour wanted — it is NOT a race between two servers, so a
|
||||
# gateway NXDOMAIN can never beat PiHole to a jackson.lan answer while PiHole is healthy.
|
||||
- name: Ensure the systemd-resolved drop-in directory exists
|
||||
ansible.builtin.file:
|
||||
path: /etc/systemd/resolved.conf.d
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
when: node_baseline_dns_mechanism == 'systemd-resolved'
|
||||
tags: [dns]
|
||||
|
||||
- name: Configure resilient DNS upstreams via a systemd-resolved drop-in
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/systemd/resolved.conf.d/99-k3s-dns.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# Managed by the Ansible node_baseline role (tags: dns) — do not edit by hand.
|
||||
# Ordered upstream list: the FIRST server is the local-zone/ad-blocking resolver and
|
||||
# stays authoritative; the rest exist only so external resolution (containerd image
|
||||
# pulls, CoreDNS `forward . /etc/resolv.conf`) survives its recursion flapping.
|
||||
[Resolve]
|
||||
DNS={{ node_baseline_dns_servers | join(' ') }}
|
||||
Domains={{ node_baseline_dns_search | join(' ') }}
|
||||
# Reset to empty deliberately. systemd ships compiled-in public fallback resolvers
|
||||
# (Cloudflare/Google) that resolved would reach for if the list above ever came up
|
||||
# empty — which would silently bypass BOTH the local zone and the network's
|
||||
# ad-blocking. This fix adds redundancy; it never adds a bypass.
|
||||
FallbackDNS=
|
||||
notify: Restart systemd-resolved
|
||||
when: node_baseline_dns_mechanism == 'systemd-resolved'
|
||||
tags: [dns]
|
||||
|
||||
# --- Mechanism B: a directly-managed /etc/resolv.conf (prod's shape today) -----------
|
||||
# Only reached when nothing else owns the file (the `contended` branch above already
|
||||
# stopped the play otherwise). glibc walks `nameserver` lines in order and falls through
|
||||
# on timeout, so ordering carries the same policy as the resolved list.
|
||||
# NOTE this REPLACES the file: any nameserver currently there and NOT in
|
||||
# node_baseline_dns_servers is dropped. That is the point — the list in group_vars is the
|
||||
# whole declared truth — so an environment that wants its existing IPv6 upstreams kept
|
||||
# must list them there too. `backup: true` leaves the pre-role copy beside it the first
|
||||
# time it changes, because this file is load-bearing for reaching the node at all.
|
||||
- name: Configure resilient DNS upstreams directly in /etc/resolv.conf
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/resolv.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
backup: true
|
||||
content: |
|
||||
# Managed by the Ansible node_baseline role (tags: dns) — do not edit by hand.
|
||||
# Ordered upstream list: the FIRST server is the local-zone resolver and stays
|
||||
# authoritative; the rest exist only so external resolution (containerd image pulls,
|
||||
# CoreDNS `forward . /etc/resolv.conf`) survives its recursion flapping.
|
||||
{% if node_baseline_dns_search | length > 0 %}
|
||||
search {{ node_baseline_dns_search | join(' ') }}
|
||||
{% endif %}
|
||||
{% for server in node_baseline_dns_servers %}
|
||||
nameserver {{ server }}
|
||||
{% endfor %}
|
||||
{% if node_baseline_dns_options | length > 0 %}
|
||||
options {{ node_baseline_dns_options | join(' ') }}
|
||||
{% endif %}
|
||||
when: node_baseline_dns_mechanism == 'resolv-conf-file'
|
||||
tags: [dns]
|
||||
Reference in new issue
Block a user