fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03)
Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.
The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.
MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
* systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
silently bypass the local zone and ad-blocking.
* resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
would destroy PiHole-first ordering.
* contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.
SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.
PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.
tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.
ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.
Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
14be9b68f6
commit
c1f24e4a6d
10 files changed
+505
-23
No files matched your search
@@ -125,10 +125,37 @@ Consequences to settle before imaging nodes 2 and 3:
|
||||
(`GATEWAY_VIP` `.53` = first of `.53-.60`).
|
||||
- [ ] **Reserve all of `192.168.2.9` and `.20-.30` outside any DHCP range** on the prod LAN before
|
||||
the build, and confirm nothing else on the network already answers on them.
|
||||
- [ ] **Resolver: configure at least two upstreams.** Node 1 currently has a single IPv4 nameserver
|
||||
(`192.168.2.1`). A single upstream is a known SPOF that has already bitten this project —
|
||||
it broke containerd image pulls and CoreDNS forwarding on dev. A fix exists but is **stranded
|
||||
unmerged** on `fix/node-dns-resilient-upstream`; land it, then apply the same pattern here.
|
||||
- [ ] **Resolver: SECOND UPSTREAM IS AN OPEN DECISION — blocks `node-baseline.yml --tags dns`.**
|
||||
Node 1 has a single IPv4 nameserver, `192.168.2.1`, which is also the LAN gateway. A single
|
||||
upstream is a known SPOF that has already bitten this project — it broke containerd image
|
||||
pulls and CoreDNS forwarding on dev.
|
||||
|
||||
The MECHANISM is now in the repo and needs no further work: the `dns`-tagged block in
|
||||
`roles/node_baseline` detects whether a node uses systemd-resolved (dev: writes
|
||||
`/etc/systemd/resolved.conf.d/99-k3s-dns.conf`, restarts resolved) or a directly-managed
|
||||
`/etc/resolv.conf` (prod's shape: writes the file, refusing if resolvconf/NetworkManager
|
||||
owns it), and configures the ordered upstream list either way. The earlier
|
||||
`fix/node-dns-resilient-upstream` branch was **dropped, not merged** — it wrote a netplan
|
||||
drop-in and ran `netplan apply`, and netplan is not installed on these Debian 13 trixie
|
||||
nodes, so it would have broken `node-baseline.yml` on every host.
|
||||
|
||||
What is still MISSING is the VALUE. `group_vars/k3s_prod/vars.yml` sets
|
||||
`node_baseline_dns_servers` to `192.168.2.1` followed by the deliberately invalid
|
||||
`BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4`, and the role's assert
|
||||
stops the play on it before writing anything. Choosing the second upstream is a **network
|
||||
decision, not a default the role may take**: a public resolver (`8.8.8.8`, `1.1.1.1`) would
|
||||
resolve reliably but would also bypass any local zone and any filtering the gateway applies.
|
||||
Decide between — and record the choice HERE, in the same change that edits the group_vars:
|
||||
- a second on-LAN resolver on `192.168.2.x`;
|
||||
- the dev-side PiHole `192.168.1.18`, **if** prod can actually route to it (unverified —
|
||||
prod sits on the workstation side of the pfSense boundary);
|
||||
- an explicit, written-down acceptance of a public resolver as fallback-only.
|
||||
|
||||
Also decide whether node 1's current IPv6 nameservers (`2001:4860:4860::8888` / `::8844`)
|
||||
are kept: the direct-file write path replaces `/etc/resolv.conf` with exactly the listed
|
||||
servers, so keeping them means listing them explicitly.
|
||||
`node_baseline_dns_search` is `[]` (no search domain, node 1's behaviour today) until prod's
|
||||
local domain is decided.
|
||||
- [ ] Confirm the prod LAN's relationship to the dev LAN (`192.168.1.x`) and to the pfSense NAT —
|
||||
prod on `192.168.2.x` sits on the *workstation* side of the boundary dev lives behind.
|
||||
|
||||
|
||||
@@ -9,8 +9,9 @@
|
||||
# Pairs with vault.yml in this same directory (ansible-vault-encrypted, operator-
|
||||
# created — see README.md here) for the one secret prod needs: ansible_become_password.
|
||||
#
|
||||
# PHASE 7 IS PREP ONLY. Four vars below (kube_vip_address, cluster_controller_api_server,
|
||||
# node_baseline_longhorn_device, node_baseline_etcd_device) are deliberately BROKEN
|
||||
# PHASE 7 IS PREP ONLY. Five vars below (kube_vip_address, cluster_controller_api_server,
|
||||
# node_baseline_longhorn_device, node_baseline_etcd_device, and the SECOND entry of
|
||||
# node_baseline_dns_servers) are deliberately BROKEN
|
||||
# PLACEHOLDERS, not real values — see their comments. Do not "fix" them with a guess;
|
||||
# follow the referenced section of PROD-PROVISIONING.md, record the decision there,
|
||||
# and update this file in the same change.
|
||||
@@ -104,3 +105,44 @@ cluster_controller_api_server: "https://{{ kube_vip_address }}:6443"
|
||||
# (Longhorn replica placement assumes symmetry) before any of this is used for real.
|
||||
node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
|
||||
node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
|
||||
|
||||
# --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) -----
|
||||
# roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1
|
||||
# pfSense) — addresses on a network prod cannot even reach. Inherited verbatim they would
|
||||
# point every prod node at a PiHole that does not answer it, so they are overridden here,
|
||||
# in the same seam as kube_vip_address / k3s_api_host / cluster_kubeconfig_name above.
|
||||
# (NOT clusters/base/cluster-vars.yaml: that ConfigMap is consumed by Flux
|
||||
# postBuild.substituteFrom for platform/ and apps/ manifests, and no Ansible code reads it.)
|
||||
#
|
||||
# WHAT IS DECIDED: 192.168.2.1 — node 1's current sole nameserver, and the LAN gateway.
|
||||
# It stays FIRST, so whatever local zone it serves keeps working (the dev outage taught
|
||||
# that ordering is policy, not preference — see the role's defaults).
|
||||
#
|
||||
# WHAT IS NOT: the SECOND upstream. 192.168.2.1 is presently gateway AND only IPv4
|
||||
# resolver, and there is no second address on this LAN that is known to answer. Picking
|
||||
# one is a NETWORK decision, not a default this role may take: a public resolver
|
||||
# (8.8.8.8, 1.1.1.1) would resolve reliably but would ALSO bypass any local zone and any
|
||||
# network-level filtering the gateway applies — precisely the bypass the dev fix refused
|
||||
# to make. Candidates the operator must choose between and record in
|
||||
# infra/PROD-PROVISIONING.md §4: a second on-LAN resolver, the dev-side PiHole IF prod can
|
||||
# route to 192.168.1.18, or an explicit, written-down decision to accept a public resolver
|
||||
# as fallback-only.
|
||||
#
|
||||
# Deliberately present-but-broken, exactly like node_baseline_longhorn_device above:
|
||||
# leaving the list at one entry, or unset, would either configure the very SPOF this exists
|
||||
# to remove or silently inherit dev's PiHole. This value can never parse as an address, and
|
||||
# the `dns`-tagged assert in roles/node_baseline stops the play on the string BLOCKED
|
||||
# before a single file is written. DO NOT replace it with a guess — replace it only once
|
||||
# §4 is decided, and record the chosen resolver there in the SAME change.
|
||||
#
|
||||
# Note the two Google IPv6 servers currently in node 1's /etc/resolv.conf
|
||||
# (2001:4860:4860::8888/::8844) are NOT carried over: the direct-file write path replaces
|
||||
# the file with exactly this list, so if they are to be kept they must be listed here
|
||||
# explicitly — as part of that same recorded decision.
|
||||
node_baseline_dns_servers:
|
||||
- "192.168.2.1"
|
||||
- "BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4"
|
||||
# Prod's local search domain is not a decided thing (dev's is jackson.lan, PiHole's zone).
|
||||
# An EMPTY list is the role's supported way to say "configure no search domain", which is
|
||||
# node 1's current behaviour — the conservative non-decision, not a missing value.
|
||||
node_baseline_dns_search: []
|
||||
@@ -34,3 +34,40 @@ node_baseline_longhorn_fs_label: longhorn
|
||||
node_baseline_etcd_device: /dev/vdc
|
||||
node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db
|
||||
node_baseline_etcd_fs_label: k3s-etcd
|
||||
|
||||
# --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
|
||||
# These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml
|
||||
# (the same seam kube_vip_address / k3s_api_host / cluster_kubeconfig_name use) — NOT in
|
||||
# clusters/base/cluster-vars.yaml, which is a Flux postBuild ConfigMap that no Ansible
|
||||
# code reads.
|
||||
#
|
||||
# WHY: the nodes shipped with ONE upstream resolver. PiHole answers the local jackson.lan
|
||||
# zone instantly but its external recursion flaps (measured 0/25 for registry-1.docker.io /
|
||||
# quay.io / ghcr.io while gitea.jackson.lan resolved 5/5), and one upstream is a hard SPOF:
|
||||
# it broke containerd image pulls ("lookup registry-1.docker.io: Try again") AND CoreDNS,
|
||||
# which forwards to /etc/resolv.conf and flooded its log with i/o timeouts to 192.168.1.18.
|
||||
#
|
||||
# ORDER IS THE POLICY. This is a RESILIENCE fix, not a bypass: the first entry stays PiHole,
|
||||
# so *.jackson.lan and network-wide ad-blocking keep working exactly as before, and the
|
||||
# second is only ever consulted when the first fails. Both write paths preserve that order
|
||||
# (systemd-resolved tries the servers of one scope in order; glibc walks `nameserver` lines
|
||||
# in order). Never put a public resolver first, and never enable `rotate` — either would
|
||||
# silently take the local zone and the ad-block policy out of the path.
|
||||
#
|
||||
# Keep in parity with infra/tofu (var.dns_server, var.network_gateway), which sets the same
|
||||
# two servers via cloud-init so a freshly rebuilt dev node boots with this list already.
|
||||
node_baseline_dns_servers:
|
||||
- 192.168.1.18 # PiHole — local jackson.lan zone + ad-blocking. PRIMARY, keep first.
|
||||
- 192.168.1.1 # pfSense gateway — reliable external recursion. FALLBACK only.
|
||||
# Search domain(s). An EMPTY list means "configure no search domain" and is a legitimate
|
||||
# value (prod uses it — its local domain is not a decided thing), not a missing one.
|
||||
node_baseline_dns_search:
|
||||
- jackson.lan
|
||||
# resolv.conf `options` for the DIRECT-file mechanism only (systemd-resolved does its own
|
||||
# per-server failover timing and ignores these). glibc's defaults are timeout:5 attempts:2,
|
||||
# i.e. up to 20s of stalled lookups before it reaches the second server — long enough for
|
||||
# containerd to give up on a pull. These shorten the failover, and deliberately do NOT
|
||||
# include `rotate`, which would round-robin the servers and break PiHole-first ordering.
|
||||
node_baseline_dns_options:
|
||||
- timeout:2
|
||||
- attempts:2
|
||||
@@ -2,3 +2,12 @@
|
||||
- name: Reload sysctl
|
||||
ansible.builtin.command: sysctl --system
|
||||
changed_when: true
|
||||
|
||||
# Restarting systemd-resolved re-reads /etc/systemd/resolved.conf.d/ and re-points the
|
||||
# 127.0.0.53 stub at the new upstream list. It does NOT touch link addressing or routes,
|
||||
# so the SSH/NAT session Ansible is running over is unaffected. Only fires when the
|
||||
# `dns`-tagged drop-in above actually changed, so a re-run of the role is a no-op.
|
||||
- name: Restart systemd-resolved
|
||||
ansible.builtin.systemd:
|
||||
name: systemd-resolved
|
||||
state: restarted
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
# TAGS — every task here carries one of `packages`, `system` or `disks`. An untagged run
|
||||
# is unchanged (all three run, in this order). The split exists because the PACKAGE floor
|
||||
# TAGS — every task here carries one of `packages`, `system`, `disks` or `dns`. An untagged
|
||||
# run is unchanged (all four run, in this order). The split exists because the PACKAGE floor
|
||||
# and the DATA-DISK work have different readiness: packages are safe on any node from
|
||||
# first contact, while the disk tasks need block devices that a new environment may not
|
||||
# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
|
||||
@@ -9,7 +9,9 @@
|
||||
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
|
||||
# Skipping is the guard: with `--tags packages` no `disks`-tagged task is selected, so a
|
||||
# BLOCKED-pending-disk-decision device path is never even templated, let alone handed to
|
||||
# mkfs. A FULL run against such a node stops at the assert below instead.
|
||||
# mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
|
||||
# at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
|
||||
# ansible-playbook playbooks/node-baseline.yml --tags dns
|
||||
|
||||
- name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
|
||||
ansible.builtin.apt:
|
||||
@@ -154,3 +156,225 @@
|
||||
regexp: '^([^#].*\sswap\s.*)$'
|
||||
replace: '# \1 # disabled for k3s/kubelet'
|
||||
tags: [system]
|
||||
|
||||
# =====================================================================================
|
||||
# DNS RESILIENCE — tag `dns` (R-DNS-01 / R-DNS-03)
|
||||
# =====================================================================================
|
||||
# Root cause (verified on dev): the nodes carried a SINGLE upstream resolver. PiHole
|
||||
# (192.168.1.18) answers `*.jackson.lan` instantly but its external recursion flaps —
|
||||
# 0/25 for registry-1.docker.io / quay.io / ghcr.io while gitea.jackson.lan resolved 5/5.
|
||||
# One upstream is a hard SPOF: containerd image pulls died with "lookup
|
||||
# registry-1.docker.io: Try again", and CoreDNS (`forward . /etc/resolv.conf`) flooded its
|
||||
# log with "i/o timeout" to 192.168.1.18. The fix is ADDITIVE — PiHole stays first so the
|
||||
# local zone and the ad-blocking policy are untouched; a second upstream is only consulted
|
||||
# when the first fails. See defaults/main.yml for the ordering rule.
|
||||
#
|
||||
# WHY THIS DETECTS INSTEAD OF ASSUMING. An earlier attempt (fix/node-dns-resilient-upstream,
|
||||
# dropped) wrote /etc/netplan/99-k3s-dns.yaml and ran `netplan apply`. netplan is NOT
|
||||
# installed on these nodes — they run the Debian 13 trixie generic cloud image
|
||||
# (infra/tofu/variables.tf) and netplan is not in node_baseline_packages — so that handler
|
||||
# would have failed command-not-found and broken node-baseline.yml on EVERY node. Worse,
|
||||
# the two environments do not even share a resolver stack: dev's /etc/resolv.conf is the
|
||||
# systemd-resolved stub (127.0.0.53) while prod's is a directly-managed file listing
|
||||
# 192.168.2.1 (it has no cloud-init at all — bare metal). So the mechanism is DETECTED per
|
||||
# node and the matching seam is written. Adding a third mechanism means adding a branch
|
||||
# here, not rewriting the role.
|
||||
#
|
||||
# All of it is `dns`-tagged, so `--tags packages` (the prod package-floor escape hatch
|
||||
# documented at the top of this file) still installs nothing but packages.
|
||||
|
||||
- name: Determine whether systemd-resolved is running on this node
|
||||
ansible.builtin.command: systemctl is-active systemd-resolved
|
||||
register: node_baseline_resolved_active
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
tags: [dns]
|
||||
|
||||
- name: Determine whether NetworkManager is running (it rewrites /etc/resolv.conf)
|
||||
ansible.builtin.command: systemctl is-active NetworkManager
|
||||
register: node_baseline_nm_active
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
tags: [dns]
|
||||
|
||||
# follow: false — we need to know whether /etc/resolv.conf IS a symlink and where it
|
||||
# points, which is the strongest single signal of who owns the file.
|
||||
- name: Inspect /etc/resolv.conf itself (symlink or real file, and its target)
|
||||
ansible.builtin.stat:
|
||||
path: /etc/resolv.conf
|
||||
follow: false
|
||||
register: node_baseline_resolv_stat
|
||||
tags: [dns]
|
||||
|
||||
- name: Read the current /etc/resolv.conf contents
|
||||
ansible.builtin.slurp:
|
||||
src: /etc/resolv.conf
|
||||
register: node_baseline_resolv_slurp
|
||||
failed_when: false
|
||||
tags: [dns]
|
||||
|
||||
# resolvconf/openresolv keeps the authoritative copy under here and regenerates
|
||||
# /etc/resolv.conf from it, so writing /etc/resolv.conf directly would be silently undone.
|
||||
- name: Detect a resolvconf/openresolv installation that would clobber /etc/resolv.conf
|
||||
ansible.builtin.stat:
|
||||
path: /etc/resolvconf/resolv.conf.d
|
||||
register: node_baseline_resolvconf_dir
|
||||
tags: [dns]
|
||||
|
||||
# THE DETECTION. Three outcomes, in priority order:
|
||||
# systemd-resolved — resolved is running AND it owns /etc/resolv.conf, proven either by
|
||||
# the symlink pointing into /run/systemd/resolve/ or by the file containing only
|
||||
# resolved's loopback stub (127.0.0.53 / 127.0.0.54). Writing a resolved drop-in is
|
||||
# then the ONLY thing that changes real behaviour; writing resolv.conf would be
|
||||
# overwritten by resolved (or, worse, land in /run and vanish at reboot).
|
||||
# contended — nobody's stub, but resolvconf or NetworkManager is present/running and
|
||||
# would regenerate the file underneath us. We do NOT write. The role fails loudly
|
||||
# below rather than leaving an operator believing a change stuck when it did not.
|
||||
# resolv-conf-file — a plain, directly-managed /etc/resolv.conf (prod's shape today).
|
||||
# Safe to own outright.
|
||||
# resolved running but NOT owning resolv.conf correctly falls through to the file branch:
|
||||
# in that state the drop-in would change nothing glibc can see.
|
||||
- name: Detect which resolver mechanism this node actually uses
|
||||
ansible.builtin.set_fact:
|
||||
node_baseline_dns_mechanism: >-
|
||||
{{
|
||||
'systemd-resolved' if (node_baseline_resolved_running and node_baseline_resolved_owns_resolv)
|
||||
else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present)
|
||||
else 'resolv-conf-file')
|
||||
}}
|
||||
vars:
|
||||
node_baseline_resolved_running: "{{ (node_baseline_resolved_active.stdout | default('') | trim) == 'active' }}"
|
||||
node_baseline_nm_running: "{{ (node_baseline_nm_active.stdout | default('') | trim) == 'active' }}"
|
||||
node_baseline_resolvconf_present: "{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}"
|
||||
node_baseline_resolv_text: "{{ (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
|
||||
node_baseline_resolv_ns: "{{ node_baseline_resolv_text | regex_findall('(?m)^\\s*nameserver\\s+(\\S+)') }}"
|
||||
node_baseline_resolved_owns_resolv: >-
|
||||
{{ ('/run/systemd/resolve/' in (node_baseline_resolv_stat.stat.lnk_source | default('')))
|
||||
or ((node_baseline_resolv_ns | length) > 0
|
||||
and (node_baseline_resolv_ns | reject('in', ['127.0.0.53', '127.0.0.54']) | list | length) == 0) }}
|
||||
tags: [dns]
|
||||
|
||||
- name: Report the detected resolver mechanism
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
resolver mechanism = {{ node_baseline_dns_mechanism }}
|
||||
(systemd-resolved is-active={{ node_baseline_resolved_active.stdout | default('') | trim }},
|
||||
NetworkManager is-active={{ node_baseline_nm_active.stdout | default('') | trim }},
|
||||
/etc/resolv.conf islnk={{ node_baseline_resolv_stat.stat.islnk | default(false) }}
|
||||
target={{ node_baseline_resolv_stat.stat.lnk_source | default('(not a symlink)') }},
|
||||
resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }})
|
||||
tags: [dns]
|
||||
|
||||
# Same shape, and the same reason, as the data-disk assert above: this environment's
|
||||
# upstream list is an OPERATOR DECISION, and a placeholder that fails LOUDLY beats both a
|
||||
# guess and a silent inheritance of dev's PiHole. group_vars/k3s_prod/vars.yml deliberately
|
||||
# carries a BLOCKED second entry (see its comments and PROD-PROVISIONING.md §4): prod's
|
||||
# 192.168.2.1 is currently gateway AND only upstream, and choosing what sits behind it is a
|
||||
# network decision this role must not make. Running against prod stops HERE, before
|
||||
# anything is written.
|
||||
- name: Assert this environment's DNS upstream list has actually been decided
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- node_baseline_dns_servers | length >= 2
|
||||
- "'BLOCKED' not in (node_baseline_dns_servers | join(' '))"
|
||||
fail_msg: >-
|
||||
node_baseline_dns_servers is not a decided, redundant upstream list for this host's
|
||||
environment (got: {{ node_baseline_dns_servers | join(', ') }}). A single upstream is
|
||||
the exact SPOF this task exists to remove — it already broke containerd image pulls
|
||||
and CoreDNS forwarding on dev — so the role will not configure one, and it will NOT
|
||||
guess a second server: a public resolver would bypass the local zone and the
|
||||
ad-blocking policy, which is a decision, not a default. Decide the second upstream,
|
||||
record it in infra/PROD-PROVISIONING.md §4, and set node_baseline_dns_servers in
|
||||
inventory/group_vars/k3s_prod/vars.yml in the SAME change. To install only the
|
||||
package floor meanwhile, re-run with `--tags packages`, which skips every DNS task.
|
||||
quiet: true
|
||||
tags: [dns]
|
||||
|
||||
# The `contended` outcome. Not a warning: an operator who is told "changed" while another
|
||||
# daemon quietly reverts the file is worse off than one who is told nothing happened.
|
||||
- name: Refuse to manage /etc/resolv.conf while another resolver manager owns it
|
||||
ansible.builtin.fail:
|
||||
msg: >-
|
||||
/etc/resolv.conf on this node is not the systemd-resolved stub, but
|
||||
resolvconf/openresolv (/etc/resolvconf/resolv.conf.d present:
|
||||
{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}) or NetworkManager
|
||||
(is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}) is in play and
|
||||
would regenerate the file underneath this role. Writing it here would look like it
|
||||
worked and silently revert. Add the matching branch to roles/node_baseline (the
|
||||
`dns`-tagged block) for whichever manager this node really uses — do not force it.
|
||||
when: node_baseline_dns_mechanism == 'contended'
|
||||
tags: [dns]
|
||||
|
||||
# --- Mechanism A: systemd-resolved (dev's shape today) -------------------------------
|
||||
# A drop-in, never an edit of /etc/systemd/resolved.conf, so a package upgrade of that file
|
||||
# never conflicts. `DNS=` is one ORDERED list inside a single resolved scope: resolved uses
|
||||
# the first server and only rotates to the next when it fails, which is exactly the
|
||||
# PiHole-primary/fallback behaviour wanted — it is NOT a race between two servers, so a
|
||||
# gateway NXDOMAIN can never beat PiHole to a jackson.lan answer while PiHole is healthy.
|
||||
- name: Ensure the systemd-resolved drop-in directory exists
|
||||
ansible.builtin.file:
|
||||
path: /etc/systemd/resolved.conf.d
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
when: node_baseline_dns_mechanism == 'systemd-resolved'
|
||||
tags: [dns]
|
||||
|
||||
- name: Configure resilient DNS upstreams via a systemd-resolved drop-in
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/systemd/resolved.conf.d/99-k3s-dns.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# Managed by the Ansible node_baseline role (tags: dns) — do not edit by hand.
|
||||
# Ordered upstream list: the FIRST server is the local-zone/ad-blocking resolver and
|
||||
# stays authoritative; the rest exist only so external resolution (containerd image
|
||||
# pulls, CoreDNS `forward . /etc/resolv.conf`) survives its recursion flapping.
|
||||
[Resolve]
|
||||
DNS={{ node_baseline_dns_servers | join(' ') }}
|
||||
Domains={{ node_baseline_dns_search | join(' ') }}
|
||||
# Reset to empty deliberately. systemd ships compiled-in public fallback resolvers
|
||||
# (Cloudflare/Google) that resolved would reach for if the list above ever came up
|
||||
# empty — which would silently bypass BOTH the local zone and the network's
|
||||
# ad-blocking. This fix adds redundancy; it never adds a bypass.
|
||||
FallbackDNS=
|
||||
notify: Restart systemd-resolved
|
||||
when: node_baseline_dns_mechanism == 'systemd-resolved'
|
||||
tags: [dns]
|
||||
|
||||
# --- Mechanism B: a directly-managed /etc/resolv.conf (prod's shape today) -----------
|
||||
# Only reached when nothing else owns the file (the `contended` branch above already
|
||||
# stopped the play otherwise). glibc walks `nameserver` lines in order and falls through
|
||||
# on timeout, so ordering carries the same policy as the resolved list.
|
||||
# NOTE this REPLACES the file: any nameserver currently there and NOT in
|
||||
# node_baseline_dns_servers is dropped. That is the point — the list in group_vars is the
|
||||
# whole declared truth — so an environment that wants its existing IPv6 upstreams kept
|
||||
# must list them there too. `backup: true` leaves the pre-role copy beside it the first
|
||||
# time it changes, because this file is load-bearing for reaching the node at all.
|
||||
- name: Configure resilient DNS upstreams directly in /etc/resolv.conf
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/resolv.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
backup: true
|
||||
content: |
|
||||
# Managed by the Ansible node_baseline role (tags: dns) — do not edit by hand.
|
||||
# Ordered upstream list: the FIRST server is the local-zone resolver and stays
|
||||
# authoritative; the rest exist only so external resolution (containerd image pulls,
|
||||
# CoreDNS `forward . /etc/resolv.conf`) survives its recursion flapping.
|
||||
{% if node_baseline_dns_search | length > 0 %}
|
||||
search {{ node_baseline_dns_search | join(' ') }}
|
||||
{% endif %}
|
||||
{% for server in node_baseline_dns_servers %}
|
||||
nameserver {{ server }}
|
||||
{% endfor %}
|
||||
{% if node_baseline_dns_options | length > 0 %}
|
||||
options {{ node_baseline_dns_options | join(' ') }}
|
||||
{% endif %}
|
||||
when: node_baseline_dns_mechanism == 'resolv-conf-file'
|
||||
tags: [dns]
|
||||
@@ -56,6 +56,9 @@ node_health_thresholds:
|
||||
nic_min_speed_mbps: 1000
|
||||
# A single upstream resolver is a known SPOF on this project — it broke containerd
|
||||
# image pulls and CoreDNS forwarding on dev (DECISIONS-LOG 2026-08-31, finding 5).
|
||||
# Counted against the node's REAL upstreams: where systemd-resolved is in use the
|
||||
# collector looks behind the 127.0.0.53 stub rather than counting it as one server.
|
||||
# The fix side of this is the `dns`-tagged block in roles/node_baseline.
|
||||
min_nameservers: 2
|
||||
# SMART (only evaluated when smartctl actually ran)
|
||||
smart_reallocated_warn: 1
|
||||
|
||||
@@ -4,8 +4,10 @@
|
||||
#
|
||||
# Run by roles/node_health via ansible.builtin.script. Prints ONE JSON document on
|
||||
# stdout and nothing else. It is strictly READ-ONLY: it opens files under /sys, /proc
|
||||
# and /etc, and shells out only to `lsblk` (a read-only query). It never writes to the
|
||||
# node, never installs anything, and never needs root.
|
||||
# and /etc, and shells out only to read-only queries (`lsblk`, `lscpu`, and `resolvectl dns`
|
||||
# — the last only when /etc/resolv.conf turns out to be the systemd-resolved stub and the
|
||||
# real upstreams have to be asked of resolved). It never writes to the node, never installs
|
||||
# anything, and never needs root.
|
||||
#
|
||||
# WHY A COLLECTOR SCRIPT AND NOT 40 ANSIBLE TASKS
|
||||
# Everything here is "read a sysfs file that may or may not exist and may or may not
|
||||
@@ -563,9 +565,113 @@ def collect_network():
|
||||
# Resolver config is a known SPOF class on this project (a single upstream broke
|
||||
# containerd pulls and CoreDNS forwarding on dev). Cheap to read here, so the health
|
||||
# report can flag it on prod too.
|
||||
resolv = read_text("/etc/resolv.conf") or ""
|
||||
nameservers = re.findall(r"^\s*nameserver\s+(\S+)", resolv, re.M)
|
||||
return {"interfaces": interfaces, "nameservers": nameservers}
|
||||
resolvers = collect_resolvers()
|
||||
return {
|
||||
"interfaces": interfaces,
|
||||
# List of REAL upstreams, or None when they could not be determined. NEVER the
|
||||
# loopback stub, and never [] standing in for "unknown".
|
||||
"nameservers": resolvers["nameservers"],
|
||||
"resolver_source": resolvers["source"],
|
||||
"resolver_stub": resolvers["stub"],
|
||||
"resolver_unavailable_reason": resolvers["unavailable_reason"],
|
||||
}
|
||||
|
||||
|
||||
# systemd-resolved's loopback listeners. 127.0.0.53 is the stub (the address written into
|
||||
# /etc/resolv.conf); 127.0.0.54 is the "bypass"/proxy listener. Neither is an UPSTREAM —
|
||||
# seeing one in /etc/resolv.conf means the real servers live inside resolved.
|
||||
RESOLVED_STUB_IPS = ("127.0.0.53", "127.0.0.54")
|
||||
|
||||
|
||||
def _resolved_upstreams():
|
||||
"""Real upstreams from systemd-resolved. Returns (servers, reason_unreadable).
|
||||
|
||||
`resolvectl dns` prints one line per scope ("Global:", "Link 2 (eth0): 1.2.3.4 ..."),
|
||||
which is far easier to parse reliably than the wrapped columns of `resolvectl status`.
|
||||
"""
|
||||
rc, out, err = run(["resolvectl", "dns", "--no-pager"])
|
||||
if rc is None:
|
||||
return None, (
|
||||
"/etc/resolv.conf holds only the systemd-resolved loopback stub, and the real "
|
||||
"upstreams behind it could not be read: resolvectl is not installed on this node"
|
||||
)
|
||||
if rc != 0:
|
||||
detail = (err.strip() or out.strip() or "no output").splitlines()[0]
|
||||
return None, (
|
||||
"/etc/resolv.conf holds only the systemd-resolved loopback stub, and "
|
||||
"`resolvectl dns` exited %s so the real upstreams could not be read: %s"
|
||||
% (rc, detail)
|
||||
)
|
||||
servers = []
|
||||
for line in out.splitlines():
|
||||
# Split on the FIRST colon only: everything after it is the server list, and IPv6
|
||||
# addresses contain colons of their own.
|
||||
_, sep, rest = line.partition(":")
|
||||
if not sep:
|
||||
continue
|
||||
for token in rest.split():
|
||||
if token not in servers:
|
||||
servers.append(token)
|
||||
if not servers:
|
||||
return None, (
|
||||
"/etc/resolv.conf holds only the systemd-resolved loopback stub and "
|
||||
"`resolvectl dns` reported NO upstream servers on any scope — resolved is "
|
||||
"running with an empty server list, so this node has no usable upstream"
|
||||
)
|
||||
return servers, None
|
||||
|
||||
|
||||
def collect_resolvers():
|
||||
"""The node's real DNS upstreams, plus how they were determined.
|
||||
|
||||
/etc/resolv.conf is NOT always the answer. Where systemd-resolved is in use (dev's
|
||||
nodes today) the file contains only resolved's loopback stub, 127.0.0.53 — reading it
|
||||
literally reports "1 nameserver" forever, which is a permanent FALSE single-upstream
|
||||
WARN that no amount of correct configuration can clear. When the stub is detected the
|
||||
real list is asked of resolved instead.
|
||||
|
||||
HONESTY DISCIPLINE, unchanged from the rest of this collector: if the stub is present
|
||||
and the real list cannot be read, "nameservers" is None with the reason recorded — NOT
|
||||
the stub address, NOT an empty list, and NOT a guess. An unknown must stay an unknown;
|
||||
the assess step turns it into a SKIP, never a PASS or a WARN.
|
||||
|
||||
LIMIT worth knowing: only systemd-resolved's stub is special-cased. Some OTHER local
|
||||
forwarder on 127.0.0.1 (dnsmasq, unbound) is reported exactly as it appears in
|
||||
resolv.conf, because nothing generic can enumerate an arbitrary forwarder's upstreams.
|
||||
"""
|
||||
path = "/etc/resolv.conf"
|
||||
# optional=False: a MISSING /etc/resolv.conf is itself a finding worth naming.
|
||||
resolv = read_text(path, optional=False)
|
||||
if resolv is None:
|
||||
return {
|
||||
"nameservers": None,
|
||||
"source": None,
|
||||
"stub": [],
|
||||
"unavailable_reason": "/etc/resolv.conf could not be read (see the unreadable list)",
|
||||
}
|
||||
listed = re.findall(r"^\s*nameserver\s+(\S+)", resolv, re.M)
|
||||
stub = [n for n in listed if n in RESOLVED_STUB_IPS]
|
||||
# Only when EVERY listed server is a stub is the file a pure indirection. A mixed file
|
||||
# (stub plus a real address) is reported as written — that is what the resolver library
|
||||
# will actually use.
|
||||
if listed and len(stub) == len(listed):
|
||||
servers, reason = _resolved_upstreams()
|
||||
if servers is None:
|
||||
UNREADABLE.append({"path": "resolvectl dns", "reason": reason})
|
||||
return {
|
||||
"nameservers": None,
|
||||
"source": None,
|
||||
"stub": listed,
|
||||
"unavailable_reason": reason,
|
||||
}
|
||||
return {
|
||||
"nameservers": servers,
|
||||
"source": "resolvectl dns (/etc/resolv.conf holds only the systemd-resolved "
|
||||
"stub %s)" % ", ".join(listed),
|
||||
"stub": listed,
|
||||
"unavailable_reason": None,
|
||||
}
|
||||
return {"nameservers": listed, "source": path, "stub": stub, "unavailable_reason": None}
|
||||
|
||||
|
||||
# --- OS / kernel / uptime -----------------------------------------------------------
|
||||
|
||||
@@ -435,32 +435,50 @@
|
||||
{{ err_total | string ~ ' error(s), ' ~ drop_total | string ~ ' drop(s) across '
|
||||
~ (up | map(attribute='name') | join(', ')) }}
|
||||
|
||||
# The collector reports the node's REAL upstreams: where /etc/resolv.conf holds only
|
||||
# systemd-resolved's 127.0.0.53 stub (dev's shape) it asks resolved for the servers behind
|
||||
# it, because counting the stub would be a permanent FALSE single-upstream WARN that no
|
||||
# correct configuration could ever clear. When that lookup fails it reports null WITH a
|
||||
# reason rather than a number — so the check SKIPs. A skip is honest; a WARN derived from
|
||||
# an unknown is a fabricated verdict, and a PASS would be worse.
|
||||
- name: Assess — DNS resolver redundancy
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
'id': 'network.resolvers',
|
||||
'title': 'Configured DNS upstreams (single upstream is a known SPOF here)',
|
||||
'status': ('PASS' if (ns4 | length) >= (node_health_thresholds.min_nameservers | int) else 'WARN'),
|
||||
'status': ('SKIP' if node_health_facts.network.nameservers is none
|
||||
else ('PASS' if (ns4 | length) >= (node_health_thresholds.min_nameservers | int)
|
||||
else 'WARN')),
|
||||
'detail': ns_detail }] }}"
|
||||
vars:
|
||||
ansible_become: false
|
||||
ns: "{{ node_health_facts.network.nameservers }}"
|
||||
# `| default([], true)` only ever fires on the null (unreadable) path, where the
|
||||
# counts below are unused because the status is already SKIP.
|
||||
ns: "{{ node_health_facts.network.nameservers | default([], true) }}"
|
||||
# Counted PER FAMILY on purpose. The outage this check exists for was a single IPv4
|
||||
# upstream; IPv6 upstreams do not cover for it unless the node actually has working
|
||||
# IPv6 egress, which this role cannot prove from sysfs alone. So the verdict keys off
|
||||
# the IPv4 count and the detail shows the full picture.
|
||||
ns4: "{{ ns | select('search', '^[0-9.]+$') | list }}"
|
||||
ns6: "{{ ns | reject('search', '^[0-9.]+$') | list }}"
|
||||
# Where the numbers came from — resolv.conf directly, or resolved behind the stub.
|
||||
ns_src: "{{ node_health_facts.network.resolver_source | default('', true) }}"
|
||||
ns_detail: >-
|
||||
{{ (ns | length) | string ~ ' nameserver(s) — IPv4: '
|
||||
{{ ('not read — ' ~ (node_health_facts.network.resolver_unavailable_reason
|
||||
| default('the collector recorded no reason', true))
|
||||
~ '. The upstream count is UNKNOWN here — not zero, and not one.')
|
||||
if node_health_facts.network.nameservers is none
|
||||
else ((ns | length) | string ~ ' nameserver(s) — IPv4: '
|
||||
~ (ns4 | join(', ') | default('none', true))
|
||||
~ '; IPv6: ' ~ (ns6 | join(', ') | default('none', true))
|
||||
~ (' (read via ' ~ ns_src ~ ')' if ns_src else '')
|
||||
~ ('. Only ' ~ (ns4 | length) | string ~ ' IPv4 upstream: a single IPv4 upstream
|
||||
already broke containerd image pulls and CoreDNS forwarding on dev, and the
|
||||
fix is stranded on fix/node-dns-resilient-upstream (DECISIONS-LOG
|
||||
2026-08-31, finding 5). IPv6 upstreams only compensate if the node has
|
||||
working IPv6 egress, which this check cannot verify.'
|
||||
if (ns4 | length) < (node_health_thresholds.min_nameservers | int) else '') }}
|
||||
already broke containerd image pulls and CoreDNS forwarding on dev. The fix
|
||||
is the `dns`-tagged block in roles/node_baseline — set this environment
|
||||
node_baseline_dns_servers (role defaults for dev, group_vars/k3s_prod for
|
||||
prod) and re-run node-baseline.yml --tags dns. IPv6 upstreams only compensate
|
||||
if the node has working IPv6 egress, which this check cannot verify.'
|
||||
if (ns4 | length) < (node_health_thresholds.min_nameservers | int) else '')) }}
|
||||
|
||||
# --- OS / firmware ------------------------------------------------------------------
|
||||
- name: Assess — pending reboot
|
||||
|
||||
@@ -124,7 +124,14 @@
|
||||
rx {{ n.statistics.rx_packets | default('?') }} pkts / errors {{ n.statistics.rx_errors | default('?') }} / dropped {{ n.statistics.rx_dropped | default('?') }}
|
||||
tx {{ n.statistics.tx_packets | default('?') }} pkts / errors {{ n.statistics.tx_errors | default('?') }} / dropped {{ n.statistics.tx_dropped | default('?') }}
|
||||
{% endfor %}
|
||||
{% if node_health_facts.network.nameservers is none %}
|
||||
nameservers NOT READ — {{ node_health_facts.network.resolver_unavailable_reason | default('the collector recorded no reason', true) | regex_replace('\s+', ' ') | trim | wordwrap(66) | indent(13) }}
|
||||
{% else %}
|
||||
nameservers {{ node_health_facts.network.nameservers | join(', ') | default('none configured', true) }}
|
||||
{% if node_health_facts.network.resolver_stub | default([], true) | length > 0 %}
|
||||
(/etc/resolv.conf holds only the systemd-resolved stub {{ node_health_facts.network.resolver_stub | join(', ') }}; the servers above are the real upstreams behind it)
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
NOT READ (and why) — these are UNKNOWNS, not passes
|
||||
|
||||
+10
-1
@@ -159,8 +159,17 @@ resource "proxmox_virtual_environment_vm" "node" {
|
||||
}
|
||||
}
|
||||
|
||||
# DNS resilience (dev only — tofu provisions the dev VMs; prod is bare metal with no
|
||||
# cloud-init). A SINGLE upstream was a hard SPOF: PiHole (var.dns_server) answers the
|
||||
# local jackson.lan zone but its external recursion flaps, which broke containerd image
|
||||
# pulls and CoreDNS forwarding. PiHole stays FIRST — the local zone and network
|
||||
# ad-blocking are unchanged — and the pfSense gateway (var.network_gateway) is added
|
||||
# purely as a fallback for external recursion. Order is the policy, not a preference.
|
||||
# This keeps a freshly built node's cloud-init in parity with what the Ansible
|
||||
# node_baseline role's `dns`-tagged block configures on an already-running node
|
||||
# (node_baseline_dns_servers), so a rebuild does not regress to one resolver.
|
||||
dns {
|
||||
servers = [var.dns_server]
|
||||
servers = [var.dns_server, var.network_gateway]
|
||||
}
|
||||
|
||||
user_account {
|
||||
|
||||
Reference in new issue
Block a user