diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index a2e8386..d24d82b 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -125,10 +125,37 @@ Consequences to settle before imaging nodes 2 and 3: (`GATEWAY_VIP` `.53` = first of `.53-.60`). - [ ] **Reserve all of `192.168.2.9` and `.20-.30` outside any DHCP range** on the prod LAN before the build, and confirm nothing else on the network already answers on them. -- [ ] **Resolver: configure at least two upstreams.** Node 1 currently has a single IPv4 nameserver - (`192.168.2.1`). A single upstream is a known SPOF that has already bitten this project — - it broke containerd image pulls and CoreDNS forwarding on dev. A fix exists but is **stranded - unmerged** on `fix/node-dns-resilient-upstream`; land it, then apply the same pattern here. +- [ ] **Resolver: SECOND UPSTREAM IS AN OPEN DECISION — blocks `node-baseline.yml --tags dns`.** + Node 1 has a single IPv4 nameserver, `192.168.2.1`, which is also the LAN gateway. A single + upstream is a known SPOF that has already bitten this project — it broke containerd image + pulls and CoreDNS forwarding on dev. + + The MECHANISM is now in the repo and needs no further work: the `dns`-tagged block in + `roles/node_baseline` detects whether a node uses systemd-resolved (dev: writes + `/etc/systemd/resolved.conf.d/99-k3s-dns.conf`, restarts resolved) or a directly-managed + `/etc/resolv.conf` (prod's shape: writes the file, refusing if resolvconf/NetworkManager + owns it), and configures the ordered upstream list either way. The earlier + `fix/node-dns-resilient-upstream` branch was **dropped, not merged** — it wrote a netplan + drop-in and ran `netplan apply`, and netplan is not installed on these Debian 13 trixie + nodes, so it would have broken `node-baseline.yml` on every host. + + What is still MISSING is the VALUE. `group_vars/k3s_prod/vars.yml` sets + `node_baseline_dns_servers` to `192.168.2.1` followed by the deliberately invalid + `BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4`, and the role's assert + stops the play on it before writing anything. Choosing the second upstream is a **network + decision, not a default the role may take**: a public resolver (`8.8.8.8`, `1.1.1.1`) would + resolve reliably but would also bypass any local zone and any filtering the gateway applies. + Decide between — and record the choice HERE, in the same change that edits the group_vars: + - a second on-LAN resolver on `192.168.2.x`; + - the dev-side PiHole `192.168.1.18`, **if** prod can actually route to it (unverified — + prod sits on the workstation side of the pfSense boundary); + - an explicit, written-down acceptance of a public resolver as fallback-only. + + Also decide whether node 1's current IPv6 nameservers (`2001:4860:4860::8888` / `::8844`) + are kept: the direct-file write path replaces `/etc/resolv.conf` with exactly the listed + servers, so keeping them means listing them explicitly. + `node_baseline_dns_search` is `[]` (no search domain, node 1's behaviour today) until prod's + local domain is decided. - [ ] Confirm the prod LAN's relationship to the dev LAN (`192.168.1.x`) and to the pfSense NAT — prod on `192.168.2.x` sits on the *workstation* side of the boundary dev lives behind. diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index 6534690..4071941 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -9,8 +9,9 @@ # Pairs with vault.yml in this same directory (ansible-vault-encrypted, operator- # created — see README.md here) for the one secret prod needs: ansible_become_password. # -# PHASE 7 IS PREP ONLY. Four vars below (kube_vip_address, cluster_controller_api_server, -# node_baseline_longhorn_device, node_baseline_etcd_device) are deliberately BROKEN +# PHASE 7 IS PREP ONLY. Five vars below (kube_vip_address, cluster_controller_api_server, +# node_baseline_longhorn_device, node_baseline_etcd_device, and the SECOND entry of +# node_baseline_dns_servers) are deliberately BROKEN # PLACEHOLDERS, not real values — see their comments. Do not "fix" them with a guess; # follow the referenced section of PROD-PROVISIONING.md, record the decision there, # and update this file in the same change. @@ -104,3 +105,44 @@ cluster_controller_api_server: "https://{{ kube_vip_address }}:6443" # (Longhorn replica placement assumes symmetry) before any of this is used for real. node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2" node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2" + +# --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) ----- +# roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1 +# pfSense) — addresses on a network prod cannot even reach. Inherited verbatim they would +# point every prod node at a PiHole that does not answer it, so they are overridden here, +# in the same seam as kube_vip_address / k3s_api_host / cluster_kubeconfig_name above. +# (NOT clusters/base/cluster-vars.yaml: that ConfigMap is consumed by Flux +# postBuild.substituteFrom for platform/ and apps/ manifests, and no Ansible code reads it.) +# +# WHAT IS DECIDED: 192.168.2.1 — node 1's current sole nameserver, and the LAN gateway. +# It stays FIRST, so whatever local zone it serves keeps working (the dev outage taught +# that ordering is policy, not preference — see the role's defaults). +# +# WHAT IS NOT: the SECOND upstream. 192.168.2.1 is presently gateway AND only IPv4 +# resolver, and there is no second address on this LAN that is known to answer. Picking +# one is a NETWORK decision, not a default this role may take: a public resolver +# (8.8.8.8, 1.1.1.1) would resolve reliably but would ALSO bypass any local zone and any +# network-level filtering the gateway applies — precisely the bypass the dev fix refused +# to make. Candidates the operator must choose between and record in +# infra/PROD-PROVISIONING.md §4: a second on-LAN resolver, the dev-side PiHole IF prod can +# route to 192.168.1.18, or an explicit, written-down decision to accept a public resolver +# as fallback-only. +# +# Deliberately present-but-broken, exactly like node_baseline_longhorn_device above: +# leaving the list at one entry, or unset, would either configure the very SPOF this exists +# to remove or silently inherit dev's PiHole. This value can never parse as an address, and +# the `dns`-tagged assert in roles/node_baseline stops the play on the string BLOCKED +# before a single file is written. DO NOT replace it with a guess — replace it only once +# §4 is decided, and record the chosen resolver there in the SAME change. +# +# Note the two Google IPv6 servers currently in node 1's /etc/resolv.conf +# (2001:4860:4860::8888/::8844) are NOT carried over: the direct-file write path replaces +# the file with exactly this list, so if they are to be kept they must be listed here +# explicitly — as part of that same recorded decision. +node_baseline_dns_servers: + - "192.168.2.1" + - "BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4" +# Prod's local search domain is not a decided thing (dev's is jackson.lan, PiHole's zone). +# An EMPTY list is the role's supported way to say "configure no search domain", which is +# node 1's current behaviour — the conservative non-decision, not a missing value. +node_baseline_dns_search: [] diff --git a/infra/ansible/roles/node_baseline/defaults/main.yml b/infra/ansible/roles/node_baseline/defaults/main.yml index 1517d37..3c527ae 100644 --- a/infra/ansible/roles/node_baseline/defaults/main.yml +++ b/infra/ansible/roles/node_baseline/defaults/main.yml @@ -34,3 +34,40 @@ node_baseline_longhorn_fs_label: longhorn node_baseline_etcd_device: /dev/vdc node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db node_baseline_etcd_fs_label: k3s-etcd + +# --- DNS resilience (R-DNS-01 / R-DNS-03) ------------------------------------------- +# These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml +# (the same seam kube_vip_address / k3s_api_host / cluster_kubeconfig_name use) — NOT in +# clusters/base/cluster-vars.yaml, which is a Flux postBuild ConfigMap that no Ansible +# code reads. +# +# WHY: the nodes shipped with ONE upstream resolver. PiHole answers the local jackson.lan +# zone instantly but its external recursion flaps (measured 0/25 for registry-1.docker.io / +# quay.io / ghcr.io while gitea.jackson.lan resolved 5/5), and one upstream is a hard SPOF: +# it broke containerd image pulls ("lookup registry-1.docker.io: Try again") AND CoreDNS, +# which forwards to /etc/resolv.conf and flooded its log with i/o timeouts to 192.168.1.18. +# +# ORDER IS THE POLICY. This is a RESILIENCE fix, not a bypass: the first entry stays PiHole, +# so *.jackson.lan and network-wide ad-blocking keep working exactly as before, and the +# second is only ever consulted when the first fails. Both write paths preserve that order +# (systemd-resolved tries the servers of one scope in order; glibc walks `nameserver` lines +# in order). Never put a public resolver first, and never enable `rotate` — either would +# silently take the local zone and the ad-block policy out of the path. +# +# Keep in parity with infra/tofu (var.dns_server, var.network_gateway), which sets the same +# two servers via cloud-init so a freshly rebuilt dev node boots with this list already. +node_baseline_dns_servers: + - 192.168.1.18 # PiHole — local jackson.lan zone + ad-blocking. PRIMARY, keep first. + - 192.168.1.1 # pfSense gateway — reliable external recursion. FALLBACK only. +# Search domain(s). An EMPTY list means "configure no search domain" and is a legitimate +# value (prod uses it — its local domain is not a decided thing), not a missing one. +node_baseline_dns_search: + - jackson.lan +# resolv.conf `options` for the DIRECT-file mechanism only (systemd-resolved does its own +# per-server failover timing and ignores these). glibc's defaults are timeout:5 attempts:2, +# i.e. up to 20s of stalled lookups before it reaches the second server — long enough for +# containerd to give up on a pull. These shorten the failover, and deliberately do NOT +# include `rotate`, which would round-robin the servers and break PiHole-first ordering. +node_baseline_dns_options: + - timeout:2 + - attempts:2 diff --git a/infra/ansible/roles/node_baseline/handlers/main.yml b/infra/ansible/roles/node_baseline/handlers/main.yml index 15d80f0..5bd7788 100644 --- a/infra/ansible/roles/node_baseline/handlers/main.yml +++ b/infra/ansible/roles/node_baseline/handlers/main.yml @@ -2,3 +2,12 @@ - name: Reload sysctl ansible.builtin.command: sysctl --system changed_when: true + +# Restarting systemd-resolved re-reads /etc/systemd/resolved.conf.d/ and re-points the +# 127.0.0.53 stub at the new upstream list. It does NOT touch link addressing or routes, +# so the SSH/NAT session Ansible is running over is unaffected. Only fires when the +# `dns`-tagged drop-in above actually changed, so a re-run of the role is a no-op. +- name: Restart systemd-resolved + ansible.builtin.systemd: + name: systemd-resolved + state: restarted diff --git a/infra/ansible/roles/node_baseline/tasks/main.yml b/infra/ansible/roles/node_baseline/tasks/main.yml index 15fd113..fddf95d 100644 --- a/infra/ansible/roles/node_baseline/tasks/main.yml +++ b/infra/ansible/roles/node_baseline/tasks/main.yml @@ -1,6 +1,6 @@ --- -# TAGS — every task here carries one of `packages`, `system` or `disks`. An untagged run -# is unchanged (all three run, in this order). The split exists because the PACKAGE floor +# TAGS — every task here carries one of `packages`, `system`, `disks` or `dns`. An untagged +# run is unchanged (all four run, in this order). The split exists because the PACKAGE floor # and the DATA-DISK work have different readiness: packages are safe on any node from # first contact, while the disk tasks need block devices that a new environment may not # have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked @@ -9,7 +9,9 @@ # ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages # Skipping is the guard: with `--tags packages` no `disks`-tagged task is selected, so a # BLOCKED-pending-disk-decision device path is never even templated, let alone handed to -# mkfs. A FULL run against such a node stops at the assert below instead. +# mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block +# at the bottom of this file) works the same way and guards its own BLOCKED placeholder: +# ansible-playbook playbooks/node-baseline.yml --tags dns - name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs) ansible.builtin.apt: @@ -154,3 +156,225 @@ regexp: '^([^#].*\sswap\s.*)$' replace: '# \1 # disabled for k3s/kubelet' tags: [system] + +# ===================================================================================== +# DNS RESILIENCE — tag `dns` (R-DNS-01 / R-DNS-03) +# ===================================================================================== +# Root cause (verified on dev): the nodes carried a SINGLE upstream resolver. PiHole +# (192.168.1.18) answers `*.jackson.lan` instantly but its external recursion flaps — +# 0/25 for registry-1.docker.io / quay.io / ghcr.io while gitea.jackson.lan resolved 5/5. +# One upstream is a hard SPOF: containerd image pulls died with "lookup +# registry-1.docker.io: Try again", and CoreDNS (`forward . /etc/resolv.conf`) flooded its +# log with "i/o timeout" to 192.168.1.18. The fix is ADDITIVE — PiHole stays first so the +# local zone and the ad-blocking policy are untouched; a second upstream is only consulted +# when the first fails. See defaults/main.yml for the ordering rule. +# +# WHY THIS DETECTS INSTEAD OF ASSUMING. An earlier attempt (fix/node-dns-resilient-upstream, +# dropped) wrote /etc/netplan/99-k3s-dns.yaml and ran `netplan apply`. netplan is NOT +# installed on these nodes — they run the Debian 13 trixie generic cloud image +# (infra/tofu/variables.tf) and netplan is not in node_baseline_packages — so that handler +# would have failed command-not-found and broken node-baseline.yml on EVERY node. Worse, +# the two environments do not even share a resolver stack: dev's /etc/resolv.conf is the +# systemd-resolved stub (127.0.0.53) while prod's is a directly-managed file listing +# 192.168.2.1 (it has no cloud-init at all — bare metal). So the mechanism is DETECTED per +# node and the matching seam is written. Adding a third mechanism means adding a branch +# here, not rewriting the role. +# +# All of it is `dns`-tagged, so `--tags packages` (the prod package-floor escape hatch +# documented at the top of this file) still installs nothing but packages. + +- name: Determine whether systemd-resolved is running on this node + ansible.builtin.command: systemctl is-active systemd-resolved + register: node_baseline_resolved_active + changed_when: false + failed_when: false + check_mode: false + tags: [dns] + +- name: Determine whether NetworkManager is running (it rewrites /etc/resolv.conf) + ansible.builtin.command: systemctl is-active NetworkManager + register: node_baseline_nm_active + changed_when: false + failed_when: false + check_mode: false + tags: [dns] + +# follow: false — we need to know whether /etc/resolv.conf IS a symlink and where it +# points, which is the strongest single signal of who owns the file. +- name: Inspect /etc/resolv.conf itself (symlink or real file, and its target) + ansible.builtin.stat: + path: /etc/resolv.conf + follow: false + register: node_baseline_resolv_stat + tags: [dns] + +- name: Read the current /etc/resolv.conf contents + ansible.builtin.slurp: + src: /etc/resolv.conf + register: node_baseline_resolv_slurp + failed_when: false + tags: [dns] + +# resolvconf/openresolv keeps the authoritative copy under here and regenerates +# /etc/resolv.conf from it, so writing /etc/resolv.conf directly would be silently undone. +- name: Detect a resolvconf/openresolv installation that would clobber /etc/resolv.conf + ansible.builtin.stat: + path: /etc/resolvconf/resolv.conf.d + register: node_baseline_resolvconf_dir + tags: [dns] + +# THE DETECTION. Three outcomes, in priority order: +# systemd-resolved — resolved is running AND it owns /etc/resolv.conf, proven either by +# the symlink pointing into /run/systemd/resolve/ or by the file containing only +# resolved's loopback stub (127.0.0.53 / 127.0.0.54). Writing a resolved drop-in is +# then the ONLY thing that changes real behaviour; writing resolv.conf would be +# overwritten by resolved (or, worse, land in /run and vanish at reboot). +# contended — nobody's stub, but resolvconf or NetworkManager is present/running and +# would regenerate the file underneath us. We do NOT write. The role fails loudly +# below rather than leaving an operator believing a change stuck when it did not. +# resolv-conf-file — a plain, directly-managed /etc/resolv.conf (prod's shape today). +# Safe to own outright. +# resolved running but NOT owning resolv.conf correctly falls through to the file branch: +# in that state the drop-in would change nothing glibc can see. +- name: Detect which resolver mechanism this node actually uses + ansible.builtin.set_fact: + node_baseline_dns_mechanism: >- + {{ + 'systemd-resolved' if (node_baseline_resolved_running and node_baseline_resolved_owns_resolv) + else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present) + else 'resolv-conf-file') + }} + vars: + node_baseline_resolved_running: "{{ (node_baseline_resolved_active.stdout | default('') | trim) == 'active' }}" + node_baseline_nm_running: "{{ (node_baseline_nm_active.stdout | default('') | trim) == 'active' }}" + node_baseline_resolvconf_present: "{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}" + node_baseline_resolv_text: "{{ (node_baseline_resolv_slurp.content | default('') | b64decode) }}" + node_baseline_resolv_ns: "{{ node_baseline_resolv_text | regex_findall('(?m)^\\s*nameserver\\s+(\\S+)') }}" + node_baseline_resolved_owns_resolv: >- + {{ ('/run/systemd/resolve/' in (node_baseline_resolv_stat.stat.lnk_source | default(''))) + or ((node_baseline_resolv_ns | length) > 0 + and (node_baseline_resolv_ns | reject('in', ['127.0.0.53', '127.0.0.54']) | list | length) == 0) }} + tags: [dns] + +- name: Report the detected resolver mechanism + ansible.builtin.debug: + msg: >- + resolver mechanism = {{ node_baseline_dns_mechanism }} + (systemd-resolved is-active={{ node_baseline_resolved_active.stdout | default('') | trim }}, + NetworkManager is-active={{ node_baseline_nm_active.stdout | default('') | trim }}, + /etc/resolv.conf islnk={{ node_baseline_resolv_stat.stat.islnk | default(false) }} + target={{ node_baseline_resolv_stat.stat.lnk_source | default('(not a symlink)') }}, + resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }}) + tags: [dns] + +# Same shape, and the same reason, as the data-disk assert above: this environment's +# upstream list is an OPERATOR DECISION, and a placeholder that fails LOUDLY beats both a +# guess and a silent inheritance of dev's PiHole. group_vars/k3s_prod/vars.yml deliberately +# carries a BLOCKED second entry (see its comments and PROD-PROVISIONING.md §4): prod's +# 192.168.2.1 is currently gateway AND only upstream, and choosing what sits behind it is a +# network decision this role must not make. Running against prod stops HERE, before +# anything is written. +- name: Assert this environment's DNS upstream list has actually been decided + ansible.builtin.assert: + that: + - node_baseline_dns_servers | length >= 2 + - "'BLOCKED' not in (node_baseline_dns_servers | join(' '))" + fail_msg: >- + node_baseline_dns_servers is not a decided, redundant upstream list for this host's + environment (got: {{ node_baseline_dns_servers | join(', ') }}). A single upstream is + the exact SPOF this task exists to remove — it already broke containerd image pulls + and CoreDNS forwarding on dev — so the role will not configure one, and it will NOT + guess a second server: a public resolver would bypass the local zone and the + ad-blocking policy, which is a decision, not a default. Decide the second upstream, + record it in infra/PROD-PROVISIONING.md §4, and set node_baseline_dns_servers in + inventory/group_vars/k3s_prod/vars.yml in the SAME change. To install only the + package floor meanwhile, re-run with `--tags packages`, which skips every DNS task. + quiet: true + tags: [dns] + +# The `contended` outcome. Not a warning: an operator who is told "changed" while another +# daemon quietly reverts the file is worse off than one who is told nothing happened. +- name: Refuse to manage /etc/resolv.conf while another resolver manager owns it + ansible.builtin.fail: + msg: >- + /etc/resolv.conf on this node is not the systemd-resolved stub, but + resolvconf/openresolv (/etc/resolvconf/resolv.conf.d present: + {{ node_baseline_resolvconf_dir.stat.exists | default(false) }}) or NetworkManager + (is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}) is in play and + would regenerate the file underneath this role. Writing it here would look like it + worked and silently revert. Add the matching branch to roles/node_baseline (the + `dns`-tagged block) for whichever manager this node really uses — do not force it. + when: node_baseline_dns_mechanism == 'contended' + tags: [dns] + +# --- Mechanism A: systemd-resolved (dev's shape today) ------------------------------- +# A drop-in, never an edit of /etc/systemd/resolved.conf, so a package upgrade of that file +# never conflicts. `DNS=` is one ORDERED list inside a single resolved scope: resolved uses +# the first server and only rotates to the next when it fails, which is exactly the +# PiHole-primary/fallback behaviour wanted — it is NOT a race between two servers, so a +# gateway NXDOMAIN can never beat PiHole to a jackson.lan answer while PiHole is healthy. +- name: Ensure the systemd-resolved drop-in directory exists + ansible.builtin.file: + path: /etc/systemd/resolved.conf.d + state: directory + owner: root + group: root + mode: "0755" + when: node_baseline_dns_mechanism == 'systemd-resolved' + tags: [dns] + +- name: Configure resilient DNS upstreams via a systemd-resolved drop-in + ansible.builtin.copy: + dest: /etc/systemd/resolved.conf.d/99-k3s-dns.conf + owner: root + group: root + mode: "0644" + content: | + # Managed by the Ansible node_baseline role (tags: dns) — do not edit by hand. + # Ordered upstream list: the FIRST server is the local-zone/ad-blocking resolver and + # stays authoritative; the rest exist only so external resolution (containerd image + # pulls, CoreDNS `forward . /etc/resolv.conf`) survives its recursion flapping. + [Resolve] + DNS={{ node_baseline_dns_servers | join(' ') }} + Domains={{ node_baseline_dns_search | join(' ') }} + # Reset to empty deliberately. systemd ships compiled-in public fallback resolvers + # (Cloudflare/Google) that resolved would reach for if the list above ever came up + # empty — which would silently bypass BOTH the local zone and the network's + # ad-blocking. This fix adds redundancy; it never adds a bypass. + FallbackDNS= + notify: Restart systemd-resolved + when: node_baseline_dns_mechanism == 'systemd-resolved' + tags: [dns] + +# --- Mechanism B: a directly-managed /etc/resolv.conf (prod's shape today) ----------- +# Only reached when nothing else owns the file (the `contended` branch above already +# stopped the play otherwise). glibc walks `nameserver` lines in order and falls through +# on timeout, so ordering carries the same policy as the resolved list. +# NOTE this REPLACES the file: any nameserver currently there and NOT in +# node_baseline_dns_servers is dropped. That is the point — the list in group_vars is the +# whole declared truth — so an environment that wants its existing IPv6 upstreams kept +# must list them there too. `backup: true` leaves the pre-role copy beside it the first +# time it changes, because this file is load-bearing for reaching the node at all. +- name: Configure resilient DNS upstreams directly in /etc/resolv.conf + ansible.builtin.copy: + dest: /etc/resolv.conf + owner: root + group: root + mode: "0644" + backup: true + content: | + # Managed by the Ansible node_baseline role (tags: dns) — do not edit by hand. + # Ordered upstream list: the FIRST server is the local-zone resolver and stays + # authoritative; the rest exist only so external resolution (containerd image pulls, + # CoreDNS `forward . /etc/resolv.conf`) survives its recursion flapping. + {% if node_baseline_dns_search | length > 0 %} + search {{ node_baseline_dns_search | join(' ') }} + {% endif %} + {% for server in node_baseline_dns_servers %} + nameserver {{ server }} + {% endfor %} + {% if node_baseline_dns_options | length > 0 %} + options {{ node_baseline_dns_options | join(' ') }} + {% endif %} + when: node_baseline_dns_mechanism == 'resolv-conf-file' + tags: [dns] diff --git a/infra/ansible/roles/node_health/defaults/main.yml b/infra/ansible/roles/node_health/defaults/main.yml index 65cc7c7..406412c 100644 --- a/infra/ansible/roles/node_health/defaults/main.yml +++ b/infra/ansible/roles/node_health/defaults/main.yml @@ -56,6 +56,9 @@ node_health_thresholds: nic_min_speed_mbps: 1000 # A single upstream resolver is a known SPOF on this project — it broke containerd # image pulls and CoreDNS forwarding on dev (DECISIONS-LOG 2026-08-31, finding 5). + # Counted against the node's REAL upstreams: where systemd-resolved is in use the + # collector looks behind the 127.0.0.53 stub rather than counting it as one server. + # The fix side of this is the `dns`-tagged block in roles/node_baseline. min_nameservers: 2 # SMART (only evaluated when smartctl actually ran) smart_reallocated_warn: 1 diff --git a/infra/ansible/roles/node_health/files/node_health_collect.py b/infra/ansible/roles/node_health/files/node_health_collect.py index f627861..37adf25 100644 --- a/infra/ansible/roles/node_health/files/node_health_collect.py +++ b/infra/ansible/roles/node_health/files/node_health_collect.py @@ -4,8 +4,10 @@ # # Run by roles/node_health via ansible.builtin.script. Prints ONE JSON document on # stdout and nothing else. It is strictly READ-ONLY: it opens files under /sys, /proc -# and /etc, and shells out only to `lsblk` (a read-only query). It never writes to the -# node, never installs anything, and never needs root. +# and /etc, and shells out only to read-only queries (`lsblk`, `lscpu`, and `resolvectl dns` +# — the last only when /etc/resolv.conf turns out to be the systemd-resolved stub and the +# real upstreams have to be asked of resolved). It never writes to the node, never installs +# anything, and never needs root. # # WHY A COLLECTOR SCRIPT AND NOT 40 ANSIBLE TASKS # Everything here is "read a sysfs file that may or may not exist and may or may not @@ -563,9 +565,113 @@ def collect_network(): # Resolver config is a known SPOF class on this project (a single upstream broke # containerd pulls and CoreDNS forwarding on dev). Cheap to read here, so the health # report can flag it on prod too. - resolv = read_text("/etc/resolv.conf") or "" - nameservers = re.findall(r"^\s*nameserver\s+(\S+)", resolv, re.M) - return {"interfaces": interfaces, "nameservers": nameservers} + resolvers = collect_resolvers() + return { + "interfaces": interfaces, + # List of REAL upstreams, or None when they could not be determined. NEVER the + # loopback stub, and never [] standing in for "unknown". + "nameservers": resolvers["nameservers"], + "resolver_source": resolvers["source"], + "resolver_stub": resolvers["stub"], + "resolver_unavailable_reason": resolvers["unavailable_reason"], + } + + +# systemd-resolved's loopback listeners. 127.0.0.53 is the stub (the address written into +# /etc/resolv.conf); 127.0.0.54 is the "bypass"/proxy listener. Neither is an UPSTREAM — +# seeing one in /etc/resolv.conf means the real servers live inside resolved. +RESOLVED_STUB_IPS = ("127.0.0.53", "127.0.0.54") + + +def _resolved_upstreams(): + """Real upstreams from systemd-resolved. Returns (servers, reason_unreadable). + + `resolvectl dns` prints one line per scope ("Global:", "Link 2 (eth0): 1.2.3.4 ..."), + which is far easier to parse reliably than the wrapped columns of `resolvectl status`. + """ + rc, out, err = run(["resolvectl", "dns", "--no-pager"]) + if rc is None: + return None, ( + "/etc/resolv.conf holds only the systemd-resolved loopback stub, and the real " + "upstreams behind it could not be read: resolvectl is not installed on this node" + ) + if rc != 0: + detail = (err.strip() or out.strip() or "no output").splitlines()[0] + return None, ( + "/etc/resolv.conf holds only the systemd-resolved loopback stub, and " + "`resolvectl dns` exited %s so the real upstreams could not be read: %s" + % (rc, detail) + ) + servers = [] + for line in out.splitlines(): + # Split on the FIRST colon only: everything after it is the server list, and IPv6 + # addresses contain colons of their own. + _, sep, rest = line.partition(":") + if not sep: + continue + for token in rest.split(): + if token not in servers: + servers.append(token) + if not servers: + return None, ( + "/etc/resolv.conf holds only the systemd-resolved loopback stub and " + "`resolvectl dns` reported NO upstream servers on any scope — resolved is " + "running with an empty server list, so this node has no usable upstream" + ) + return servers, None + + +def collect_resolvers(): + """The node's real DNS upstreams, plus how they were determined. + + /etc/resolv.conf is NOT always the answer. Where systemd-resolved is in use (dev's + nodes today) the file contains only resolved's loopback stub, 127.0.0.53 — reading it + literally reports "1 nameserver" forever, which is a permanent FALSE single-upstream + WARN that no amount of correct configuration can clear. When the stub is detected the + real list is asked of resolved instead. + + HONESTY DISCIPLINE, unchanged from the rest of this collector: if the stub is present + and the real list cannot be read, "nameservers" is None with the reason recorded — NOT + the stub address, NOT an empty list, and NOT a guess. An unknown must stay an unknown; + the assess step turns it into a SKIP, never a PASS or a WARN. + + LIMIT worth knowing: only systemd-resolved's stub is special-cased. Some OTHER local + forwarder on 127.0.0.1 (dnsmasq, unbound) is reported exactly as it appears in + resolv.conf, because nothing generic can enumerate an arbitrary forwarder's upstreams. + """ + path = "/etc/resolv.conf" + # optional=False: a MISSING /etc/resolv.conf is itself a finding worth naming. + resolv = read_text(path, optional=False) + if resolv is None: + return { + "nameservers": None, + "source": None, + "stub": [], + "unavailable_reason": "/etc/resolv.conf could not be read (see the unreadable list)", + } + listed = re.findall(r"^\s*nameserver\s+(\S+)", resolv, re.M) + stub = [n for n in listed if n in RESOLVED_STUB_IPS] + # Only when EVERY listed server is a stub is the file a pure indirection. A mixed file + # (stub plus a real address) is reported as written — that is what the resolver library + # will actually use. + if listed and len(stub) == len(listed): + servers, reason = _resolved_upstreams() + if servers is None: + UNREADABLE.append({"path": "resolvectl dns", "reason": reason}) + return { + "nameservers": None, + "source": None, + "stub": listed, + "unavailable_reason": reason, + } + return { + "nameservers": servers, + "source": "resolvectl dns (/etc/resolv.conf holds only the systemd-resolved " + "stub %s)" % ", ".join(listed), + "stub": listed, + "unavailable_reason": None, + } + return {"nameservers": listed, "source": path, "stub": stub, "unavailable_reason": None} # --- OS / kernel / uptime ----------------------------------------------------------- diff --git a/infra/ansible/roles/node_health/tasks/assess.yml b/infra/ansible/roles/node_health/tasks/assess.yml index 8c0dd96..d5d3237 100644 --- a/infra/ansible/roles/node_health/tasks/assess.yml +++ b/infra/ansible/roles/node_health/tasks/assess.yml @@ -435,32 +435,50 @@ {{ err_total | string ~ ' error(s), ' ~ drop_total | string ~ ' drop(s) across ' ~ (up | map(attribute='name') | join(', ')) }} +# The collector reports the node's REAL upstreams: where /etc/resolv.conf holds only +# systemd-resolved's 127.0.0.53 stub (dev's shape) it asks resolved for the servers behind +# it, because counting the stub would be a permanent FALSE single-upstream WARN that no +# correct configuration could ever clear. When that lookup fails it reports null WITH a +# reason rather than a number — so the check SKIPs. A skip is honest; a WARN derived from +# an unknown is a fabricated verdict, and a PASS would be worse. - name: Assess — DNS resolver redundancy ansible.builtin.set_fact: node_health_checks: "{{ node_health_checks + [{ 'id': 'network.resolvers', 'title': 'Configured DNS upstreams (single upstream is a known SPOF here)', - 'status': ('PASS' if (ns4 | length) >= (node_health_thresholds.min_nameservers | int) else 'WARN'), + 'status': ('SKIP' if node_health_facts.network.nameservers is none + else ('PASS' if (ns4 | length) >= (node_health_thresholds.min_nameservers | int) + else 'WARN')), 'detail': ns_detail }] }}" vars: ansible_become: false - ns: "{{ node_health_facts.network.nameservers }}" + # `| default([], true)` only ever fires on the null (unreadable) path, where the + # counts below are unused because the status is already SKIP. + ns: "{{ node_health_facts.network.nameservers | default([], true) }}" # Counted PER FAMILY on purpose. The outage this check exists for was a single IPv4 # upstream; IPv6 upstreams do not cover for it unless the node actually has working # IPv6 egress, which this role cannot prove from sysfs alone. So the verdict keys off # the IPv4 count and the detail shows the full picture. ns4: "{{ ns | select('search', '^[0-9.]+$') | list }}" ns6: "{{ ns | reject('search', '^[0-9.]+$') | list }}" + # Where the numbers came from — resolv.conf directly, or resolved behind the stub. + ns_src: "{{ node_health_facts.network.resolver_source | default('', true) }}" ns_detail: >- - {{ (ns | length) | string ~ ' nameserver(s) — IPv4: ' + {{ ('not read — ' ~ (node_health_facts.network.resolver_unavailable_reason + | default('the collector recorded no reason', true)) + ~ '. The upstream count is UNKNOWN here — not zero, and not one.') + if node_health_facts.network.nameservers is none + else ((ns | length) | string ~ ' nameserver(s) — IPv4: ' ~ (ns4 | join(', ') | default('none', true)) ~ '; IPv6: ' ~ (ns6 | join(', ') | default('none', true)) + ~ (' (read via ' ~ ns_src ~ ')' if ns_src else '') ~ ('. Only ' ~ (ns4 | length) | string ~ ' IPv4 upstream: a single IPv4 upstream - already broke containerd image pulls and CoreDNS forwarding on dev, and the - fix is stranded on fix/node-dns-resilient-upstream (DECISIONS-LOG - 2026-08-31, finding 5). IPv6 upstreams only compensate if the node has - working IPv6 egress, which this check cannot verify.' - if (ns4 | length) < (node_health_thresholds.min_nameservers | int) else '') }} + already broke containerd image pulls and CoreDNS forwarding on dev. The fix + is the `dns`-tagged block in roles/node_baseline — set this environment + node_baseline_dns_servers (role defaults for dev, group_vars/k3s_prod for + prod) and re-run node-baseline.yml --tags dns. IPv6 upstreams only compensate + if the node has working IPv6 egress, which this check cannot verify.' + if (ns4 | length) < (node_health_thresholds.min_nameservers | int) else '')) }} # --- OS / firmware ------------------------------------------------------------------ - name: Assess — pending reboot diff --git a/infra/ansible/roles/node_health/templates/summary.txt.j2 b/infra/ansible/roles/node_health/templates/summary.txt.j2 index 248f98b..6babeb8 100644 --- a/infra/ansible/roles/node_health/templates/summary.txt.j2 +++ b/infra/ansible/roles/node_health/templates/summary.txt.j2 @@ -124,7 +124,14 @@ rx {{ n.statistics.rx_packets | default('?') }} pkts / errors {{ n.statistics.rx_errors | default('?') }} / dropped {{ n.statistics.rx_dropped | default('?') }} tx {{ n.statistics.tx_packets | default('?') }} pkts / errors {{ n.statistics.tx_errors | default('?') }} / dropped {{ n.statistics.tx_dropped | default('?') }} {% endfor %} +{% if node_health_facts.network.nameservers is none %} + nameservers NOT READ — {{ node_health_facts.network.resolver_unavailable_reason | default('the collector recorded no reason', true) | regex_replace('\s+', ' ') | trim | wordwrap(66) | indent(13) }} +{% else %} nameservers {{ node_health_facts.network.nameservers | join(', ') | default('none configured', true) }} +{% if node_health_facts.network.resolver_stub | default([], true) | length > 0 %} + (/etc/resolv.conf holds only the systemd-resolved stub {{ node_health_facts.network.resolver_stub | join(', ') }}; the servers above are the real upstreams behind it) +{% endif %} +{% endif %} -------------------------------------------------------------------------------- NOT READ (and why) — these are UNKNOWNS, not passes diff --git a/infra/tofu/main.tf b/infra/tofu/main.tf index 1b502bb..0487b0b 100644 --- a/infra/tofu/main.tf +++ b/infra/tofu/main.tf @@ -159,8 +159,17 @@ resource "proxmox_virtual_environment_vm" "node" { } } + # DNS resilience (dev only — tofu provisions the dev VMs; prod is bare metal with no + # cloud-init). A SINGLE upstream was a hard SPOF: PiHole (var.dns_server) answers the + # local jackson.lan zone but its external recursion flaps, which broke containerd image + # pulls and CoreDNS forwarding. PiHole stays FIRST — the local zone and network + # ad-blocking are unchanged — and the pfSense gateway (var.network_gateway) is added + # purely as a fallback for external recursion. Order is the policy, not a preference. + # This keeps a freshly built node's cloud-init in parity with what the Ansible + # node_baseline role's `dns`-tagged block configures on an already-running node + # (node_baseline_dns_servers), so a rebuild does not regress to one resolver. dns { - servers = [var.dns_server] + servers = [var.dns_server, var.network_gateway] } user_account {