fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03)

Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.

The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.

MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
  * systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
    symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
    loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
    FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
    silently bypass the local zone and ad-blocking.
  * resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
    lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
    would destroy PiHole-first ordering.
  * contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
    and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.

SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.

PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.

tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.

ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.

Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.

Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -9,8 +9,9 @@
 # Pairs with vault.yml in this same directory (ansible-vault-encrypted, operator-
 # created — see README.md here) for the one secret prod needs: ansible_become_password.
 #
-# PHASE 7 IS PREP ONLY. Four vars below (kube_vip_address, cluster_controller_api_server,
-# node_baseline_longhorn_device, node_baseline_etcd_device) are deliberately BROKEN
+# PHASE 7 IS PREP ONLY. Five vars below (kube_vip_address, cluster_controller_api_server,
+# node_baseline_longhorn_device, node_baseline_etcd_device, and the SECOND entry of
+# node_baseline_dns_servers) are deliberately BROKEN
 # PLACEHOLDERS, not real values — see their comments. Do not "fix" them with a guess;
 # follow the referenced section of PROD-PROVISIONING.md, record the decision there,
 # and update this file in the same change.
@@ -104,3 +105,44 @@ cluster_controller_api_server: "https://{{ kube_vip_address }}:6443"
 # (Longhorn replica placement assumes symmetry) before any of this is used for real.
 node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
 node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
+
+# --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) -----
+# roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1
+# pfSense) — addresses on a network prod cannot even reach. Inherited verbatim they would
+# point every prod node at a PiHole that does not answer it, so they are overridden here,
+# in the same seam as kube_vip_address / k3s_api_host / cluster_kubeconfig_name above.
+# (NOT clusters/base/cluster-vars.yaml: that ConfigMap is consumed by Flux
+# postBuild.substituteFrom for platform/ and apps/ manifests, and no Ansible code reads it.)
+#
+# WHAT IS DECIDED: 192.168.2.1 — node 1's current sole nameserver, and the LAN gateway.
+# It stays FIRST, so whatever local zone it serves keeps working (the dev outage taught
+# that ordering is policy, not preference — see the role's defaults).
+#
+# WHAT IS NOT: the SECOND upstream. 192.168.2.1 is presently gateway AND only IPv4
+# resolver, and there is no second address on this LAN that is known to answer. Picking
+# one is a NETWORK decision, not a default this role may take: a public resolver
+# (8.8.8.8, 1.1.1.1) would resolve reliably but would ALSO bypass any local zone and any
+# network-level filtering the gateway applies — precisely the bypass the dev fix refused
+# to make. Candidates the operator must choose between and record in
+# infra/PROD-PROVISIONING.md §4: a second on-LAN resolver, the dev-side PiHole IF prod can
+# route to 192.168.1.18, or an explicit, written-down decision to accept a public resolver
+# as fallback-only.
+#
+# Deliberately present-but-broken, exactly like node_baseline_longhorn_device above:
+# leaving the list at one entry, or unset, would either configure the very SPOF this exists
+# to remove or silently inherit dev's PiHole. This value can never parse as an address, and
+# the `dns`-tagged assert in roles/node_baseline stops the play on the string BLOCKED
+# before a single file is written. DO NOT replace it with a guess — replace it only once
+# §4 is decided, and record the chosen resolver there in the SAME change.
+#
+# Note the two Google IPv6 servers currently in node 1's /etc/resolv.conf
+# (2001:4860:4860::8888/::8844) are NOT carried over: the direct-file write path replaces
+# the file with exactly this list, so if they are to be kept they must be listed here
+# explicitly — as part of that same recorded decision.
+node_baseline_dns_servers:
+  - "192.168.2.1"
+  - "BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4"
+# Prod's local search domain is not a decided thing (dev's is jackson.lan, PiHole's zone).
+# An EMPTY list is the role's supported way to say "configure no search domain", which is
+# node 1's current behaviour — the conservative non-decision, not a missing value.
+node_baseline_dns_search: []