jakman85andClaude Opus 5
c1f24e4a6d
Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.
The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.
MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
* systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
silently bypass the local zone and ad-blocking.
* resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
would destroy PiHole-first ordering.
* contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.
SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.
PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.
tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.
ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.
Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
74 lines
4.4 KiB
YAML
74 lines
4.4 KiB
YAML
---
|
|
# node_baseline — first-boot floor for k3s/Cilium/Longhorn, moved out of cloud-init
|
|
# (Proxmox provisioning is API-only). Runs before k3s (site.yml). Idempotent.
|
|
node_baseline_packages:
|
|
# VM-only in practice (binds the virtio-serial channel Proxmox attaches) but harmless to install
|
|
# unconditionally: the task below only *starts* it when that channel exists, so on bare metal
|
|
# (prod) it installs and stays a dormant, disabled unit. Kept in the shared list rather than
|
|
# split per-platform so dev and prod install from the exact same package set (R-ENV-02 reuse).
|
|
- qemu-guest-agent
|
|
- open-iscsi
|
|
- nfs-common
|
|
- curl
|
|
- cryptsetup # Longhorn >=1.12 RequiredPackages check needs the cryptsetup userspace tool (dm_crypt module alone isn't enough)
|
|
# Drive health (smartctl) is a PERMANENT part of a bare-metal node's baseline, not a one-off
|
|
# manual install: prod runs on real disks that can fail/degrade (see infra/PROD-PROVISIONING.md
|
|
# §2/§5 — a ~15-year-old consumer SSD on node 1), where dev's virtio disks never needed it.
|
|
# Installed here (not hand-run with apt-get) so it's declared, idempotent, and present on every
|
|
# node from first baseline — package installation is the role's job, never a manual step
|
|
# (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo).
|
|
- smartmontools
|
|
node_baseline_kernel_modules:
|
|
- overlay
|
|
- br_netfilter
|
|
- iscsi_tcp
|
|
- dm_crypt
|
|
- nfs
|
|
node_baseline_longhorn_device: /dev/vdb
|
|
node_baseline_longhorn_mount: /var/lib/longhorn
|
|
node_baseline_longhorn_fs_label: longhorn
|
|
# Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
|
|
# so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
|
|
# See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
|
|
# device renames (matches the Longhorn LABEL= approach).
|
|
node_baseline_etcd_device: /dev/vdc
|
|
node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db
|
|
node_baseline_etcd_fs_label: k3s-etcd
|
|
|
|
# --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
|
|
# These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml
|
|
# (the same seam kube_vip_address / k3s_api_host / cluster_kubeconfig_name use) — NOT in
|
|
# clusters/base/cluster-vars.yaml, which is a Flux postBuild ConfigMap that no Ansible
|
|
# code reads.
|
|
#
|
|
# WHY: the nodes shipped with ONE upstream resolver. PiHole answers the local jackson.lan
|
|
# zone instantly but its external recursion flaps (measured 0/25 for registry-1.docker.io /
|
|
# quay.io / ghcr.io while gitea.jackson.lan resolved 5/5), and one upstream is a hard SPOF:
|
|
# it broke containerd image pulls ("lookup registry-1.docker.io: Try again") AND CoreDNS,
|
|
# which forwards to /etc/resolv.conf and flooded its log with i/o timeouts to 192.168.1.18.
|
|
#
|
|
# ORDER IS THE POLICY. This is a RESILIENCE fix, not a bypass: the first entry stays PiHole,
|
|
# so *.jackson.lan and network-wide ad-blocking keep working exactly as before, and the
|
|
# second is only ever consulted when the first fails. Both write paths preserve that order
|
|
# (systemd-resolved tries the servers of one scope in order; glibc walks `nameserver` lines
|
|
# in order). Never put a public resolver first, and never enable `rotate` — either would
|
|
# silently take the local zone and the ad-block policy out of the path.
|
|
#
|
|
# Keep in parity with infra/tofu (var.dns_server, var.network_gateway), which sets the same
|
|
# two servers via cloud-init so a freshly rebuilt dev node boots with this list already.
|
|
node_baseline_dns_servers:
|
|
- 192.168.1.18 # PiHole — local jackson.lan zone + ad-blocking. PRIMARY, keep first.
|
|
- 192.168.1.1 # pfSense gateway — reliable external recursion. FALLBACK only.
|
|
# Search domain(s). An EMPTY list means "configure no search domain" and is a legitimate
|
|
# value (prod uses it — its local domain is not a decided thing), not a missing one.
|
|
node_baseline_dns_search:
|
|
- jackson.lan
|
|
# resolv.conf `options` for the DIRECT-file mechanism only (systemd-resolved does its own
|
|
# per-server failover timing and ignores these). glibc's defaults are timeout:5 attempts:2,
|
|
# i.e. up to 20s of stalled lookups before it reaches the second server — long enough for
|
|
# containerd to give up on a pull. These shorten the failover, and deliberately do NOT
|
|
# include `rotate`, which would round-robin the servers and break PiHole-first ordering.
|
|
node_baseline_dns_options:
|
|
- timeout:2
|
|
- attempts:2
|