--- # TAGS — every task here carries one of `packages`, `kernel`, `system`, `disks` or `dns`. An # untagged run is unchanged (all five run, in this order). The split exists because the PACKAGE floor # and the DATA-DISK work have different readiness: packages are safe on any node from # first contact, while the disk tasks need block devices that a new environment may not # have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked # on a SMART read that needs `smartmontools` — which ships in node_baseline_packages). # `--tags packages` breaks that circle: it installs the package floor and NOTHING else. # ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages # Skipping is the guard: with `--tags packages` no `disks`-tagged task is selected, so a # BLOCKED-pending-disk-decision device path is never even templated, let alone handed to # mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block # at the bottom of this file) works the same way and guards its own BLOCKED placeholder: # ansible-playbook playbooks/node-baseline.yml --tags dns # `kernel` (the backports-kernel block below) is separated for the same reason `packages` # is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and # it must be runnable without touching a disk decision that is still open: # ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel - name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs) ansible.builtin.apt: name: "{{ node_baseline_packages }}" state: present update_cache: true cache_valid_time: 3600 tags: [packages] # ===================================================================================== # BACKPORTS KERNEL — tag `kernel` (EFI memattr; infra/PROD-PROVISIONING.md §0b and §3) # ===================================================================================== # A STANDING DEVIATION from the Debian 13 baseline, encoded here so every current AND # FUTURE prod node gets it from provisioning instead of from someone remembering. # # WHAT IT FIXES. HP R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI # Memory Attributes Table. `efi_memattr_init()` in Linux < 6.13 computes # `tbl_size = sizeof(*tbl) + num_entries * desc_size` from two UNVALIDATED u32 fields into a # signed int and `memblock_reserve()`s the product; when bit 31 lands the value sign-extends # through phys_addr_t, `memblock_cap_size()` clamps it to PHYS_ADDR_MAX, and EVERY physical # address above the table is reserved. k3s-prod-2 booted with 2.6 GiB of its 16 GB, twice. # Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" # (first released in v6.13) validates the size and rejects the table instead. It is absent # from v6.12 and from linux-6.12.y, has no `Cc: stable` and no `Fixes:` tag, so no automatic # stable feed will ever pick it up. There is NO kernel cmdline workaround — # `efi_memattr_init()` runs from `efi_config_parse_tables()` before any `efi=` bail-out could # take effect. Booting a >= 6.13 kernel is the only fix, and ALL THREE prod nodes are on the # same lottery — node 1 and node 3 have merely drawn safe tickets so far. # # WHY IT IS SCOPED OFF FOR DEV. `node_baseline_backports_kernel` defaults to FALSE in # defaults/main.yml and is set true in inventory/group_vars/k3s_prod/vars.yml only. Dev is # Proxmox VMs whose firmware publishes no MAT at all — genuinely unaffected — and pulling a # LIVE cluster onto a backports kernel as a side effect of sharing this role would be a real # change for no benefit. See defaults/main.yml for why this is a variable rather than a # `k3s-prod-*` hostname test. # # IT DOES NOT REBOOT. A new kernel is inert until the node reboots, and this role # deliberately will not take one: k3s-prod-2 hung on a warm reboot on 2026-09-03 — powered, # off the LAN for ~38 minutes, power button required (§0b "SECOND FAULT"). Rebooting is # therefore a SERIALIZED, SUPERVISED operator step — one node at a time, verify each returns # before touching the next, never node 2 first — and a role cannot enforce that. The last # task here reports the pending reboot loudly instead of letting a green play imply the node # is already fixed. - name: Backports kernel for the EFI memattr fix (>= 6.13, commit 8fbe4c49c0cc) when: node_baseline_backports_kernel | bool tags: [kernel] block: # The suite name and the running release must agree, or apt resolves no candidate and # `state: latest` reports "ok" while leaving the node on the vulnerable 6.12 kernel — # a green play that fixed nothing. Fail on the mismatch instead (e.g. a node # dist-upgraded to forky while group_vars still says trixie-backports). - name: Assert the configured backports suite matches this node's Debian release ansible.builtin.assert: that: - ansible_facts['distribution'] == 'Debian' - node_baseline_backports_suite == (ansible_facts['distribution_release'] ~ '-backports') fail_msg: >- node_baseline_backports_kernel is enabled but node_baseline_backports_suite ({{ node_baseline_backports_suite }}) is not the backports suite for this node's release ({{ ansible_facts['distribution'] }} {{ ansible_facts['distribution_release'] | default('?') }}). apt would resolve no candidate and this play would report success while leaving the node on the vulnerable kernel. Fix the suite in group_vars — do not remove this assert. quiet: true # deb822 `.sources`, not a legacy one-line `deb` entry: it is the format Debian 13 # prefers, it carries `Signed-By` explicitly (binding this source to the archive keyring # rather than to whatever is in /etc/apt/trusted.gpg.d), and a `copy` of exact content is # idempotent in a way `apt_repository` — which rewrites and reformats — is not. # # THERE IS DELIBERATELY NO /etc/apt/preferences.d PIN, and that is not an oversight. The # backports Release file carries `NotAutomatic: yes` + `ButAutomaticUpgrades: yes`, which # apt already reads as: priority 100 so NOTHING is pulled from backports by accident, but # 500 for a package that was explicitly installed from there so it keeps getting updates. # Writing an explicit `Pin-Priority: 100` looks like belt-and-braces and is actively # HARMFUL — it overrides the ButAutomaticUpgrades half, and the kernel would then never # be upgraded again by `apt upgrade`. Enabling the source IS the whole configuration. - name: Enable the Debian backports apt source (deb822) ansible.builtin.copy: dest: "/etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources" owner: root group: root mode: "0644" content: | # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand. # A DELIBERATE, STANDING deviation from the Debian 13 baseline, enabled ONLY so this # machine can boot a kernel >= 6.13 that validates the corrupt EFI Memory Attributes # Table its firmware publishes (upstream 8fbe4c49c0cc). See # infra/PROD-PROVISIONING.md §0b/§3, and {{ node_baseline_backports_kernel_stamp }} # on this node. # # Backports is NotAutomatic, so nothing is taken from here unless asked for by name # with `-t {{ node_baseline_backports_suite }}`. Do NOT add an apt pin for it: # ButAutomaticUpgrades is what keeps the installed kernel patched, and an explicit # Pin-Priority would switch that off. Types: deb URIs: {{ node_baseline_backports_mirror }} Suites: {{ node_baseline_backports_suite }} Components: {{ node_baseline_backports_components | join(' ') }} Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg register: node_baseline_backports_source - name: Refresh the apt cache after enabling backports ansible.builtin.apt: update_cache: true when: node_baseline_backports_source is changed # `state: latest`, NOT `state: present`. linux-image-amd64 is ALREADY installed on these # nodes from trixie main (6.12.107-1), so `present` would find the name satisfied and do # nothing at all — the single most likely way to write this task and have it silently fix # nothing forever. `latest` + `default_release` resolves the candidate inside backports # and upgrades the metapackage to it. Once the node is current a re-run is a genuine # no-op: no change, no churn, no spurious reboot flag. Scope is exactly # node_baseline_backports_kernel_packages — one metapackage — so nothing else on the node # is dragged out of the main archive. - name: Install the fixed kernel from backports (>= 6.13) ansible.builtin.apt: name: "{{ node_baseline_backports_kernel_packages }}" state: latest default_release: "{{ node_baseline_backports_suite }}" update_cache: true cache_valid_time: 3600 register: node_baseline_backports_kernel_apt # WHICH VERSION DID THIS NODE ACTUALLY LAND ON. Read back rather than assumed: the # metapackage tracks a moving target, so "we installed linux-image-amd64" is not an # answer anybody can roll back from six months later. - name: Read back the installed backports kernel metapackage version ansible.builtin.command: dpkg-query -W -f '${Version}' {{ node_baseline_backports_kernel_packages | first }} register: node_baseline_backports_kernel_version changed_when: false check_mode: false # `linux-version` ships in linux-base and knows Debian's kernel version ordering, which # neither `sort -V` nor a Jinja sort gets right across +deb13 / ABI suffixes. The full # list is the ROLLBACK INVENTORY: every entry is still installed and selectable from # GRUB's "Advanced options" without uninstalling anything. - name: List every kernel image installed on this node, newest first ansible.builtin.shell: cmd: set -o pipefail; linux-version list | linux-version sort --reverse executable: /bin/bash register: node_baseline_installed_kernels changed_when: false check_mode: false # The node must be able to explain itself to an operator standing in front of it with no # repo checkout. Content is deliberately time-INVARIANT (no timestamp) so re-running the # role is a real no-op rather than a permanent daily "changed". - name: Record the deviation, the exact version, and the rollback path on the node ansible.builtin.copy: dest: "{{ node_baseline_backports_kernel_stamp }}" owner: root group: root mode: "0644" content: | # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand. # # THIS NODE DELIBERATELY DOES NOT RUN THE DEBIAN 13 STOCK KERNEL. # # WHY # This machine's HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table. # Linux < 6.13 reserves memory from it without validating its size, and an unlucky # boot reserves everything above the table: 2.6 GiB of 16 GB, observed twice on # k3s-prod-2. Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size # is clearly bogus" (first released in v6.13) adds the validation; it is NOT in # v6.12 and NOT in linux-6.12.y. There is NO kernel cmdline workaround. # Full write-up: infra/PROD-PROVISIONING.md §0b in the home-network-k3s-impl repo. # # PROOF THE FIX IS ACTIVE (not merely a lucky boot) # BOOT-INDEPENDENT, and this is the one to trust — the string only exists in # >= 6.13, so its presence in the RUNNING image is proof regardless of the boot: # zstd -dc < <(tail -c +$(...)) ... | grep 'Corrupted EFI Memory Attributes' # (roles/node_health does the version check for you; see the health report's # firmware.efi_memattr finding.) # # FROM dmesg, WHICH IS A LOTTERY. efi_memattr_init() on >= 6.13 has THREE gates # and each prints a DIFFERENT line — all three are rejections: # tbl->version > 2 'Unexpected EFI Memory Attributes table version %d' # (NOTE: printed with %d, so garbage may read NEGATIVE) # bad desc_size 'Unexpected EFI Memory Attributes descriptor size ...' # num_entries > 64K '[Firmware Bug]: Corrupted EFI Memory Attributes # Table detected! (version == ..., num_entries == ...)' # Which one appears depends on WHICH field this boot's garbage corrupted, so do # NOT treat the absence of the FW_BUG line as the fix being absent — all three # prod nodes printed the *version* line on 2026-09-04. # The one line that means the fix is NOT active: # 'Failed to map EFI Memory Attributes table @ 0x...' — the kernel trusted the # corrupt table. On a >= 6.13 kernel this should be unreachable; if it appears, # escalate, do not shrug. # # INSTALLED # source /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources # package {{ node_baseline_backports_kernel_packages | join(' ') }} # version {{ node_baseline_backports_kernel_version.stdout | default('(not read)') }} # kernels installed on this node, newest first: {% for kv in (node_baseline_installed_kernels.stdout_lines | default([])) %} # {{ kv }} {% endfor %} # # SECURITY COST # Packages from *-backports are provided AS IS. They are NOT covered by the Debian # Security Team's guarantees for the main archive: no security-tracker commitment, # no DSAs. Accepted deliberately, in exchange for the node keeping its RAM. # # ROLLBACK — no repo checkout needed # 1. NON-DESTRUCTIVE, try this first. Every kernel listed above is still installed # and bootable: pick the 6.12 one from GRUB -> "Advanced options for Debian # GNU/Linux". That answers "is the new kernel the problem?" without # uninstalling anything. Note it reinstates the memattr bug for that boot. # 2. PERMANENT. Set node_baseline_backports_kernel: false in # inventory/group_vars/k3s_prod/vars.yml FIRST — otherwise the next Ansible run # puts the backports kernel straight back — then, on this node: # apt-get install --reinstall linux-image-amd64/{{ ansible_facts['distribution_release'] }} # apt-get purge # rm /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources # apt-get update && update-grub && reboot # # EXIT CONDITION # If 8fbe4c49c0cc is ever picked up into linux-6.12.y (and so into Debian's 6.12 # rebase), this deviation can be removed entirely: flip the variable to false, # roll back as above, and delete the `kernel` block from roles/node_baseline. # Say the pending reboot out loud. A green play here does NOT mean the node is fixed — # it means the fixed kernel is on disk. See the block header for why the role will not # reboot on its own. - name: Report whether this node is running the kernel it now has installed ansible.builtin.debug: msg: >- {{ ('REBOOT REQUIRED — running ' ~ ansible_facts['kernel'] ~ ', newest installed is ' ~ newest ~ '. The EFI memattr fix is NOT active until this node reboots. Reboot prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER node 2 first (infra/PROD-PROVISIONING.md §0b "SECOND FAULT"). Afterwards confirm with: dmesg | grep -i "Corrupted EFI Memory Attributes Table"') if newest != ansible_facts['kernel'] else ('Running the newest installed kernel (' ~ ansible_facts['kernel'] ~ '); no reboot pending for the memattr fix.') }} vars: newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}" - name: Persist kernel modules for k3s/Cilium/Longhorn ansible.builtin.copy: dest: /etc/modules-load.d/k3s-cilium-longhorn.conf owner: root group: root mode: "0644" content: "{{ node_baseline_kernel_modules | join('\n') }}\n" tags: [system] - name: Load kernel modules now community.general.modprobe: name: "{{ item }}" state: present loop: "{{ node_baseline_kernel_modules }}" tags: [system] - name: Install sysctl tuning for k3s/Cilium ansible.builtin.copy: dest: /etc/sysctl.d/99-k3s-cilium.conf owner: root group: root mode: "0644" content: | net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 net.ipv6.conf.all.forwarding = 1 fs.inotify.max_user_instances = 8192 fs.inotify.max_user_watches = 524288 notify: Reload sysctl tags: [system] - name: Enable and start iscsid (Longhorn prerequisite) ansible.builtin.systemd: name: iscsid enabled: true state: started tags: [system] # qemu-guest-agent binds to the virtio-serial channel /dev/virtio-ports/org.qemu.guest_agent.0, # which Proxmox attaches only when the VM's `agent` option is enabled. Our tofu keeps agent # DISABLED on purpose (enabling it makes `tofu apply` block up to 15m waiting for an agent the # stock Debian cloud image hasn't installed yet — bpg gotcha). So we ENABLE the unit for the day # a channel exists, but only START it when the channel is actually present — never fail the # baseline over a missing hypervisor channel. - name: Check whether the qemu guest-agent virtio channel is attached ansible.builtin.stat: path: /dev/virtio-ports/org.qemu.guest_agent.0 register: node_baseline_qga_channel tags: [system] - name: Enable qemu-guest-agent (start only when its virtio channel is present) ansible.builtin.systemd: name: qemu-guest-agent enabled: true state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}" tags: [system] # ===================================================================================== # DATA DISKS — tag `disks`, in tasks/disks.yml # ===================================================================================== # Split into its own file for two reasons. It is the only DESTRUCTIVE part of the role # (prod re-purposes the partition already mounted at /srv as the Longhorn store), so it # should be readable end-to-end on its own; and both environments' storage shapes — dev's # two dedicated virtio disks, prod's single partitioned physical disk with etcd left on # the OS filesystem — are declared there as an explicit enum rather than inferred from # which variables happen to be set. See tasks/disks.yml and infra/PROD-PROVISIONING.md §2. # # The `disks` tag is applied ONCE here, in BOTH places it is needed: # * `tags:` on the include itself, so `--tags packages` never even reaches the include # and a placeholder device path is not so much as templated; and # * `apply: tags:`, so the tag reaches the tasks INSIDE the file. This half is not # optional and not decoration — a DYNAMIC include (`include_tasks`) does NOT propagate # its own tags to its children the way `import_tasks` does, so without `apply` the # included tasks are untagged and `--tags disks` selects the include and then runs # NOTHING inside it. That failure is silent and reports success, which is the worst # shape a guard can have. # A FULL run against a node whose layout is undeclared stops on the assert at the top of # that file instead of failing obscurely at mkfs time. - name: Provision the data disks (Longhorn store, etcd data dir) ansible.builtin.include_tasks: file: disks.yml apply: tags: [disks] tags: [disks] - name: Disable swap now ansible.builtin.command: swapoff -a changed_when: false tags: [system] - name: Neutralize swap entries in fstab (kubelet requires swap off) ansible.builtin.replace: path: /etc/fstab regexp: '^([^#].*\sswap\s.*)$' replace: '# \1 # disabled for k3s/kubelet' tags: [system] # ===================================================================================== # DNS RESILIENCE — tag `dns` (R-DNS-01 / R-DNS-03) # ===================================================================================== # Root cause (verified on dev): the nodes carried a SINGLE upstream resolver. PiHole # (192.168.1.18) answers `*.jackson.lan` instantly but its external recursion flaps — # 0/25 for registry-1.docker.io / quay.io / ghcr.io while gitea.jackson.lan resolved 5/5. # One upstream is a hard SPOF: containerd image pulls died with "lookup # registry-1.docker.io: Try again", and CoreDNS (`forward . /etc/resolv.conf`) flooded its # log with "i/o timeout" to 192.168.1.18. The fix is ADDITIVE — PiHole stays first so the # local zone and the ad-blocking policy are untouched; a second upstream is only consulted # when the first fails. See defaults/main.yml for the ordering rule. # # WHY THIS DETECTS INSTEAD OF ASSUMING. An earlier attempt (fix/node-dns-resilient-upstream, # dropped) wrote /etc/netplan/99-k3s-dns.yaml and ran `netplan apply`. netplan is NOT # installed on these nodes — they run the Debian 13 trixie generic cloud image # (infra/tofu/variables.tf) and netplan is not in node_baseline_packages — so that handler # would have failed command-not-found and broken node-baseline.yml on EVERY node. Worse, # the two environments do not even share a resolver stack: dev's /etc/resolv.conf is the # systemd-resolved stub (127.0.0.53) while prod's is a directly-managed file listing # 192.168.2.1 (it has no cloud-init at all — bare metal). So the mechanism is DETECTED per # node and the matching seam is written. Adding a third mechanism means adding a branch # here, not rewriting the role. # # All of it is `dns`-tagged, so `--tags packages` (the prod package-floor escape hatch # documented at the top of this file) still installs nothing but packages. - name: Determine whether systemd-resolved is running on this node ansible.builtin.command: systemctl is-active systemd-resolved register: node_baseline_resolved_active changed_when: false failed_when: false check_mode: false tags: [dns] - name: Determine whether NetworkManager is running (it rewrites /etc/resolv.conf) ansible.builtin.command: systemctl is-active NetworkManager register: node_baseline_nm_active changed_when: false failed_when: false check_mode: false tags: [dns] # follow: false — we need to know whether /etc/resolv.conf IS a symlink and where it # points, which is the strongest single signal of who owns the file. - name: Inspect /etc/resolv.conf itself (symlink or real file, and its target) ansible.builtin.stat: path: /etc/resolv.conf follow: false register: node_baseline_resolv_stat tags: [dns] - name: Read the current /etc/resolv.conf contents ansible.builtin.slurp: src: /etc/resolv.conf register: node_baseline_resolv_slurp failed_when: false tags: [dns] # resolvconf/openresolv keeps the authoritative copy under here and regenerates # /etc/resolv.conf from it, so writing /etc/resolv.conf directly would be silently undone. - name: Detect a resolvconf/openresolv installation that would clobber /etc/resolv.conf ansible.builtin.stat: path: /etc/resolvconf/resolv.conf.d register: node_baseline_resolvconf_dir tags: [dns] # THE DETECTION. Three outcomes, in priority order: # systemd-resolved — resolved is running AND it owns /etc/resolv.conf, proven either by # the symlink pointing into /run/systemd/resolve/ or by the file containing only # resolved's loopback stub (127.0.0.53 / 127.0.0.54). Writing a resolved drop-in is # then the ONLY thing that changes real behaviour; writing resolv.conf would be # overwritten by resolved (or, worse, land in /run and vanish at reboot). # contended — nobody's stub, but resolvconf or NetworkManager is present/running and # would regenerate the file underneath us. We do NOT write. The role fails loudly # below rather than leaving an operator believing a change stuck when it did not. # resolv-conf-file — a plain, directly-managed /etc/resolv.conf (prod's shape today). # Safe to own outright. # resolved running but NOT owning resolv.conf correctly falls through to the file branch: # in that state the drop-in would change nothing glibc can see. - name: Detect which resolver mechanism this node actually uses ansible.builtin.set_fact: node_baseline_dns_mechanism: >- {{ 'systemd-resolved' if (node_baseline_resolved_running and node_baseline_resolved_owns_resolv) else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present) else 'resolv-conf-file') }} vars: node_baseline_resolved_running: "{{ (node_baseline_resolved_active.stdout | default('') | trim) == 'active' }}" node_baseline_nm_running: "{{ (node_baseline_nm_active.stdout | default('') | trim) == 'active' }}" node_baseline_resolvconf_present: "{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}" node_baseline_resolv_text: "{{ (node_baseline_resolv_slurp.content | default('') | b64decode) }}" node_baseline_resolv_ns: "{{ node_baseline_resolv_text | regex_findall('(?m)^\\s*nameserver\\s+(\\S+)') }}" node_baseline_resolved_owns_resolv: >- {{ ('/run/systemd/resolve/' in (node_baseline_resolv_stat.stat.lnk_source | default(''))) or ((node_baseline_resolv_ns | length) > 0 and (node_baseline_resolv_ns | reject('in', ['127.0.0.53', '127.0.0.54']) | list | length) == 0) }} tags: [dns] - name: Report the detected resolver mechanism ansible.builtin.debug: msg: >- resolver mechanism = {{ node_baseline_dns_mechanism }} (systemd-resolved is-active={{ node_baseline_resolved_active.stdout | default('') | trim }}, NetworkManager is-active={{ node_baseline_nm_active.stdout | default('') | trim }}, /etc/resolv.conf islnk={{ node_baseline_resolv_stat.stat.islnk | default(false) }} target={{ node_baseline_resolv_stat.stat.lnk_source | default('(not a symlink)') }}, resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }}) tags: [dns] # Same shape, and the same reason, as the data-disk assert above: this environment's # upstream list is an OPERATOR DECISION, and a placeholder that fails LOUDLY beats both a # guess and a silent inheritance of dev's PiHole. group_vars/k3s_prod/vars.yml deliberately # carries a BLOCKED second entry (see its comments and PROD-PROVISIONING.md §4): prod's # 192.168.2.1 is currently gateway AND only upstream, and choosing what sits behind it is a # network decision this role must not make. Running against prod stops HERE, before # anything is written. - name: Assert this environment's DNS upstream list has actually been decided ansible.builtin.assert: that: - node_baseline_dns_servers | length >= 2 - "'BLOCKED' not in (node_baseline_dns_servers | join(' '))" fail_msg: >- node_baseline_dns_servers is not a decided, redundant upstream list for this host's environment (got: {{ node_baseline_dns_servers | join(', ') }}). A single upstream is the exact SPOF this task exists to remove — it already broke containerd image pulls and CoreDNS forwarding on dev — so the role will not configure one, and it will NOT guess a second server: a public resolver would bypass the local zone and the ad-blocking policy, which is a decision, not a default. Decide the second upstream, record it in infra/PROD-PROVISIONING.md §4, and set node_baseline_dns_servers in inventory/group_vars/k3s_prod/vars.yml in the SAME change. To install only the package floor meanwhile, re-run with `--tags packages`, which skips every DNS task. quiet: true tags: [dns] # The `contended` outcome. Not a warning: an operator who is told "changed" while another # daemon quietly reverts the file is worse off than one who is told nothing happened. - name: Refuse to manage /etc/resolv.conf while another resolver manager owns it ansible.builtin.fail: msg: >- /etc/resolv.conf on this node is not the systemd-resolved stub, but resolvconf/openresolv (/etc/resolvconf/resolv.conf.d present: {{ node_baseline_resolvconf_dir.stat.exists | default(false) }}) or NetworkManager (is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}) is in play and would regenerate the file underneath this role. Writing it here would look like it worked and silently revert. Add the matching branch to roles/node_baseline (the `dns`-tagged block) for whichever manager this node really uses — do not force it. when: node_baseline_dns_mechanism == 'contended' tags: [dns] # --- Mechanism A: systemd-resolved (dev's shape today) ------------------------------- # A drop-in, never an edit of /etc/systemd/resolved.conf, so a package upgrade of that file # never conflicts. `DNS=` is one ORDERED list inside a single resolved scope: resolved uses # the first server and only rotates to the next when it fails, which is exactly the # PiHole-primary/fallback behaviour wanted — it is NOT a race between two servers, so a # gateway NXDOMAIN can never beat PiHole to a jackson.lan answer while PiHole is healthy. - name: Ensure the systemd-resolved drop-in directory exists ansible.builtin.file: path: /etc/systemd/resolved.conf.d state: directory owner: root group: root mode: "0755" when: node_baseline_dns_mechanism == 'systemd-resolved' tags: [dns] - name: Configure resilient DNS upstreams via a systemd-resolved drop-in ansible.builtin.copy: dest: /etc/systemd/resolved.conf.d/99-k3s-dns.conf owner: root group: root mode: "0644" content: | # Managed by the Ansible node_baseline role (tags: dns) — do not edit by hand. # Ordered upstream list: the FIRST server is the local-zone/ad-blocking resolver and # stays authoritative; the rest exist only so external resolution (containerd image # pulls, CoreDNS `forward . /etc/resolv.conf`) survives its recursion flapping. [Resolve] DNS={{ node_baseline_dns_servers | join(' ') }} Domains={{ node_baseline_dns_search | join(' ') }} # Reset to empty deliberately. systemd ships compiled-in public fallback resolvers # (Cloudflare/Google) that resolved would reach for if the list above ever came up # empty — which would silently bypass BOTH the local zone and the network's # ad-blocking. This fix adds redundancy; it never adds a bypass. FallbackDNS= notify: Restart systemd-resolved when: node_baseline_dns_mechanism == 'systemd-resolved' tags: [dns] # --- Mechanism B: a directly-managed /etc/resolv.conf (prod's shape today) ----------- # Only reached when nothing else owns the file (the `contended` branch above already # stopped the play otherwise). glibc walks `nameserver` lines in order and falls through # on timeout, so ordering carries the same policy as the resolved list. # NOTE this REPLACES the file: any nameserver currently there and NOT in # node_baseline_dns_servers is dropped. That is the point — the list in group_vars is the # whole declared truth — so an environment that wants its existing IPv6 upstreams kept # must list them there too. `backup: true` leaves the pre-role copy beside it the first # time it changes, because this file is load-bearing for reaching the node at all. - name: Configure resilient DNS upstreams directly in /etc/resolv.conf ansible.builtin.copy: dest: /etc/resolv.conf owner: root group: root mode: "0644" backup: true content: | # Managed by the Ansible node_baseline role (tags: dns) — do not edit by hand. # Ordered upstream list: the FIRST server is the local-zone resolver and stays # authoritative; the rest exist only so external resolution (containerd image pulls, # CoreDNS `forward . /etc/resolv.conf`) survives its recursion flapping. {% if node_baseline_dns_search | length > 0 %} search {{ node_baseline_dns_search | join(' ') }} {% endif %} {% for server in node_baseline_dns_servers %} nameserver {{ server }} {% endfor %} {% if node_baseline_dns_options | length > 0 %} options {{ node_baseline_dns_options | join(' ') }} {% endif %} when: node_baseline_dns_mechanism == 'resolv-conf-file' tags: [dns]