home-network-k3s-impl/infra/ansible/roles/node_baseline/tasks/main.yml

---
# TAGS — every task here carries one of `packages`, `kernel`, `system`, `disks` or `dns`. An
# untagged run is unchanged (all five run, in this order). The split exists because the PACKAGE floor
# and the DATA-DISK work have different readiness: packages are safe on any node from
# first contact, while the disk tasks need block devices that a new environment may not
# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
# on a SMART read that needs `smartmontools` — which ships in node_baseline_packages).
# `--tags packages` breaks that circle: it installs the package floor and NOTHING else.
#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
# Skipping is the guard: with `--tags packages` no `disks`-tagged task is selected, so a
# BLOCKED-pending-disk-decision device path is never even templated, let alone handed to
# mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
# at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
#   ansible-playbook playbooks/node-baseline.yml --tags dns
# `kernel` (the backports-kernel block below) is separated for the same reason `packages`
# is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
# it must be runnable without touching a disk decision that is still open:
#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel

- name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
  ansible.builtin.apt:
    name: "{{ node_baseline_packages }}"
    state: present
    update_cache: true
    cache_valid_time: 3600
  tags: [packages]

# =====================================================================================
# BACKPORTS KERNEL — tag `kernel` (EFI memattr; infra/PROD-PROVISIONING.md §0b and §3)
# =====================================================================================
# A STANDING DEVIATION from the Debian 13 baseline, encoded here so every current AND
# FUTURE prod node gets it from provisioning instead of from someone remembering.
#
# WHAT IT FIXES. HP R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI
# Memory Attributes Table. `efi_memattr_init()` in Linux < 6.13 computes
# `tbl_size = sizeof(*tbl) + num_entries * desc_size` from two UNVALIDATED u32 fields into a
# signed int and `memblock_reserve()`s the product; when bit 31 lands the value sign-extends
# through phys_addr_t, `memblock_cap_size()` clamps it to PHYS_ADDR_MAX, and EVERY physical
# address above the table is reserved. k3s-prod-2 booted with 2.6 GiB of its 16 GB, twice.
# Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus"
# (first released in v6.13) validates the size and rejects the table instead. It is absent
# from v6.12 and from linux-6.12.y, has no `Cc: stable` and no `Fixes:` tag, so no automatic
# stable feed will ever pick it up. There is NO kernel cmdline workaround —
# `efi_memattr_init()` runs from `efi_config_parse_tables()` before any `efi=` bail-out could
# take effect. Booting a >= 6.13 kernel is the only fix, and ALL THREE prod nodes are on the
# same lottery — node 1 and node 3 have merely drawn safe tickets so far.
#
# WHY IT IS SCOPED OFF FOR DEV. `node_baseline_backports_kernel` defaults to FALSE in
# defaults/main.yml and is set true in inventory/group_vars/k3s_prod/vars.yml only. Dev is
# Proxmox VMs whose firmware publishes no MAT at all — genuinely unaffected — and pulling a
# LIVE cluster onto a backports kernel as a side effect of sharing this role would be a real
# change for no benefit. See defaults/main.yml for why this is a variable rather than a
# `k3s-prod-*` hostname test.
#
# IT DOES NOT REBOOT. A new kernel is inert until the node reboots, and this role
# deliberately will not take one: k3s-prod-2 hung on a warm reboot on 2026-09-03 — powered,
# off the LAN for ~38 minutes, power button required (§0b "SECOND FAULT"). Rebooting is
# therefore a SERIALIZED, SUPERVISED operator step — one node at a time, verify each returns
# before touching the next, never node 2 first — and a role cannot enforce that. The last
# task here reports the pending reboot loudly instead of letting a green play imply the node
# is already fixed.
- name: Backports kernel for the EFI memattr fix (>= 6.13, commit 8fbe4c49c0cc)
  when: node_baseline_backports_kernel | bool
  tags: [kernel]
  block:

    # The suite name and the running release must agree, or apt resolves no candidate and
    # `state: latest` reports "ok" while leaving the node on the vulnerable 6.12 kernel —
    # a green play that fixed nothing. Fail on the mismatch instead (e.g. a node
    # dist-upgraded to forky while group_vars still says trixie-backports).
    - name: Assert the configured backports suite matches this node's Debian release
      ansible.builtin.assert:
        that:
          - ansible_facts['distribution'] == 'Debian'
          - node_baseline_backports_suite == (ansible_facts['distribution_release'] ~ '-backports')
        fail_msg: >-
          node_baseline_backports_kernel is enabled but node_baseline_backports_suite
          ({{ node_baseline_backports_suite }}) is not the backports suite for this node's
          release ({{ ansible_facts['distribution'] }}
          {{ ansible_facts['distribution_release'] | default('?') }}). apt would resolve no
          candidate and this play would report success while leaving the node on the
          vulnerable kernel. Fix the suite in group_vars — do not remove this assert.
        quiet: true

    # deb822 `.sources`, not a legacy one-line `deb` entry: it is the format Debian 13
    # prefers, it carries `Signed-By` explicitly (binding this source to the archive keyring
    # rather than to whatever is in /etc/apt/trusted.gpg.d), and a `copy` of exact content is
    # idempotent in a way `apt_repository` — which rewrites and reformats — is not.
    #
    # THERE IS DELIBERATELY NO /etc/apt/preferences.d PIN, and that is not an oversight. The
    # backports Release file carries `NotAutomatic: yes` + `ButAutomaticUpgrades: yes`, which
    # apt already reads as: priority 100 so NOTHING is pulled from backports by accident, but
    # 500 for a package that was explicitly installed from there so it keeps getting updates.
    # Writing an explicit `Pin-Priority: 100` looks like belt-and-braces and is actively
    # HARMFUL — it overrides the ButAutomaticUpgrades half, and the kernel would then never
    # be upgraded again by `apt upgrade`. Enabling the source IS the whole configuration.
    - name: Enable the Debian backports apt source (deb822)
      ansible.builtin.copy:
        dest: "/etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources"
        owner: root
        group: root
        mode: "0644"
        content: |
          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
          # A DELIBERATE, STANDING deviation from the Debian 13 baseline, enabled ONLY so this
          # machine can boot a kernel >= 6.13 that validates the corrupt EFI Memory Attributes
          # Table its firmware publishes (upstream 8fbe4c49c0cc). See
          # infra/PROD-PROVISIONING.md §0b/§3, and {{ node_baseline_backports_kernel_stamp }}
          # on this node.
          #
          # Backports is NotAutomatic, so nothing is taken from here unless asked for by name
          # with `-t {{ node_baseline_backports_suite }}`. Do NOT add an apt pin for it:
          # ButAutomaticUpgrades is what keeps the installed kernel patched, and an explicit
          # Pin-Priority would switch that off.
          Types: deb
          URIs: {{ node_baseline_backports_mirror }}
          Suites: {{ node_baseline_backports_suite }}
          Components: {{ node_baseline_backports_components | join(' ') }}
          Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
      register: node_baseline_backports_source

    - name: Refresh the apt cache after enabling backports
      ansible.builtin.apt:
        update_cache: true
      when: node_baseline_backports_source is changed

    # `state: latest`, NOT `state: present`. linux-image-amd64 is ALREADY installed on these
    # nodes from trixie main (6.12.107-1), so `present` would find the name satisfied and do
    # nothing at all — the single most likely way to write this task and have it silently fix
    # nothing forever. `latest` + `default_release` resolves the candidate inside backports
    # and upgrades the metapackage to it. Once the node is current a re-run is a genuine
    # no-op: no change, no churn, no spurious reboot flag. Scope is exactly
    # node_baseline_backports_kernel_packages — one metapackage — so nothing else on the node
    # is dragged out of the main archive.
    - name: Install the fixed kernel from backports (>= 6.13)
      ansible.builtin.apt:
        name: "{{ node_baseline_backports_kernel_packages }}"
        state: latest
        default_release: "{{ node_baseline_backports_suite }}"
        update_cache: true
        cache_valid_time: 3600
      register: node_baseline_backports_kernel_apt

    # WHICH VERSION DID THIS NODE ACTUALLY LAND ON. Read back rather than assumed: the
    # metapackage tracks a moving target, so "we installed linux-image-amd64" is not an
    # answer anybody can roll back from six months later.
    - name: Read back the installed backports kernel metapackage version
      ansible.builtin.command: dpkg-query -W -f '${Version}' {{ node_baseline_backports_kernel_packages | first }}
      register: node_baseline_backports_kernel_version
      changed_when: false
      check_mode: false

    # `linux-version` ships in linux-base and knows Debian's kernel version ordering, which
    # neither `sort -V` nor a Jinja sort gets right across +deb13 / ABI suffixes. The full
    # list is the ROLLBACK INVENTORY: every entry is still installed and selectable from
    # GRUB's "Advanced options" without uninstalling anything.
    - name: List every kernel image installed on this node, newest first
      ansible.builtin.shell:
        cmd: set -o pipefail; linux-version list | linux-version sort --reverse
        executable: /bin/bash
      register: node_baseline_installed_kernels
      changed_when: false
      check_mode: false

    # The node must be able to explain itself to an operator standing in front of it with no
    # repo checkout. Content is deliberately time-INVARIANT (no timestamp) so re-running the
    # role is a real no-op rather than a permanent daily "changed".
    - name: Record the deviation, the exact version, and the rollback path on the node
      ansible.builtin.copy:
        dest: "{{ node_baseline_backports_kernel_stamp }}"
        owner: root
        group: root
        mode: "0644"
        content: |
          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
          #
          # THIS NODE DELIBERATELY DOES NOT RUN THE DEBIAN 13 STOCK KERNEL.
          #
          # WHY
          #   This machine's HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table.
          #   Linux < 6.13 reserves memory from it without validating its size, and an unlucky
          #   boot reserves everything above the table: 2.6 GiB of 16 GB, observed twice on
          #   k3s-prod-2. Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size
          #   is clearly bogus" (first released in v6.13) adds the validation; it is NOT in
          #   v6.12 and NOT in linux-6.12.y. There is NO kernel cmdline workaround.
          #   Full write-up: infra/PROD-PROVISIONING.md §0b in the home-network-k3s-impl repo.
          #
          # PROOF THE FIX IS ACTIVE (not merely a lucky boot)
          #   BOOT-INDEPENDENT, and this is the one to trust — the string only exists in
          #   >= 6.13, so its presence in the RUNNING image is proof regardless of the boot:
          #     zstd -dc < <(tail -c +$(...)) ... | grep 'Corrupted EFI Memory Attributes'
          #   (roles/node_health does the version check for you; see the health report's
          #   firmware.efi_memattr finding.)
          #
          #   FROM dmesg, WHICH IS A LOTTERY. efi_memattr_init() on >= 6.13 has THREE gates
          #   and each prints a DIFFERENT line — all three are rejections:
          #     tbl->version > 2         'Unexpected EFI Memory Attributes table version %d'
          #                              (NOTE: printed with %d, so garbage may read NEGATIVE)
          #     bad desc_size            'Unexpected EFI Memory Attributes descriptor size ...'
          #     num_entries > 64K        '[Firmware Bug]: Corrupted EFI Memory Attributes
          #                               Table detected! (version == ..., num_entries == ...)'
          #   Which one appears depends on WHICH field this boot's garbage corrupted, so do
          #   NOT treat the absence of the FW_BUG line as the fix being absent — all three
          #   prod nodes printed the *version* line on 2026-09-04.
          #   The one line that means the fix is NOT active:
          #     'Failed to map EFI Memory Attributes table @ 0x...'  — the kernel trusted the
          #     corrupt table. On a >= 6.13 kernel this should be unreachable; if it appears,
          #     escalate, do not shrug.
          #
          # INSTALLED
          #   source   /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
          #   package  {{ node_baseline_backports_kernel_packages | join(' ') }}
          #   version  {{ node_baseline_backports_kernel_version.stdout | default('(not read)') }}
          #   kernels installed on this node, newest first:
          {% for kv in (node_baseline_installed_kernels.stdout_lines | default([])) %}
          #     {{ kv }}
          {% endfor %}
          #
          # SECURITY COST
          #   Packages from *-backports are provided AS IS. They are NOT covered by the Debian
          #   Security Team's guarantees for the main archive: no security-tracker commitment,
          #   no DSAs. Accepted deliberately, in exchange for the node keeping its RAM.
          #
          # ROLLBACK — no repo checkout needed
          #   1. NON-DESTRUCTIVE, try this first. Every kernel listed above is still installed
          #      and bootable: pick the 6.12 one from GRUB -> "Advanced options for Debian
          #      GNU/Linux". That answers "is the new kernel the problem?" without
          #      uninstalling anything. Note it reinstates the memattr bug for that boot.
          #   2. PERMANENT. Set node_baseline_backports_kernel: false in
          #      inventory/group_vars/k3s_prod/vars.yml FIRST — otherwise the next Ansible run
          #      puts the backports kernel straight back — then, on this node:
          #        apt-get install --reinstall linux-image-amd64/{{ ansible_facts['distribution_release'] }}
          #        apt-get purge <the backports linux-image-*+deb13-amd64 listed above>
          #        rm /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
          #        apt-get update && update-grub && reboot
          #
          # EXIT CONDITION
          #   If 8fbe4c49c0cc is ever picked up into linux-6.12.y (and so into Debian's 6.12
          #   rebase), this deviation can be removed entirely: flip the variable to false,
          #   roll back as above, and delete the `kernel` block from roles/node_baseline.

    # Say the pending reboot out loud. A green play here does NOT mean the node is fixed —
    # it means the fixed kernel is on disk. See the block header for why the role will not
    # reboot on its own.
    - name: Report whether this node is running the kernel it now has installed
      ansible.builtin.debug:
        msg: >-
          {{ ('REBOOT REQUIRED — running ' ~ ansible_facts['kernel'] ~ ', newest installed is '
              ~ newest ~ '. The EFI memattr fix is NOT active until this node reboots. Reboot
              prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER
              node 2 first (infra/PROD-PROVISIONING.md §0b "SECOND FAULT"). Afterwards confirm
              with: dmesg | grep -i "Corrupted EFI Memory Attributes Table"')
             if newest != ansible_facts['kernel']
             else ('Running the newest installed kernel (' ~ ansible_facts['kernel']
                   ~ '); no reboot pending for the memattr fix.') }}
      vars:
        newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"

- name: Persist kernel modules for k3s/Cilium/Longhorn
  ansible.builtin.copy:
    dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
    owner: root
    group: root
    mode: "0644"
    content: "{{ node_baseline_kernel_modules | join('\n') }}\n"
  tags: [system]

- name: Load kernel modules now
  community.general.modprobe:
    name: "{{ item }}"
    state: present
  loop: "{{ node_baseline_kernel_modules }}"
  tags: [system]

- name: Install sysctl tuning for k3s/Cilium
  ansible.builtin.copy:
    dest: /etc/sysctl.d/99-k3s-cilium.conf
    owner: root
    group: root
    mode: "0644"
    content: |
      net.bridge.bridge-nf-call-iptables  = 1
      net.bridge.bridge-nf-call-ip6tables = 1
      net.ipv4.ip_forward                 = 1
      net.ipv6.conf.all.forwarding        = 1
      fs.inotify.max_user_instances       = 8192
      fs.inotify.max_user_watches         = 524288
  notify: Reload sysctl
  tags: [system]

- name: Enable and start iscsid (Longhorn prerequisite)
  ansible.builtin.systemd:
    name: iscsid
    enabled: true
    state: started
  tags: [system]

# qemu-guest-agent binds to the virtio-serial channel /dev/virtio-ports/org.qemu.guest_agent.0,
# which Proxmox attaches only when the VM's `agent` option is enabled. Our tofu keeps agent
# DISABLED on purpose (enabling it makes `tofu apply` block up to 15m waiting for an agent the
# stock Debian cloud image hasn't installed yet — bpg gotcha). So we ENABLE the unit for the day
# a channel exists, but only START it when the channel is actually present — never fail the
# baseline over a missing hypervisor channel.
- name: Check whether the qemu guest-agent virtio channel is attached
  ansible.builtin.stat:
    path: /dev/virtio-ports/org.qemu.guest_agent.0
  register: node_baseline_qga_channel
  tags: [system]

- name: Enable qemu-guest-agent (start only when its virtio channel is present)
  ansible.builtin.systemd:
    name: qemu-guest-agent
    enabled: true
    state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
  tags: [system]

# =====================================================================================
# DATA DISKS — tag `disks`, in tasks/disks.yml
# =====================================================================================
# Split into its own file for two reasons. It is the only DESTRUCTIVE part of the role
# (prod re-purposes the partition already mounted at /srv as the Longhorn store), so it
# should be readable end-to-end on its own; and both environments' storage shapes — dev's
# two dedicated virtio disks, prod's single partitioned physical disk with etcd left on
# the OS filesystem — are declared there as an explicit enum rather than inferred from
# which variables happen to be set. See tasks/disks.yml and infra/PROD-PROVISIONING.md §2.
#
# The `disks` tag is applied ONCE here, in BOTH places it is needed:
#   * `tags:` on the include itself, so `--tags packages` never even reaches the include
#     and a placeholder device path is not so much as templated; and
#   * `apply: tags:`, so the tag reaches the tasks INSIDE the file. This half is not
#     optional and not decoration — a DYNAMIC include (`include_tasks`) does NOT propagate
#     its own tags to its children the way `import_tasks` does, so without `apply` the
#     included tasks are untagged and `--tags disks` selects the include and then runs
#     NOTHING inside it. That failure is silent and reports success, which is the worst
#     shape a guard can have.
# A FULL run against a node whose layout is undeclared stops on the assert at the top of
# that file instead of failing obscurely at mkfs time.
- name: Provision the data disks (Longhorn store, etcd data dir)
  ansible.builtin.include_tasks:
    file: disks.yml
    apply:
      tags: [disks]
  tags: [disks]

- name: Disable swap now
  ansible.builtin.command: swapoff -a
  changed_when: false
  tags: [system]

- name: Neutralize swap entries in fstab (kubelet requires swap off)
  ansible.builtin.replace:
    path: /etc/fstab
    regexp: '^([^#].*\sswap\s.*)$'
    replace: '# \1  # disabled for k3s/kubelet'
  tags: [system]

# =====================================================================================
# DNS RESILIENCE — tag `dns` (R-DNS-01 / R-DNS-03)
# =====================================================================================
# Root cause (verified on dev): the nodes carried a SINGLE upstream resolver. PiHole
# (192.168.1.18) answers `*.jackson.lan` instantly but its external recursion flaps —
# 0/25 for registry-1.docker.io / quay.io / ghcr.io while gitea.jackson.lan resolved 5/5.
# One upstream is a hard SPOF: containerd image pulls died with "lookup
# registry-1.docker.io: Try again", and CoreDNS (`forward . /etc/resolv.conf`) flooded its
# log with "i/o timeout" to 192.168.1.18. The fix is ADDITIVE — PiHole stays first so the
# local zone and the ad-blocking policy are untouched; a second upstream is only consulted
# when the first fails. See defaults/main.yml for the ordering rule.
#
# WHY THIS DETECTS INSTEAD OF ASSUMING. An earlier attempt (fix/node-dns-resilient-upstream,
# dropped) wrote /etc/netplan/99-k3s-dns.yaml and ran `netplan apply`. netplan is NOT
# installed on these nodes — they run the Debian 13 trixie generic cloud image
# (infra/tofu/variables.tf) and netplan is not in node_baseline_packages — so that handler
# would have failed command-not-found and broken node-baseline.yml on EVERY node. Worse,
# the two environments do not even share a resolver stack: dev's /etc/resolv.conf is the
# systemd-resolved stub (127.0.0.53) while prod's is a directly-managed file listing
# 192.168.2.1 (it has no cloud-init at all — bare metal). So the mechanism is DETECTED per
# node and the matching seam is written. Adding a third mechanism means adding a branch
# here, not rewriting the role.
#
# All of it is `dns`-tagged, so `--tags packages` (the prod package-floor escape hatch
# documented at the top of this file) still installs nothing but packages.

- name: Determine whether systemd-resolved is running on this node
  ansible.builtin.command: systemctl is-active systemd-resolved
  register: node_baseline_resolved_active
  changed_when: false
  failed_when: false
  check_mode: false
  tags: [dns]

- name: Determine whether NetworkManager is running (it rewrites /etc/resolv.conf)
  ansible.builtin.command: systemctl is-active NetworkManager
  register: node_baseline_nm_active
  changed_when: false
  failed_when: false
  check_mode: false
  tags: [dns]

# follow: false — we need to know whether /etc/resolv.conf IS a symlink and where it
# points, which is the strongest single signal of who owns the file.
- name: Inspect /etc/resolv.conf itself (symlink or real file, and its target)
  ansible.builtin.stat:
    path: /etc/resolv.conf
    follow: false
  register: node_baseline_resolv_stat
  tags: [dns]

- name: Read the current /etc/resolv.conf contents
  ansible.builtin.slurp:
    src: /etc/resolv.conf
  register: node_baseline_resolv_slurp
  failed_when: false
  tags: [dns]

# resolvconf/openresolv keeps the authoritative copy under here and regenerates
# /etc/resolv.conf from it, so writing /etc/resolv.conf directly would be silently undone.
- name: Detect a resolvconf/openresolv installation that would clobber /etc/resolv.conf
  ansible.builtin.stat:
    path: /etc/resolvconf/resolv.conf.d
  register: node_baseline_resolvconf_dir
  tags: [dns]

# THE DETECTION. Three outcomes, in priority order:
#   systemd-resolved — resolved is running AND it owns /etc/resolv.conf, proven either by
#       the symlink pointing into /run/systemd/resolve/ or by the file containing only
#       resolved's loopback stub (127.0.0.53 / 127.0.0.54). Writing a resolved drop-in is
#       then the ONLY thing that changes real behaviour; writing resolv.conf would be
#       overwritten by resolved (or, worse, land in /run and vanish at reboot).
#   contended — nobody's stub, but resolvconf or NetworkManager is present/running and
#       would regenerate the file underneath us. We do NOT write. The role fails loudly
#       below rather than leaving an operator believing a change stuck when it did not.
#   resolv-conf-file — a plain, directly-managed /etc/resolv.conf (prod's shape today).
#       Safe to own outright.
# resolved running but NOT owning resolv.conf correctly falls through to the file branch:
# in that state the drop-in would change nothing glibc can see.
- name: Detect which resolver mechanism this node actually uses
  ansible.builtin.set_fact:
    node_baseline_dns_mechanism: >-
      {{
        'systemd-resolved' if (node_baseline_resolved_running and node_baseline_resolved_owns_resolv)
        else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present)
              else 'resolv-conf-file')
      }}
  vars:
    node_baseline_resolved_running: "{{ (node_baseline_resolved_active.stdout | default('') | trim) == 'active' }}"
    node_baseline_nm_running: "{{ (node_baseline_nm_active.stdout | default('') | trim) == 'active' }}"
    node_baseline_resolvconf_present: "{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}"
    node_baseline_resolv_text: "{{ (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
    node_baseline_resolv_ns: "{{ node_baseline_resolv_text | regex_findall('(?m)^\\s*nameserver\\s+(\\S+)') }}"
    node_baseline_resolved_owns_resolv: >-
      {{ ('/run/systemd/resolve/' in (node_baseline_resolv_stat.stat.lnk_source | default('')))
         or ((node_baseline_resolv_ns | length) > 0
             and (node_baseline_resolv_ns | reject('in', ['127.0.0.53', '127.0.0.54']) | list | length) == 0) }}
  tags: [dns]

- name: Report the detected resolver mechanism
  ansible.builtin.debug:
    msg: >-
      resolver mechanism = {{ node_baseline_dns_mechanism }}
      (systemd-resolved is-active={{ node_baseline_resolved_active.stdout | default('') | trim }},
      NetworkManager is-active={{ node_baseline_nm_active.stdout | default('') | trim }},
      /etc/resolv.conf islnk={{ node_baseline_resolv_stat.stat.islnk | default(false) }}
      target={{ node_baseline_resolv_stat.stat.lnk_source | default('(not a symlink)') }},
      resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }})
  tags: [dns]

# Same shape, and the same reason, as the data-disk assert above: this environment's
# upstream list is an OPERATOR DECISION, and a placeholder that fails LOUDLY beats both a
# guess and a silent inheritance of dev's PiHole. group_vars/k3s_prod/vars.yml deliberately
# carries a BLOCKED second entry (see its comments and PROD-PROVISIONING.md §4): prod's
# 192.168.2.1 is currently gateway AND only upstream, and choosing what sits behind it is a
# network decision this role must not make. Running against prod stops HERE, before
# anything is written.
- name: Assert this environment's DNS upstream list has actually been decided
  ansible.builtin.assert:
    that:
      - node_baseline_dns_servers | length >= 2
      - "'BLOCKED' not in (node_baseline_dns_servers | join(' '))"
    fail_msg: >-
      node_baseline_dns_servers is not a decided, redundant upstream list for this host's
      environment (got: {{ node_baseline_dns_servers | join(', ') }}). A single upstream is
      the exact SPOF this task exists to remove — it already broke containerd image pulls
      and CoreDNS forwarding on dev — so the role will not configure one, and it will NOT
      guess a second server: a public resolver would bypass the local zone and the
      ad-blocking policy, which is a decision, not a default. Decide the second upstream,
      record it in infra/PROD-PROVISIONING.md §4, and set node_baseline_dns_servers in
      inventory/group_vars/k3s_prod/vars.yml in the SAME change. To install only the
      package floor meanwhile, re-run with `--tags packages`, which skips every DNS task.
    quiet: true
  tags: [dns]

# The `contended` outcome. Not a warning: an operator who is told "changed" while another
# daemon quietly reverts the file is worse off than one who is told nothing happened.
- name: Refuse to manage /etc/resolv.conf while another resolver manager owns it
  ansible.builtin.fail:
    msg: >-
      /etc/resolv.conf on this node is not the systemd-resolved stub, but
      resolvconf/openresolv (/etc/resolvconf/resolv.conf.d present:
      {{ node_baseline_resolvconf_dir.stat.exists | default(false) }}) or NetworkManager
      (is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}) is in play and
      would regenerate the file underneath this role. Writing it here would look like it
      worked and silently revert. Add the matching branch to roles/node_baseline (the
      `dns`-tagged block) for whichever manager this node really uses — do not force it.
  when: node_baseline_dns_mechanism == 'contended'
  tags: [dns]

# --- Mechanism A: systemd-resolved (dev's shape today) -------------------------------
# A drop-in, never an edit of /etc/systemd/resolved.conf, so a package upgrade of that file
# never conflicts. `DNS=` is one ORDERED list inside a single resolved scope: resolved uses
# the first server and only rotates to the next when it fails, which is exactly the
# PiHole-primary/fallback behaviour wanted — it is NOT a race between two servers, so a
# gateway NXDOMAIN can never beat PiHole to a jackson.lan answer while PiHole is healthy.
- name: Ensure the systemd-resolved drop-in directory exists
  ansible.builtin.file:
    path: /etc/systemd/resolved.conf.d
    state: directory
    owner: root
    group: root
    mode: "0755"
  when: node_baseline_dns_mechanism == 'systemd-resolved'
  tags: [dns]

- name: Configure resilient DNS upstreams via a systemd-resolved drop-in
  ansible.builtin.copy:
    dest: /etc/systemd/resolved.conf.d/99-k3s-dns.conf
    owner: root
    group: root
    mode: "0644"
    content: |
      # Managed by the Ansible node_baseline role (tags: dns) — do not edit by hand.
      # Ordered upstream list: the FIRST server is the local-zone/ad-blocking resolver and
      # stays authoritative; the rest exist only so external resolution (containerd image
      # pulls, CoreDNS `forward . /etc/resolv.conf`) survives its recursion flapping.
      [Resolve]
      DNS={{ node_baseline_dns_servers | join(' ') }}
      Domains={{ node_baseline_dns_search | join(' ') }}
      # Reset to empty deliberately. systemd ships compiled-in public fallback resolvers
      # (Cloudflare/Google) that resolved would reach for if the list above ever came up
      # empty — which would silently bypass BOTH the local zone and the network's
      # ad-blocking. This fix adds redundancy; it never adds a bypass.
      FallbackDNS=
  notify: Restart systemd-resolved
  when: node_baseline_dns_mechanism == 'systemd-resolved'
  tags: [dns]

# --- Mechanism B: a directly-managed /etc/resolv.conf (prod's shape today) -----------
# Only reached when nothing else owns the file (the `contended` branch above already
# stopped the play otherwise). glibc walks `nameserver` lines in order and falls through
# on timeout, so ordering carries the same policy as the resolved list.
# NOTE this REPLACES the file: any nameserver currently there and NOT in
# node_baseline_dns_servers is dropped. That is the point — the list in group_vars is the
# whole declared truth — so an environment that wants its existing IPv6 upstreams kept
# must list them there too. `backup: true` leaves the pre-role copy beside it the first
# time it changes, because this file is load-bearing for reaching the node at all.
- name: Configure resilient DNS upstreams directly in /etc/resolv.conf
  ansible.builtin.copy:
    dest: /etc/resolv.conf
    owner: root
    group: root
    mode: "0644"
    backup: true
    content: |
      # Managed by the Ansible node_baseline role (tags: dns) — do not edit by hand.
      # Ordered upstream list: the FIRST server is the local-zone resolver and stays
      # authoritative; the rest exist only so external resolution (containerd image pulls,
      # CoreDNS `forward . /etc/resolv.conf`) survives its recursion flapping.
      {% if node_baseline_dns_search | length > 0 %}
      search {{ node_baseline_dns_search | join(' ') }}
      {% endif %}
      {% for server in node_baseline_dns_servers %}
      nameserver {{ server }}
      {% endfor %}
      {% if node_baseline_dns_options | length > 0 %}
      options {{ node_baseline_dns_options | join(' ') }}
      {% endif %}
  when: node_baseline_dns_mechanism == 'resolv-conf-file'
  tags: [dns]