jakman85andClaude Opus 5
a984442cae
Two debt items, both about making a latent problem impossible rather than fixed. === 1. Node timezone pinned to Etc/UTC (node_baseline, `system` tag) === The prod nodes disagreed, measured from k3s's own etcd snapshot annotations: prod-1/2 at -05:00 (Central), prod-3 at -04:00 (Eastern). Functionally harmless — Kubernetes and etcd are UTC internally, and in UTC those snapshots are ~4 minutes apart, which is correct staggering — but it makes cross-node log correlation actively misleading, and this cluster has had two unplanned reboots in two days. `Etc/UTC` chosen on repo convention, not just principle: catalog/plex.yaml records "no other service in the repo sets a timezone ... so use the safe default" with TZ: Etc/UTC, and apps/wave-0/plex.yaml matches (platform/crafty is America/Chicago DELIBERATELY — a user-facing game schedule, not a log clock). Decisive extra fact: THE THREE DEV NODES ARE ALREADY Etc/UTC, so this makes prod match dev rather than inventing a convention. UTC also removes the class — no DST discontinuity, node time == platform time. New knob `node_baseline_timezone`, overridable per environment via group_vars (the same seam as node_baseline_backports_kernel) but deliberately NOT restated there: no environment needs to differ, and a second copy is a second thing to disagree. Guarded by an assert in the role's existing BLOCKED style before the module runs. `--check --diff --tags system -e k3s_hosts=k3s_prod` INDEPENDENTLY CONFIRMS the offset evidence, and is the whole verification — the play was never run for real: k3s-prod-1: -"America/Chicago" +"Etc/UTC" changed k3s-prod-2: -"America/Chicago" +"Etc/UTC" changed k3s-prod-3: -"America/New_York" +"Etc/UTC" changed The same --check against DEV reports the task `ok` on all three — idempotence demonstrated against live hosts, in check mode only. `--list-hosts` confirms `-e k3s_hosts=k3s_prod` selects the three prod nodes (the diagnostic plays take their OWN var; the wrong flag silently targets dev). *** AUTHORED, NOT APPLIED. *** Applying it is a live-cluster action and was deliberately left for the operator to schedule: prod's Longhorn had only just finished resyncing after a node reboot, and changing a node's clock config during storage recovery is not an unattended move. `ansible-inventory --list` was NOT used anywhere — it prints DECRYPTED vault values (the passphrase is mounted into the dev container) and has leaked the prod become-password to a transcript once already. === 2. New L0 check `placeholder-guard-coupling` (R-ENV-02, R-STATE-03) === On 2026-09-05 the PM changed a placeholder in clusters/base/cluster-vars.yaml from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID...`. The guards in bootstrap/standup.sh matched the glob `TBD-*` — a PREFIX. All three silently died, and `resume` would have enabled velero on prod against a nonexistent hostname: installs clean, reports healthy, no backups exist. Found by accident. A guard in one file keyed on the SHAPE of a value in another file, with nothing tying them. The check exercises the REAL predicate — it does not reimplement it. standup.sh is not sourceable (top-level argv parsing hard-exits, cd, export, EXIT trap), so the check extracts the predicate's own source lines and sources only those. Faithfulness is pinned by: discovery of every `*_is_placeholder` (not a hardcoded name), exactly-one-definition, "extracted block is a verbatim substring", a self-containment gate, a canary (must refuse "", must accept a real URL), and a scan for a divergent second copy. If the predicate grows a dependency the check FAILS LOUDLY and names the remedy rather than degrading to testing a copy — the failure mode it exists for. Both sides derive from real files: guarded knobs from a taint trace in standup.sh (`x="$(cvar KEY)"` -> alias hop -> `*_is_placeholder "$x"`); values from base cluster-vars plus each overlay's JSON6902 /data/KEY patches. The independent detector is deliberately NOT the guard: seed idioms UNION the guard's own words with glob anchors STRIPPED — so a prefix-anchored guard cannot pass, and deleting an arm cannot make a value stop being placeholder-shaped. Real tree: 32 assertions, 18 values through the extracted predicate, PASS. NON-VACUITY VERIFIED BY ME, twice, on throwaway copies (the real repo and the read-only spec repo untouched). My first perturbation tripped the self-containment rail instead of the anchoring property, so I redid it shape-preserving — substring globs -> prefix globs, nothing else — and it failed on exactly the right property, naming mid-string and tail acceptances with the diagnosis "This is the 2026-09-05 failure exactly". Control: unperturbed copy exits 0. === 3. ci/touched-requirements.sh: bootstrap/ was UNMAPPED (found by the agent) === The path->requirement map had no `bootstrap/*` entry, so a change touching standup.sh ALONGSIDE any already-mapped path scoped the L0 run to that other path's requirements and silently excluded the new check — the one that exists to catch exactly the 2026-09-05 class. A safety net with a hole where the incident was. Mapped to R-STATE-03 R-GITOPS-01; verified the map now resolves bootstrap paths. VERIFIED: full CI gate PASS. `run.sh --layer l0` 8/8 PASS including the new check. Coverage unchanged at 45/51 (the new check tags existing requirements). Refs R-ENV-02, R-STATE-03, R-TEST-01, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]>
1022 lines
61 KiB
YAML
1022 lines
61 KiB
YAML
---
|
|
# TAGS — every task here carries one of `packages`, `kernel`, `system`, `wireless`, `disks` or
|
|
# `dns`. An untagged run is unchanged (all six run, in this order). The split exists because the PACKAGE floor
|
|
# and the DATA-DISK work have different readiness: packages are safe on any node from
|
|
# first contact, while the disk tasks need block devices that a new environment may not
|
|
# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
|
|
# on a SMART read that needs `smartmontools` — which ships in node_baseline_packages).
|
|
# `--tags packages` breaks that circle: it installs the package floor and NOTHING else.
|
|
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
|
|
# Skipping is the guard: with `--tags packages` no `disks`-tagged task is selected, so a
|
|
# BLOCKED-pending-disk-decision device path is never even templated, let alone handed to
|
|
# mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
|
|
# at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
|
|
# ansible-playbook playbooks/node-baseline.yml --tags dns
|
|
# `kernel` (the backports-kernel block below) is separated for the same reason `packages`
|
|
# is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
|
|
# it must be runnable without touching a disk decision that is still open:
|
|
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
|
|
# `wireless` (a cluster node gets exactly ONE path onto the network) is separated for a third
|
|
# reason: it must be runnable BEFORE `dns` and without touching packages, the kernel or a disk
|
|
# decision, because a node with a second DHCP-managed interface reverts the `dns` write. It is
|
|
# placed before the disk work for the same reason, and the pair is the whole remedy for the
|
|
# k3s-prod-3 incident:
|
|
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags wireless,dns
|
|
|
|
- name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
|
|
ansible.builtin.apt:
|
|
name: "{{ node_baseline_packages }}"
|
|
state: present
|
|
update_cache: true
|
|
cache_valid_time: 3600
|
|
tags: [packages]
|
|
|
|
# =====================================================================================
|
|
# BACKPORTS KERNEL — tag `kernel` (EFI memattr; infra/PROD-PROVISIONING.md §0b and §3)
|
|
# =====================================================================================
|
|
# A STANDING DEVIATION from the Debian 13 baseline, encoded here so every current AND
|
|
# FUTURE prod node gets it from provisioning instead of from someone remembering.
|
|
#
|
|
# WHAT IT FIXES. HP R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI
|
|
# Memory Attributes Table. `efi_memattr_init()` in Linux < 6.13 computes
|
|
# `tbl_size = sizeof(*tbl) + num_entries * desc_size` from two UNVALIDATED u32 fields into a
|
|
# signed int and `memblock_reserve()`s the product; when bit 31 lands the value sign-extends
|
|
# through phys_addr_t, `memblock_cap_size()` clamps it to PHYS_ADDR_MAX, and EVERY physical
|
|
# address above the table is reserved. k3s-prod-2 booted with 2.6 GiB of its 16 GB, twice.
|
|
# Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus"
|
|
# (first released in v6.13) validates the size and rejects the table instead. It is absent
|
|
# from v6.12 and from linux-6.12.y, has no `Cc: stable` and no `Fixes:` tag, so no automatic
|
|
# stable feed will ever pick it up. There is NO kernel cmdline workaround —
|
|
# `efi_memattr_init()` runs from `efi_config_parse_tables()` before any `efi=` bail-out could
|
|
# take effect. Booting a >= 6.13 kernel is the only fix, and ALL THREE prod nodes are on the
|
|
# same lottery — node 1 and node 3 have merely drawn safe tickets so far.
|
|
#
|
|
# WHY IT IS SCOPED OFF FOR DEV. `node_baseline_backports_kernel` defaults to FALSE in
|
|
# defaults/main.yml and is set true in inventory/group_vars/k3s_prod/vars.yml only. Dev is
|
|
# Proxmox VMs whose firmware publishes no MAT at all — genuinely unaffected — and pulling a
|
|
# LIVE cluster onto a backports kernel as a side effect of sharing this role would be a real
|
|
# change for no benefit. See defaults/main.yml for why this is a variable rather than a
|
|
# `k3s-prod-*` hostname test.
|
|
#
|
|
# IT DOES NOT REBOOT. A new kernel is inert until the node reboots, and this role
|
|
# deliberately will not take one: k3s-prod-2 hung on a warm reboot on 2026-09-03 — powered,
|
|
# off the LAN for ~38 minutes, power button required (§0b "SECOND FAULT"). Rebooting is
|
|
# therefore a SERIALIZED, SUPERVISED operator step — one node at a time, verify each returns
|
|
# before touching the next, never node 2 first — and a role cannot enforce that. The last
|
|
# task here reports the pending reboot loudly instead of letting a green play imply the node
|
|
# is already fixed.
|
|
- name: Backports kernel for the EFI memattr fix (>= 6.13, commit 8fbe4c49c0cc)
|
|
when: node_baseline_backports_kernel | bool
|
|
tags: [kernel]
|
|
block:
|
|
|
|
# The suite name and the running release must agree, or apt resolves no candidate and
|
|
# `state: latest` reports "ok" while leaving the node on the vulnerable 6.12 kernel —
|
|
# a green play that fixed nothing. Fail on the mismatch instead (e.g. a node
|
|
# dist-upgraded to forky while group_vars still says trixie-backports).
|
|
- name: Assert the configured backports suite matches this node's Debian release
|
|
ansible.builtin.assert:
|
|
that:
|
|
- ansible_facts['distribution'] == 'Debian'
|
|
- node_baseline_backports_suite == (ansible_facts['distribution_release'] ~ '-backports')
|
|
fail_msg: >-
|
|
node_baseline_backports_kernel is enabled but node_baseline_backports_suite
|
|
({{ node_baseline_backports_suite }}) is not the backports suite for this node's
|
|
release ({{ ansible_facts['distribution'] }}
|
|
{{ ansible_facts['distribution_release'] | default('?') }}). apt would resolve no
|
|
candidate and this play would report success while leaving the node on the
|
|
vulnerable kernel. Fix the suite in group_vars — do not remove this assert.
|
|
quiet: true
|
|
|
|
# deb822 `.sources`, not a legacy one-line `deb` entry: it is the format Debian 13
|
|
# prefers, it carries `Signed-By` explicitly (binding this source to the archive keyring
|
|
# rather than to whatever is in /etc/apt/trusted.gpg.d), and a `copy` of exact content is
|
|
# idempotent in a way `apt_repository` — which rewrites and reformats — is not.
|
|
#
|
|
# THERE IS DELIBERATELY NO /etc/apt/preferences.d PIN, and that is not an oversight. The
|
|
# backports Release file carries `NotAutomatic: yes` + `ButAutomaticUpgrades: yes`, which
|
|
# apt already reads as: priority 100 so NOTHING is pulled from backports by accident, but
|
|
# 500 for a package that was explicitly installed from there so it keeps getting updates.
|
|
# Writing an explicit `Pin-Priority: 100` looks like belt-and-braces and is actively
|
|
# HARMFUL — it overrides the ButAutomaticUpgrades half, and the kernel would then never
|
|
# be upgraded again by `apt upgrade`. Enabling the source IS the whole configuration.
|
|
- name: Enable the Debian backports apt source (deb822)
|
|
ansible.builtin.copy:
|
|
dest: "/etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources"
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
# Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
|
|
# A DELIBERATE, STANDING deviation from the Debian 13 baseline, enabled ONLY so this
|
|
# machine can boot a kernel >= 6.13 that validates the corrupt EFI Memory Attributes
|
|
# Table its firmware publishes (upstream 8fbe4c49c0cc). See
|
|
# infra/PROD-PROVISIONING.md §0b/§3, and {{ node_baseline_backports_kernel_stamp }}
|
|
# on this node.
|
|
#
|
|
# Backports is NotAutomatic, so nothing is taken from here unless asked for by name
|
|
# with `-t {{ node_baseline_backports_suite }}`. Do NOT add an apt pin for it:
|
|
# ButAutomaticUpgrades is what keeps the installed kernel patched, and an explicit
|
|
# Pin-Priority would switch that off.
|
|
Types: deb
|
|
URIs: {{ node_baseline_backports_mirror }}
|
|
Suites: {{ node_baseline_backports_suite }}
|
|
Components: {{ node_baseline_backports_components | join(' ') }}
|
|
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
|
|
register: node_baseline_backports_source
|
|
|
|
- name: Refresh the apt cache after enabling backports
|
|
ansible.builtin.apt:
|
|
update_cache: true
|
|
when: node_baseline_backports_source is changed
|
|
|
|
# `state: latest`, NOT `state: present`. linux-image-amd64 is ALREADY installed on these
|
|
# nodes from trixie main (6.12.107-1), so `present` would find the name satisfied and do
|
|
# nothing at all — the single most likely way to write this task and have it silently fix
|
|
# nothing forever. `latest` + `default_release` resolves the candidate inside backports
|
|
# and upgrades the metapackage to it. Once the node is current a re-run is a genuine
|
|
# no-op: no change, no churn, no spurious reboot flag. Scope is exactly
|
|
# node_baseline_backports_kernel_packages — one metapackage — so nothing else on the node
|
|
# is dragged out of the main archive.
|
|
- name: Install the fixed kernel from backports (>= 6.13)
|
|
ansible.builtin.apt:
|
|
name: "{{ node_baseline_backports_kernel_packages }}"
|
|
state: latest
|
|
default_release: "{{ node_baseline_backports_suite }}"
|
|
update_cache: true
|
|
cache_valid_time: 3600
|
|
register: node_baseline_backports_kernel_apt
|
|
|
|
# WHICH VERSION DID THIS NODE ACTUALLY LAND ON. Read back rather than assumed: the
|
|
# metapackage tracks a moving target, so "we installed linux-image-amd64" is not an
|
|
# answer anybody can roll back from six months later.
|
|
- name: Read back the installed backports kernel metapackage version
|
|
ansible.builtin.command: dpkg-query -W -f '${Version}' {{ node_baseline_backports_kernel_packages | first }}
|
|
register: node_baseline_backports_kernel_version
|
|
changed_when: false
|
|
check_mode: false
|
|
|
|
# `linux-version` ships in linux-base and knows Debian's kernel version ordering, which
|
|
# neither `sort -V` nor a Jinja sort gets right across +deb13 / ABI suffixes. The full
|
|
# list is the ROLLBACK INVENTORY: every entry is still installed and selectable from
|
|
# GRUB's "Advanced options" without uninstalling anything.
|
|
- name: List every kernel image installed on this node, newest first
|
|
ansible.builtin.shell:
|
|
cmd: set -o pipefail; linux-version list | linux-version sort --reverse
|
|
executable: /bin/bash
|
|
register: node_baseline_installed_kernels
|
|
changed_when: false
|
|
check_mode: false
|
|
|
|
# The node must be able to explain itself to an operator standing in front of it with no
|
|
# repo checkout. Content is deliberately time-INVARIANT (no timestamp) so re-running the
|
|
# role is a real no-op rather than a permanent daily "changed".
|
|
- name: Record the deviation, the exact version, and the rollback path on the node
|
|
ansible.builtin.copy:
|
|
dest: "{{ node_baseline_backports_kernel_stamp }}"
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
# Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
|
|
#
|
|
# THIS NODE DELIBERATELY DOES NOT RUN THE DEBIAN 13 STOCK KERNEL.
|
|
#
|
|
# WHY
|
|
# This machine's HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table.
|
|
# Linux < 6.13 reserves memory from it without validating its size, and an unlucky
|
|
# boot reserves everything above the table: 2.6 GiB of 16 GB, observed twice on
|
|
# k3s-prod-2. Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size
|
|
# is clearly bogus" (first released in v6.13) adds the validation; it is NOT in
|
|
# v6.12 and NOT in linux-6.12.y. There is NO kernel cmdline workaround.
|
|
# Full write-up: infra/PROD-PROVISIONING.md §0b in the home-network-k3s-impl repo.
|
|
#
|
|
# PROOF THE FIX IS ACTIVE (not merely a lucky boot)
|
|
# BOOT-INDEPENDENT, and this is the one to trust — the string only exists in
|
|
# >= 6.13, so its presence in the RUNNING image is proof regardless of the boot:
|
|
# zstd -dc < <(tail -c +$(...)) ... | grep 'Corrupted EFI Memory Attributes'
|
|
# (roles/node_health does the version check for you; see the health report's
|
|
# firmware.efi_memattr finding.)
|
|
#
|
|
# FROM dmesg, WHICH IS A LOTTERY. efi_memattr_init() on >= 6.13 has THREE gates
|
|
# and each prints a DIFFERENT line — all three are rejections:
|
|
# tbl->version > 2 'Unexpected EFI Memory Attributes table version %d'
|
|
# (NOTE: printed with %d, so garbage may read NEGATIVE)
|
|
# bad desc_size 'Unexpected EFI Memory Attributes descriptor size ...'
|
|
# num_entries > 64K '[Firmware Bug]: Corrupted EFI Memory Attributes
|
|
# Table detected! (version == ..., num_entries == ...)'
|
|
# Which one appears depends on WHICH field this boot's garbage corrupted, so do
|
|
# NOT treat the absence of the FW_BUG line as the fix being absent — all three
|
|
# prod nodes printed the *version* line on 2026-09-04.
|
|
# The one line that means the fix is NOT active:
|
|
# 'Failed to map EFI Memory Attributes table @ 0x...' — the kernel trusted the
|
|
# corrupt table. On a >= 6.13 kernel this should be unreachable; if it appears,
|
|
# escalate, do not shrug.
|
|
#
|
|
# INSTALLED
|
|
# source /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
|
|
# package {{ node_baseline_backports_kernel_packages | join(' ') }}
|
|
# version {{ node_baseline_backports_kernel_version.stdout | default('(not read)') }}
|
|
# kernels installed on this node, newest first:
|
|
{% for kv in (node_baseline_installed_kernels.stdout_lines | default([])) %}
|
|
# {{ kv }}
|
|
{% endfor %}
|
|
#
|
|
# SECURITY COST
|
|
# Packages from *-backports are provided AS IS. They are NOT covered by the Debian
|
|
# Security Team's guarantees for the main archive: no security-tracker commitment,
|
|
# no DSAs. Accepted deliberately, in exchange for the node keeping its RAM.
|
|
#
|
|
# ROLLBACK — no repo checkout needed
|
|
# 1. NON-DESTRUCTIVE, try this first. Every kernel listed above is still installed
|
|
# and bootable: pick the 6.12 one from GRUB -> "Advanced options for Debian
|
|
# GNU/Linux". That answers "is the new kernel the problem?" without
|
|
# uninstalling anything. Note it reinstates the memattr bug for that boot.
|
|
# 2. PERMANENT. Set node_baseline_backports_kernel: false in
|
|
# inventory/group_vars/k3s_prod/vars.yml FIRST — otherwise the next Ansible run
|
|
# puts the backports kernel straight back — then, on this node:
|
|
# apt-get install --reinstall linux-image-amd64/{{ ansible_facts['distribution_release'] }}
|
|
# apt-get purge <the backports linux-image-*+deb13-amd64 listed above>
|
|
# rm /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
|
|
# apt-get update && update-grub && reboot
|
|
#
|
|
# EXIT CONDITION
|
|
# If 8fbe4c49c0cc is ever picked up into linux-6.12.y (and so into Debian's 6.12
|
|
# rebase), this deviation can be removed entirely: flip the variable to false,
|
|
# roll back as above, and delete the `kernel` block from roles/node_baseline.
|
|
|
|
# Say the pending reboot out loud. A green play here does NOT mean the node is fixed —
|
|
# it means the fixed kernel is on disk. See the block header for why the role will not
|
|
# reboot on its own.
|
|
- name: Report whether this node is running the kernel it now has installed
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
{{ ('REBOOT REQUIRED — running ' ~ ansible_facts['kernel'] ~ ', newest installed is '
|
|
~ newest ~ '. The EFI memattr fix is NOT active until this node reboots. Reboot
|
|
prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER
|
|
node 2 first (infra/PROD-PROVISIONING.md §0b "SECOND FAULT"). Afterwards confirm
|
|
with: dmesg | grep -i "Corrupted EFI Memory Attributes Table"')
|
|
if newest != ansible_facts['kernel']
|
|
else ('Running the newest installed kernel (' ~ ansible_facts['kernel']
|
|
~ '); no reboot pending for the memattr fix.') }}
|
|
vars:
|
|
newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
|
|
|
|
# =====================================================================================
|
|
# NODE CLOCK — tag `system`: every node renders time in the SAME zone, and that zone is UTC
|
|
# =====================================================================================
|
|
# MEASURED DRIFT (2026-09-05), read off k3s's own etcd snapshot annotations on live prod:
|
|
# k3s-prod-1 2026-09-05T16:31:28-05:00 Central
|
|
# k3s-prod-2 2026-09-05T16:29:07-05:00 Central
|
|
# k3s-prod-3 2026-09-05T17:27:35-04:00 Eastern <-- a different UTC offset
|
|
# Functionally harmless (in UTC those three are ~4 minutes apart, which is the correct
|
|
# staggering — Kubernetes, etcd and containerd are UTC internally). Operationally it is a
|
|
# trap: any timestamp a node renders in LOCAL time (journald's default output, `ls -l`, an
|
|
# etcd snapshot filename) is not comparable across nodes without knowing each node's zone,
|
|
# and cross-node log correlation during an incident is precisely when nobody knows that.
|
|
#
|
|
# Rationale for UTC rather than standardising on one local zone, and for this being a knob
|
|
# whose override seam is inventory/group_vars/<env>: see node_baseline_timezone in
|
|
# defaults/main.yml. NOT a cluster-vars knob — no Ansible code reads that ConfigMap.
|
|
#
|
|
# IDEMPOTENCE / BLAST RADIUS. community.general.timezone is a converge-to-state module: it
|
|
# reports `changed` only when /etc/localtime + /etc/timezone do not already name this zone,
|
|
# and it does NOT step the clock (no time jump, no NTP interaction) — only the RENDERING of
|
|
# an unchanged monotonic clock moves. No reboot is required. Already-running services that
|
|
# render local time keep the old zone until they are next restarted (the module restarts
|
|
# cron itself, which is the one scheduler on these nodes that cares); k3s, etcd and the
|
|
# kubelet are unaffected either way because they emit UTC regardless.
|
|
- name: Assert this node's timezone has actually been decided
|
|
ansible.builtin.assert:
|
|
that:
|
|
- node_baseline_timezone is string
|
|
- node_baseline_timezone | length > 0
|
|
- "'BLOCKED' not in node_baseline_timezone"
|
|
- node_baseline_timezone is match('^[A-Za-z][A-Za-z0-9+_-]*(/[A-Za-z0-9+_.-]+)*$')
|
|
fail_msg: >-
|
|
node_baseline_timezone is not a usable zoneinfo name (got:
|
|
'{{ node_baseline_timezone | default("<undefined>") }}'). It must be a name
|
|
/usr/share/zoneinfo actually carries — `timedatectl list-timezones` on the node — and
|
|
the repo default (Etc/UTC) is almost certainly what you want; see the rationale in
|
|
roles/node_baseline/defaults/main.yml. Empty is NOT "leave it alone": a node whose zone
|
|
nobody has decided is the config drift this task exists to remove.
|
|
quiet: true
|
|
tags: [system]
|
|
|
|
- name: Pin the node timezone (cross-node log correlation; UTC by default)
|
|
community.general.timezone:
|
|
name: "{{ node_baseline_timezone }}"
|
|
tags: [system]
|
|
|
|
- name: Persist kernel modules for k3s/Cilium/Longhorn
|
|
ansible.builtin.copy:
|
|
dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: "{{ node_baseline_kernel_modules | join('\n') }}\n"
|
|
tags: [system]
|
|
|
|
- name: Load kernel modules now
|
|
community.general.modprobe:
|
|
name: "{{ item }}"
|
|
state: present
|
|
loop: "{{ node_baseline_kernel_modules }}"
|
|
tags: [system]
|
|
|
|
- name: Install sysctl tuning for k3s/Cilium
|
|
ansible.builtin.copy:
|
|
dest: /etc/sysctl.d/99-k3s-cilium.conf
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
net.bridge.bridge-nf-call-iptables = 1
|
|
net.bridge.bridge-nf-call-ip6tables = 1
|
|
net.ipv4.ip_forward = 1
|
|
net.ipv6.conf.all.forwarding = 1
|
|
fs.inotify.max_user_instances = 8192
|
|
fs.inotify.max_user_watches = 524288
|
|
notify: Reload sysctl
|
|
tags: [system]
|
|
|
|
- name: Enable and start iscsid (Longhorn prerequisite)
|
|
ansible.builtin.systemd:
|
|
name: iscsid
|
|
enabled: true
|
|
state: started
|
|
tags: [system]
|
|
|
|
# qemu-guest-agent binds to the virtio-serial channel /dev/virtio-ports/org.qemu.guest_agent.0,
|
|
# which Proxmox attaches only when the VM's `agent` option is enabled. Our tofu keeps agent
|
|
# DISABLED on purpose (enabling it makes `tofu apply` block up to 15m waiting for an agent the
|
|
# stock Debian cloud image hasn't installed yet — bpg gotcha). So we ENABLE the unit for the day
|
|
# a channel exists, but only START it when the channel is actually present — never fail the
|
|
# baseline over a missing hypervisor channel.
|
|
- name: Check whether the qemu guest-agent virtio channel is attached
|
|
ansible.builtin.stat:
|
|
path: /dev/virtio-ports/org.qemu.guest_agent.0
|
|
register: node_baseline_qga_channel
|
|
tags: [system]
|
|
|
|
- name: Enable qemu-guest-agent (start only when its virtio channel is present)
|
|
ansible.builtin.systemd:
|
|
name: qemu-guest-agent
|
|
enabled: true
|
|
state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
|
|
tags: [system]
|
|
|
|
# =====================================================================================
|
|
# WIRELESS — tag `wireless`: a cluster node gets exactly ONE path onto the network
|
|
# =====================================================================================
|
|
# WHAT WENT WRONG (k3s-prod-3, reported by the operator 2026-09-04). The node was up on BOTH
|
|
# eno1 (192.168.2.12 — the inventory address) and wlp1s0 (192.168.2.195, DHCP, the SAME /24).
|
|
# The wifi lease carried a dhcpcd run with it, and dhcpcd's `20-resolv.conf` hook REWROTE
|
|
# /etc/resolv.conf, undoing the resolvers this role had just written: the file read
|
|
# "# Generated by dhcpcd from wlp1s0.dhcp, wlp1s0.dhcp6, wlp1s0.ra" with nameserver
|
|
# 192.168.2.1 plus two Google IPv6 servers, timestamped THREE MINUTES AFTER the role's own
|
|
# write (01:32 -> 01:35). The role had reported `changed` and green. That is exactly the
|
|
# "successful run that fixed nothing" the `contended` branch of the DNS block below exists to
|
|
# prevent, and the detection blind spot that let it through is closed there.
|
|
#
|
|
# WHY THE INTERFACE IS REMOVED RATHER THAN JUST REPORTED. Closing the detection hole alone
|
|
# would have turned a silent revert into a loud failure — an improvement, but it leaves the
|
|
# node with a second, DHCP-managed path onto the LAN, and on a k3s server that is three
|
|
# distinct problems beyond DNS:
|
|
# * a SECOND DEFAULT ROUTE, whose metric decides which source address the node uses;
|
|
# * a SECOND ADDRESS ON THE SAME L2 — the segment where kube-vip floats 192.168.2.9 and
|
|
# Cilium L2-announces 192.168.2.20-.30 (§4). One host answering ARP on one subnet from
|
|
# two NICs is the exact shape neither mechanism can reason about;
|
|
# * an UNMANAGED address: nothing in this repo knows about .195, and the DHCP server is
|
|
# free to hand out a different one after any lease expiry.
|
|
# None of that is a resolver bug, and none of it is fixed by writing resolv.conf harder.
|
|
#
|
|
# WHY THE DRIVER LAYER — the four alternatives, and why each is weaker. What has to hold is:
|
|
# survives a REBOOT, survives a LEASE RENEWAL, and cannot be undone by a routine `ifup`.
|
|
# Measured on the nodes themselves — all three are HP EliteDesk 800 G5 minis, eno1 is
|
|
# `e1000e`, the wifi NIC is an Intel AX200 on `iwlwifi`, and the OS is Debian 13 trixie with
|
|
# ifupdown + `networking.service`, NO NetworkManager, NO systemd-networkd, NO systemd-resolved:
|
|
# `ip link set <wifi> down` — NOT persistent. Any `ifup`, any hotplug event, any reboot
|
|
# undoes it. It is what had already been done by hand on node 3, and the stale dhcpcd
|
|
# /etc/resolv.conf outlived it — the link was down and the wrong resolvers were still
|
|
# in the file, across a reboot.
|
|
# mask the service that manages it — THERE IS NO SUCH SERVICE HERE. dhcpcd on these nodes
|
|
# is `dhcpcd-base`: no dhcpcd.service exists at all (`systemctl is-enabled dhcpcd` =
|
|
# not-found), it is spawned per-interface by ifupdown. The only unit that can bring
|
|
# wireless up is `networking.service` — which ALSO owns eno1's static address, so
|
|
# masking it would cut the one SSH path to a machine whose operator is not on site.
|
|
# `rfkill` soft-block — weaker on every axis. The netdev still EXISTS, so ifupdown and
|
|
# dhcpcd can still be pointed at it; persistence depends on systemd-rfkill restoring
|
|
# saved state (the kernel's own `persistent` flag reads 0 on these devices, and the
|
|
# `rfkill` CLI is not even installed); and one `rfkill unblock wlan` undoes it.
|
|
# the BIOS `M.2 WLAN/BT` toggle — genuinely the deepest layer, and it reads `Enable` on all
|
|
# three nodes (hp-bioscfg survey, §1b). NOT taken here: it needs a REBOOT to take
|
|
# effect, and a prod reboot is a serialized, supervised operator step (§0b "SECOND
|
|
# FAULT" — k3s-prod-2 hung on a warm reboot and needed the power button). It also
|
|
# disables Bluetooth on the same M.2 card. Left as an operator recommendation in §4.
|
|
# THE DRIVER MODULE — chosen. `blacklist` plus `install <module> /bin/false` in
|
|
# /etc/modprobe.d means kmod refuses to load the wifi driver on this boot and every
|
|
# future one, so THE NETDEV NEVER EXISTS. No interface means no DHCP lease, no
|
|
# resolv.conf hook, no second route, no second address — and every re-activation path
|
|
# above (udev hotplug, a hand-run `ifup`, wpa_supplicant, a hand-run dhcpcd) needs a
|
|
# device that is not there. `blacklist` alone would still allow a dependency- or
|
|
# hand-triggered `modprobe`; the `install ... /bin/false` line is what makes it refuse
|
|
# outright. It is ONE declarative file, readable on the node, and undone by deleting
|
|
# it. Paired with the runtime unload below, it needs NO reboot to take effect.
|
|
#
|
|
# SAFE AND IDEMPOTENT ON ALL THREE PROD NODES *AND* ON THE LIVE DEV CLUSTER. Nothing here is
|
|
# hostname- or hardware-specific: the block acts only on interfaces the node itself reports as
|
|
# real 802.11 devices (a `phy80211` link in sysfs) and on the driver modules behind them.
|
|
# * dev (Proxmox VMs, virtio NICs) has no such interface, so the discovery comes back empty,
|
|
# every task below is skipped, and NOT EVEN THE modprobe.d FILE IS WRITTEN. The block
|
|
# cannot touch dev's networking, which is why it is safe to have on by default.
|
|
# * k3s-prod-1 (wlp1s0) and k3s-prod-2 (wlp2s0) already have the link DOWN, so the link-down
|
|
# task is skipped by its own `when` and only the modprobe.d file and the module unload
|
|
# remain — and both are no-ops on a re-run.
|
|
# * k3s-prod-3 (wlp1s0) is the reported node.
|
|
# A re-run changes nothing on any of them: the file is content-identical, `ip link set down`
|
|
# is gated on the link not already being down, and the unload is gated on the module still
|
|
# being loaded.
|
|
#
|
|
# AND IT REFUSES TO CUT ITS OWN PATH IN. The assert below stops the play if the address
|
|
# Ansible is connected to is bound to a wireless interface, or if the default route leaves
|
|
# over one, or if the wireless driver module is also the driver of the default-route
|
|
# interface. That is the one case where doing this would lock everybody out of a machine
|
|
# nobody can walk up to, so it is checked per node, on every run, from that node's own facts.
|
|
- name: Survey this node's 802.11 devices, their drivers and any live DHCP client
|
|
ansible.builtin.shell:
|
|
executable: /bin/sh
|
|
cmd: |
|
|
set -u
|
|
# Flat KEY|value lines so the Ansible side needs one single-group regex per fact and
|
|
# no parsing cleverness. A wireless interface is one with a phy80211 link in sysfs —
|
|
# that is the kernel's own answer to "is this 802.11", not a name pattern like wlp*.
|
|
wifi=''
|
|
for d in /sys/class/net/*; do
|
|
[ -e "$d/phy80211" ] || continue
|
|
i=$(basename "$d")
|
|
wifi="$wifi $i"
|
|
m=$(basename "$(readlink -f "$d/device/driver" 2>/dev/null || echo unknown)")
|
|
state=$(cat "$d/operstate" 2>/dev/null || echo unknown)
|
|
printf 'IFACE|%s\n' "$i"
|
|
printf 'DETAIL|%s driver=%s operstate=%s\n' "$i" "$m" "$state"
|
|
# LIVE means "not already down" — the link-down task keys off this, so a node whose
|
|
# radio is already down reports no change.
|
|
[ "$state" = down ] || printf 'IFACE_LIVE|%s\n' "$i"
|
|
ip -o addr show dev "$i" 2>/dev/null | awk '{print "IFACE_ADDR|" $4}'
|
|
[ "$m" = unknown ] || printf 'MODULE|%s\n' "$m"
|
|
done
|
|
for m in $(printf '%s\n' $wifi | while read -r i; do
|
|
[ -n "$i" ] && basename "$(readlink -f "/sys/class/net/$i/device/driver" 2>/dev/null || echo unknown)"
|
|
done | sort -u); do
|
|
[ "$m" = unknown ] && continue
|
|
[ -d "/sys/module/$m" ] || continue
|
|
printf 'MODULE_LOADED|%s\n' "$m"
|
|
# Holders must be unloaded FIRST: `modprobe -r iwlwifi` fails while iwlmvm still
|
|
# holds it. Read from sysfs rather than assumed, so a different wifi driver stack
|
|
# (ath*, rtw*, mt76*) works without editing this role.
|
|
for h in $(ls "/sys/module/$m/holders" 2>/dev/null); do printf 'MODULE_HOLDER|%s\n' "$h"; done
|
|
done
|
|
dr=$(ip -o route show default 2>/dev/null | awk '{for (i = 1; i < NF; i++) if ($i == "dev") print $(i + 1)}' | head -1)
|
|
printf 'DEFROUTE_IFACE|%s\n' "${dr:-none}"
|
|
if [ -n "${dr:-}" ] && [ -e "/sys/class/net/$dr" ]; then
|
|
printf 'DEFROUTE_MODULE|%s\n' "$(basename "$(readlink -f "/sys/class/net/$dr/device/driver" 2>/dev/null || echo unknown)")"
|
|
fi
|
|
# Any DHCP client process, with the interfaces it was started for. Used only to stop a
|
|
# client that holds a WIRELESS interface and nothing else.
|
|
for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
|
|
printf 'DHCPCLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
|
|
done
|
|
exit 0
|
|
register: node_baseline_wifi_scan
|
|
changed_when: false
|
|
check_mode: false
|
|
when: node_baseline_disable_wireless | bool
|
|
tags: [wireless]
|
|
|
|
- name: Interpret the wireless survey
|
|
ansible.builtin.set_fact:
|
|
node_baseline_wifi_ifaces: "{{ scan | regex_findall('(?m)^IFACE\\|(.+)$') }}"
|
|
node_baseline_wifi_live: "{{ scan | regex_findall('(?m)^IFACE_LIVE\\|(.+)$') }}"
|
|
node_baseline_wifi_addrs: "{{ scan | regex_findall('(?m)^IFACE_ADDR\\|(\\S+)') | map('regex_replace', '/.*$', '') | list }}"
|
|
node_baseline_wifi_modules: "{{ scan | regex_findall('(?m)^MODULE\\|(.+)$') | unique }}"
|
|
node_baseline_wifi_modules_loaded: "{{ scan | regex_findall('(?m)^MODULE_LOADED\\|(.+)$') | unique }}"
|
|
node_baseline_wifi_module_holders: "{{ scan | regex_findall('(?m)^MODULE_HOLDER\\|(.+)$') | unique }}"
|
|
node_baseline_wifi_detail: "{{ scan | regex_findall('(?m)^DETAIL\\|(.+)$') }}"
|
|
node_baseline_wifi_dhcp_clients: "{{ scan | regex_findall('(?m)^DHCPCLIENT\\|(.+)$') }}"
|
|
node_baseline_defroute_iface: "{{ (scan | regex_findall('(?m)^DEFROUTE_IFACE\\|(.+)$') | first) | default('none', true) }}"
|
|
node_baseline_defroute_module: "{{ (scan | regex_findall('(?m)^DEFROUTE_MODULE\\|(.+)$') | first) | default('unknown', true) }}"
|
|
vars:
|
|
scan: "{{ node_baseline_wifi_scan.stdout | default('') }}"
|
|
when: node_baseline_disable_wireless | bool
|
|
tags: [wireless]
|
|
|
|
- name: Report this node's wireless hardware
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
wireless devices = {{ node_baseline_wifi_detail | default([]) | join('; ') | default('(none)', true) }}
|
|
| driver modules = {{ node_baseline_wifi_modules | default([]) | join(', ') | default('(none)', true) }}
|
|
(loaded: {{ node_baseline_wifi_modules_loaded | default([]) | join(', ') | default('(none)', true) }},
|
|
held by: {{ node_baseline_wifi_module_holders | default([]) | join(', ') | default('(none)', true) }})
|
|
| default route via {{ node_baseline_defroute_iface | default('?') }}
|
|
(driver {{ node_baseline_defroute_module | default('?') }})
|
|
when: node_baseline_disable_wireless | bool
|
|
tags: [wireless]
|
|
|
|
# The one way this block could do real harm is by removing the interface it is being run
|
|
# over. Checked from the node's own facts, per node, every run — not assumed from the fact
|
|
# that prod "should" be wired.
|
|
- name: Refuse to disable wireless if the path to this node runs OVER the wireless interface
|
|
ansible.builtin.assert:
|
|
that:
|
|
- node_baseline_defroute_iface not in node_baseline_wifi_ifaces
|
|
- node_baseline_defroute_module not in node_baseline_wifi_modules
|
|
- (ansible_host | default(inventory_hostname)) not in node_baseline_wifi_addrs
|
|
fail_msg: >-
|
|
REFUSING to disable wireless on this node: the connection to it appears to BE the
|
|
wireless path. ansible_host={{ ansible_host | default(inventory_hostname) }}, wireless
|
|
interfaces={{ node_baseline_wifi_ifaces | join(', ') }} carrying
|
|
{{ node_baseline_wifi_addrs | join(', ') | default('no addresses', true) }}, default route
|
|
via {{ node_baseline_defroute_iface }} (driver {{ node_baseline_defroute_module }}).
|
|
Disabling the driver here would drop this node off the network with no remote way back.
|
|
Give the node a wired address, point the inventory at it, and re-run.
|
|
quiet: true
|
|
when:
|
|
- node_baseline_disable_wireless | bool
|
|
- node_baseline_wifi_ifaces | length > 0
|
|
tags: [wireless]
|
|
|
|
# THE PERSISTENT HALF. This file is the whole reboot story: kmod consults /etc/modprobe.d on
|
|
# every load attempt, so from here on the driver cannot be loaded by udev at boot, as a
|
|
# dependency, or by hand. Written ONLY when the node actually has a wireless driver, so a
|
|
# node without wifi hardware (every dev VM) gets no file and no change at all.
|
|
# `update-initramfs` is deliberately NOT run: no wifi module is loaded from the initramfs on
|
|
# these nodes (root is a local SATA/NVMe device), the driver is autoloaded by udev after
|
|
# switch-root, and re-generating an initramfs on a prod node is a bigger risk than the one it
|
|
# would cover.
|
|
- name: Block the wireless driver module(s) from ever loading again
|
|
ansible.builtin.copy:
|
|
dest: "{{ node_baseline_wireless_modprobe_file }}"
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
# Managed by the Ansible node_baseline role (tags: wireless) — do not edit by hand.
|
|
#
|
|
# WHY THIS NODE HAS NO WIRELESS. A k3s server gets exactly ONE path onto the network.
|
|
# A second, DHCP-managed interface on the same LAN gives the node a second default
|
|
# route, a second address nothing in the cluster's config knows about, and a DHCP
|
|
# client whose resolv.conf hook silently overwrites the resolvers Ansible manages
|
|
# (that is what happened on k3s-prod-3, 2026-09-04). The link is also brought down and
|
|
# the module unloaded at the same time, so no reboot is needed for this to take effect.
|
|
#
|
|
# `blacklist` stops udev's alias-driven autoload; the `install ... /bin/false` line is
|
|
# what also refuses a dependency-triggered or hand-run `modprobe`. Both are needed.
|
|
#
|
|
# TO RE-ENABLE WIRELESS ON THIS NODE: delete this file and `modprobe` the module(s)
|
|
# below (or reboot). To stop Ansible putting it back, set
|
|
# node_baseline_disable_wireless: false for this host first — otherwise the next
|
|
# node-baseline run recreates it. Full rationale: roles/node_baseline/tasks/main.yml,
|
|
# the `wireless` block, and infra/PROD-PROVISIONING.md §4.
|
|
{% for module in node_baseline_wifi_modules %}
|
|
blacklist {{ module }}
|
|
install {{ module }} /bin/false
|
|
{% endfor %}
|
|
when:
|
|
- node_baseline_disable_wireless | bool
|
|
- node_baseline_wifi_modules | length > 0
|
|
tags: [wireless]
|
|
|
|
# THE RUNTIME HALF — everything below makes the change true NOW, so the node does not have to
|
|
# be rebooted to stop carrying a second address (this role never reboots: §0b "SECOND FAULT").
|
|
#
|
|
# Stop a DHCP client that is holding a wireless interface first, so it releases its lease
|
|
# instead of being left holding a device that is about to disappear. HEAVILY GUARDED: it runs
|
|
# only for clients whose command line names a wireless interface and names NO non-wireless
|
|
# interface, because `dhcpcd -x` against a shared master process would also drop a wired
|
|
# lease. On these prod nodes eno1 is a static ifupdown address with no DHCP client at all, so
|
|
# this task has nothing to do; it exists so the ordering is correct on a node where wifi is
|
|
# still actively leasing.
|
|
- name: Stop the dhcpcd instance that holds a wireless interface (and nothing else)
|
|
ansible.builtin.command:
|
|
argv: [dhcpcd, --exit, "{{ item }}"]
|
|
loop: "{{ node_baseline_wifi_ifaces }}"
|
|
register: node_baseline_wifi_dhcpcd_exit
|
|
changed_when: rc == 0
|
|
failed_when: false
|
|
when:
|
|
- node_baseline_disable_wireless | bool
|
|
- node_baseline_wifi_dhcp_wifi_only | length > 0
|
|
- node_baseline_wifi_dhcp_wifi_only | select('search', 'dhcpcd') | list | length > 0
|
|
vars:
|
|
# A client is "wireless-only" when its argv mentions a wireless interface and mentions no
|
|
# other interface this node has. Empty interface sets are turned into a pattern that
|
|
# cannot match, so an odd node fails towards doing nothing rather than towards guessing.
|
|
node_baseline_non_wifi_ifaces: "{{ ansible_facts['interfaces'] | default([]) | difference(node_baseline_wifi_ifaces) | difference(['lo']) }}"
|
|
node_baseline_wifi_dhcp_wifi_only: >-
|
|
{{ node_baseline_wifi_dhcp_clients
|
|
| select('search', '\b(' ~ ((node_baseline_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
|
|
| reject('search', '\b(' ~ ((node_baseline_non_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
|
|
| list }}
|
|
tags: [wireless]
|
|
|
|
- name: Take each wireless link down now
|
|
ansible.builtin.command:
|
|
argv: [ip, link, set, dev, "{{ item }}", down]
|
|
loop: "{{ node_baseline_wifi_live }}"
|
|
changed_when: true
|
|
when: node_baseline_disable_wireless | bool
|
|
tags: [wireless]
|
|
|
|
# Unload the driver so the netdev is gone immediately — holders (iwlmvm on these nodes) first,
|
|
# then the driver itself, because `modprobe -r` refuses a module that is still held.
|
|
# NOT fatal on failure: the persistent block above is already in place, so a module that
|
|
# cannot be unloaded right now is gone at the next boot regardless. The report below says so
|
|
# out loud rather than letting a green play imply the radio is already off.
|
|
- name: Unload the wireless driver module(s) now (no reboot needed)
|
|
community.general.modprobe:
|
|
name: "{{ item }}"
|
|
state: absent
|
|
loop: "{{ node_baseline_wifi_module_holders + node_baseline_wifi_modules_loaded }}"
|
|
register: node_baseline_wifi_unload
|
|
failed_when: false
|
|
when: node_baseline_disable_wireless | bool
|
|
tags: [wireless]
|
|
|
|
- name: Report the wireless state this node is left in
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
{{ 'No 802.11 device on this node — nothing to disable (no modprobe.d file written).'
|
|
if (node_baseline_wifi_ifaces | length) == 0
|
|
else ((('WIRELESS DISABLED: ' ~ (node_baseline_wifi_modules | join(', ')) ~ ' blocked in '
|
|
~ node_baseline_wireless_modprobe_file ~ ' (survives reboot) and unloaded now; '
|
|
~ (node_baseline_wifi_ifaces | join(', ')) ~ ' is down and the netdev is gone.')
|
|
if (failed_unloads | length) == 0
|
|
else ('WIRELESS BLOCKED FOR THE NEXT BOOT BUT STILL LOADED NOW: could not unload '
|
|
~ (failed_unloads | join(', ')) ~ '. ' ~ node_baseline_wireless_modprobe_file
|
|
~ ' is in place, so the driver will not load again after a reboot, but the '
|
|
~ 'radio is live until then — check `ip link` and whether something still '
|
|
~ 'holds the device.'))) }}
|
|
vars:
|
|
failed_unloads: >-
|
|
{{ (node_baseline_wifi_unload.results | default([]))
|
|
| selectattr('failed', 'defined') | selectattr('failed') | map(attribute='item') | list }}
|
|
when: node_baseline_disable_wireless | bool
|
|
tags: [wireless]
|
|
|
|
# =====================================================================================
|
|
# DATA DISKS — tag `disks`, in tasks/disks.yml
|
|
# =====================================================================================
|
|
# Split into its own file for two reasons. It is the only DESTRUCTIVE part of the role
|
|
# (prod re-purposes the partition already mounted at /srv as the Longhorn store), so it
|
|
# should be readable end-to-end on its own; and both environments' storage shapes — dev's
|
|
# two dedicated virtio disks, prod's single partitioned physical disk with etcd left on
|
|
# the OS filesystem — are declared there as an explicit enum rather than inferred from
|
|
# which variables happen to be set. See tasks/disks.yml and infra/PROD-PROVISIONING.md §2.
|
|
#
|
|
# The `disks` tag is applied ONCE here, in BOTH places it is needed:
|
|
# * `tags:` on the include itself, so `--tags packages` never even reaches the include
|
|
# and a placeholder device path is not so much as templated; and
|
|
# * `apply: tags:`, so the tag reaches the tasks INSIDE the file. This half is not
|
|
# optional and not decoration — a DYNAMIC include (`include_tasks`) does NOT propagate
|
|
# its own tags to its children the way `import_tasks` does, so without `apply` the
|
|
# included tasks are untagged and `--tags disks` selects the include and then runs
|
|
# NOTHING inside it. That failure is silent and reports success, which is the worst
|
|
# shape a guard can have.
|
|
# A FULL run against a node whose layout is undeclared stops on the assert at the top of
|
|
# that file instead of failing obscurely at mkfs time.
|
|
- name: Provision the data disks (Longhorn store, etcd data dir)
|
|
ansible.builtin.include_tasks:
|
|
file: disks.yml
|
|
apply:
|
|
tags: [disks]
|
|
tags: [disks]
|
|
|
|
- name: Disable swap now
|
|
ansible.builtin.command: swapoff -a
|
|
changed_when: false
|
|
tags: [system]
|
|
|
|
- name: Neutralize swap entries in fstab (kubelet requires swap off)
|
|
ansible.builtin.replace:
|
|
path: /etc/fstab
|
|
regexp: '^([^#].*\sswap\s.*)$'
|
|
replace: '# \1 # disabled for k3s/kubelet'
|
|
tags: [system]
|
|
|
|
# =====================================================================================
|
|
# DNS RESILIENCE — tag `dns` (R-DNS-01 / R-DNS-03)
|
|
# =====================================================================================
|
|
# Root cause (verified on dev): the nodes carried a SINGLE upstream resolver. PiHole
|
|
# (192.168.1.18) answers `*.jackson.lan` instantly but its external recursion flaps —
|
|
# 0/25 for registry-1.docker.io / quay.io / ghcr.io while gitea.jackson.lan resolved 5/5.
|
|
# One upstream is a hard SPOF: containerd image pulls died with "lookup
|
|
# registry-1.docker.io: Try again", and CoreDNS (`forward . /etc/resolv.conf`) flooded its
|
|
# log with "i/o timeout" to 192.168.1.18. The fix is ADDITIVE — PiHole stays first so the
|
|
# local zone and the ad-blocking policy are untouched; a second upstream is only consulted
|
|
# when the first fails. See defaults/main.yml for the ordering rule.
|
|
#
|
|
# WHY THIS DETECTS INSTEAD OF ASSUMING. An earlier attempt (fix/node-dns-resilient-upstream,
|
|
# dropped) wrote /etc/netplan/99-k3s-dns.yaml and ran `netplan apply`. netplan is NOT
|
|
# installed on these nodes — they run the Debian 13 trixie generic cloud image
|
|
# (infra/tofu/variables.tf) and netplan is not in node_baseline_packages — so that handler
|
|
# would have failed command-not-found and broken node-baseline.yml on EVERY node. Worse,
|
|
# the two environments do not even share a resolver stack: dev's /etc/resolv.conf is the
|
|
# systemd-resolved stub (127.0.0.53) while prod's is a directly-managed file listing
|
|
# 192.168.2.1 (it has no cloud-init at all — bare metal). So the mechanism is DETECTED per
|
|
# node and the matching seam is written. Adding a third mechanism means adding a branch
|
|
# here, not rewriting the role.
|
|
#
|
|
# AND DETECTION IS ONLY AS GOOD AS THE SET OF OWNERS IT KNOWS ABOUT. It originally looked for
|
|
# resolvconf/openresolv and NetworkManager, missed dhcpcd, and therefore reported a green,
|
|
# `changed` run on k3s-prod-3 whose resolvers a wifi lease reverted three minutes later. dhcpcd
|
|
# is now part of that set — see the detection task below and the `contended` failure.
|
|
#
|
|
# All of it is `dns`-tagged, so `--tags packages` (the prod package-floor escape hatch
|
|
# documented at the top of this file) still installs nothing but packages.
|
|
|
|
- name: Determine whether systemd-resolved is running on this node
|
|
ansible.builtin.command: systemctl is-active systemd-resolved
|
|
register: node_baseline_resolved_active
|
|
changed_when: false
|
|
failed_when: false
|
|
check_mode: false
|
|
tags: [dns]
|
|
|
|
- name: Determine whether NetworkManager is running (it rewrites /etc/resolv.conf)
|
|
ansible.builtin.command: systemctl is-active NetworkManager
|
|
register: node_baseline_nm_active
|
|
changed_when: false
|
|
failed_when: false
|
|
check_mode: false
|
|
tags: [dns]
|
|
|
|
# THE dhcpcd BLIND SPOT, CLOSED (k3s-prod-3, 2026-09-04). dhcpcd's `20-resolv.conf` hook
|
|
# rewrites /etc/resolv.conf on every lease, renewal and carrier change, and it is a THIRD
|
|
# way to own that file — neither resolvconf nor NetworkManager. On node 3 it silently reverted
|
|
# this role's write three minutes later, and because nothing below looked for it the role had
|
|
# already reported `changed` and green: the precise failure the `contended` branch exists to
|
|
# make impossible.
|
|
#
|
|
# WHY THIS LOOKS FOR A LIVE CLIENT AND NOT FOR AN INSTALLED PACKAGE. `dhcpcd-base` is
|
|
# installed on all three prod nodes and always will be — it is the DHCP client ifupdown
|
|
# reaches for — and there is NO dhcpcd.service on them at all (`systemctl is-active dhcpcd` =
|
|
# not-found; it is spawned per-interface by ifupdown). Keying off the package, or off a unit
|
|
# that does not exist, would put every prod node permanently in `contended` and leave the DNS
|
|
# block unable to configure a node it is perfectly safe to configure. What actually matters is
|
|
# whether a DHCP client CAN still write the file, so all three real signals are checked:
|
|
# CLIENT| a dhcpcd/dhclient process running right now (with the argv that started it);
|
|
# UNIT| a dhcpcd/dhclient unit that IS active or IS enabled, on a distro that has one;
|
|
# STANZA| an UNCOMMENTED `iface <x> inet[6] dhcp` in /etc/network/interfaces{,.d} — the
|
|
# configuration that makes ifupdown spawn a client at the next boot or `ifup`, even
|
|
# when none is running at this instant.
|
|
# Any one of them means "another manager is in play" and the play stops below. None of them
|
|
# means no client can run, and the file is genuinely ours to own.
|
|
- name: Determine whether a DHCP client (dhcpcd/dhclient) can rewrite /etc/resolv.conf
|
|
ansible.builtin.shell:
|
|
executable: /bin/sh
|
|
cmd: |
|
|
set -u
|
|
for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
|
|
printf 'CLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
|
|
done
|
|
for u in dhcpcd dhcpcd5 dhclient isc-dhcp-client; do
|
|
a=$(systemctl is-active "$u" 2>/dev/null || true)
|
|
e=$(systemctl is-enabled "$u" 2>/dev/null || true)
|
|
# Compared exactly, never with a substring match: `is-active` answers "inactive",
|
|
# which contains the word "active".
|
|
if [ "$a" = active ] || [ "$e" = enabled ] || [ "$e" = enabled-runtime ]; then
|
|
printf 'UNIT|%s active=%s enabled=%s\n' "$u" "$a" "$e"
|
|
fi
|
|
done
|
|
# `^[[:space:]]*iface` cannot match a commented-out stanza, which is how the wireless
|
|
# stanza is left on these nodes.
|
|
grep -rhE '^[[:space:]]*iface[[:space:]]+[^[:space:]]+[[:space:]]+inet6?[[:space:]]+dhcp' \
|
|
/etc/network/interfaces /etc/network/interfaces.d/ 2>/dev/null \
|
|
| sed -e 's/^[[:space:]]*/STANZA|/'
|
|
exit 0
|
|
register: node_baseline_dhcp_client_scan
|
|
changed_when: false
|
|
check_mode: false
|
|
tags: [dns]
|
|
|
|
# follow: false — we need to know whether /etc/resolv.conf IS a symlink and where it
|
|
# points, which is the strongest single signal of who owns the file.
|
|
- name: Inspect /etc/resolv.conf itself (symlink or real file, and its target)
|
|
ansible.builtin.stat:
|
|
path: /etc/resolv.conf
|
|
follow: false
|
|
register: node_baseline_resolv_stat
|
|
tags: [dns]
|
|
|
|
- name: Read the current /etc/resolv.conf contents
|
|
ansible.builtin.slurp:
|
|
src: /etc/resolv.conf
|
|
register: node_baseline_resolv_slurp
|
|
failed_when: false
|
|
tags: [dns]
|
|
|
|
# resolvconf/openresolv keeps the authoritative copy under here and regenerates
|
|
# /etc/resolv.conf from it, so writing /etc/resolv.conf directly would be silently undone.
|
|
- name: Detect a resolvconf/openresolv installation that would clobber /etc/resolv.conf
|
|
ansible.builtin.stat:
|
|
path: /etc/resolvconf/resolv.conf.d
|
|
register: node_baseline_resolvconf_dir
|
|
tags: [dns]
|
|
|
|
# THE DETECTION. Three outcomes, in priority order:
|
|
# systemd-resolved — resolved is running AND it owns /etc/resolv.conf, proven either by
|
|
# the symlink pointing into /run/systemd/resolve/ or by the file containing only
|
|
# resolved's loopback stub (127.0.0.53 / 127.0.0.54). Writing a resolved drop-in is
|
|
# then the ONLY thing that changes real behaviour; writing resolv.conf would be
|
|
# overwritten by resolved (or, worse, land in /run and vanish at reboot).
|
|
# contended — nobody's stub, but resolvconf, NetworkManager or a DHCP client (dhcpcd /
|
|
# dhclient) is present/running/configured and would regenerate the file underneath us.
|
|
# We do NOT write. The role fails loudly below rather than leaving an operator
|
|
# believing a change stuck when it did not.
|
|
# resolv-conf-file — a plain, directly-managed /etc/resolv.conf (prod's shape today).
|
|
# Safe to own outright.
|
|
# resolved running but NOT owning resolv.conf correctly falls through to the file branch:
|
|
# in that state the drop-in would change nothing glibc can see.
|
|
- name: Detect which resolver mechanism this node actually uses
|
|
ansible.builtin.set_fact:
|
|
node_baseline_dns_mechanism: >-
|
|
{{
|
|
'systemd-resolved' if (node_baseline_resolved_running and node_baseline_resolved_owns_resolv)
|
|
else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present
|
|
or node_baseline_dhcp_client_live)
|
|
else 'resolv-conf-file')
|
|
}}
|
|
# Kept as facts (not just task vars) so the failure message, the report and the
|
|
# reclaim note below all quote the SAME evidence this decision was made on.
|
|
node_baseline_dhcp_client_evidence: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) | regex_findall('(?m)^(?:CLIENT|UNIT|STANZA)\\|(.+)$') }}"
|
|
# A file dhcpcd WROTE, versus a dhcpcd that can still write it — two different facts, and
|
|
# only the second one is contention. See the reclaim note below.
|
|
node_baseline_resolv_dhcpcd_written: "{{ 'Generated by dhcpcd' in (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
|
|
vars:
|
|
node_baseline_resolved_running: "{{ (node_baseline_resolved_active.stdout | default('') | trim) == 'active' }}"
|
|
node_baseline_nm_running: "{{ (node_baseline_nm_active.stdout | default('') | trim) == 'active' }}"
|
|
node_baseline_resolvconf_present: "{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}"
|
|
node_baseline_dhcp_client_live: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) is search('(?m)^(?:CLIENT|UNIT|STANZA)\\|') }}"
|
|
node_baseline_resolv_text: "{{ (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
|
|
node_baseline_resolv_ns: "{{ node_baseline_resolv_text | regex_findall('(?m)^\\s*nameserver\\s+(\\S+)') }}"
|
|
node_baseline_resolved_owns_resolv: >-
|
|
{{ ('/run/systemd/resolve/' in (node_baseline_resolv_stat.stat.lnk_source | default('')))
|
|
or ((node_baseline_resolv_ns | length) > 0
|
|
and (node_baseline_resolv_ns | reject('in', ['127.0.0.53', '127.0.0.54']) | list | length) == 0) }}
|
|
tags: [dns]
|
|
|
|
- name: Report the detected resolver mechanism
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
resolver mechanism = {{ node_baseline_dns_mechanism }}
|
|
(systemd-resolved is-active={{ node_baseline_resolved_active.stdout | default('') | trim }},
|
|
NetworkManager is-active={{ node_baseline_nm_active.stdout | default('') | trim }},
|
|
/etc/resolv.conf islnk={{ node_baseline_resolv_stat.stat.islnk | default(false) }}
|
|
target={{ node_baseline_resolv_stat.stat.lnk_source | default('(not a symlink)') }},
|
|
resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }},
|
|
DHCP client evidence={{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }},
|
|
resolv.conf written by dhcpcd={{ node_baseline_resolv_dhcpcd_written | default(false) }})
|
|
tags: [dns]
|
|
|
|
# Same shape, and the same reason, as the data-disk assert above: this environment's
|
|
# upstream list is an OPERATOR DECISION, and a placeholder that fails LOUDLY beats both a
|
|
# guess and a silent inheritance of dev's PiHole. group_vars/k3s_prod/vars.yml deliberately
|
|
# carries a BLOCKED second entry (see its comments and PROD-PROVISIONING.md §4): prod's
|
|
# 192.168.2.1 is currently gateway AND only upstream, and choosing what sits behind it is a
|
|
# network decision this role must not make. Running against prod stops HERE, before
|
|
# anything is written.
|
|
- name: Assert this environment's DNS upstream list has actually been decided
|
|
ansible.builtin.assert:
|
|
that:
|
|
- node_baseline_dns_servers | length >= 2
|
|
- "'BLOCKED' not in (node_baseline_dns_servers | join(' '))"
|
|
fail_msg: >-
|
|
node_baseline_dns_servers is not a decided, redundant upstream list for this host's
|
|
environment (got: {{ node_baseline_dns_servers | join(', ') }}). A single upstream is
|
|
the exact SPOF this task exists to remove — it already broke containerd image pulls
|
|
and CoreDNS forwarding on dev — so the role will not configure one, and it will NOT
|
|
guess a second server: a public resolver would bypass the local zone and the
|
|
ad-blocking policy, which is a decision, not a default. Decide the second upstream,
|
|
record it in infra/PROD-PROVISIONING.md §4, and set node_baseline_dns_servers in
|
|
inventory/group_vars/k3s_prod/vars.yml in the SAME change. To install only the
|
|
package floor meanwhile, re-run with `--tags packages`, which skips every DNS task.
|
|
quiet: true
|
|
tags: [dns]
|
|
|
|
# The `contended` outcome. Not a warning: an operator who is told "changed" while another
|
|
# daemon quietly reverts the file is worse off than one who is told nothing happened.
|
|
- name: Refuse to manage /etc/resolv.conf while another resolver manager owns it
|
|
ansible.builtin.fail:
|
|
msg: >-
|
|
/etc/resolv.conf on this node is not the systemd-resolved stub, but something else would
|
|
regenerate the file underneath this role: resolvconf/openresolv
|
|
(/etc/resolvconf/resolv.conf.d present:
|
|
{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}), NetworkManager
|
|
(is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}), or a DHCP client
|
|
that can still write it — dhcpcd/dhclient evidence:
|
|
{{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }}.
|
|
Writing it here would look like it worked and silently revert — which is exactly what
|
|
happened on k3s-prod-3 on 2026-09-04, where a wifi lease's dhcpcd hook reverted this
|
|
role's resolvers three minutes after a green run. If the evidence above is a DHCP client
|
|
on a WIRELESS interface, the fix is the `wireless` block earlier in this role (a cluster
|
|
node gets one network path) — run it, then re-run this. If it is a DHCP client on the
|
|
node's real interface, or another manager entirely, add the matching branch to
|
|
roles/node_baseline for the manager this node really uses — do not force it.
|
|
when: node_baseline_dns_mechanism == 'contended'
|
|
tags: [dns]
|
|
|
|
# THE CORPSE CASE, and why it is NOT a hole in the fail-closed rule above. After the
|
|
# `wireless` block disables the radio, k3s-prod-3 is left with a /etc/resolv.conf that dhcpcd
|
|
# WROTE (its "# Generated by dhcpcd from wlp1s0.dhcp..." header, the gateway as sole IPv4
|
|
# resolver, two Google IPv6 servers) and NO dhcpcd that can ever run again. `contended` means
|
|
# "another manager is live and will undo this write"; a file left behind by a client that no
|
|
# longer exists is not a manager, it is the damage — and refusing to write here would leave the
|
|
# node permanently mis-resolved, which is the very outcome the operator reported. So the file
|
|
# branch below reclaims it, and says so.
|
|
- name: Note that a dhcpcd-written /etc/resolv.conf is being reclaimed
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
/etc/resolv.conf on this node was written by dhcpcd, but no DHCP client can write it any
|
|
more (no process, no active/enabled unit, no `inet dhcp` stanza). Taking ownership and
|
|
replacing it with the declared upstreams
|
|
({{ node_baseline_dns_servers | join(', ') }}). If a DHCP client ever comes back on this
|
|
node, the `contended` check above will stop the play instead of letting it revert
|
|
silently.
|
|
when:
|
|
- node_baseline_dns_mechanism == 'resolv-conf-file'
|
|
- node_baseline_resolv_dhcpcd_written | default(false)
|
|
tags: [dns]
|
|
|
|
# --- Mechanism A: systemd-resolved (dev's shape today) -------------------------------
|
|
# A drop-in, never an edit of /etc/systemd/resolved.conf, so a package upgrade of that file
|
|
# never conflicts. `DNS=` is one ORDERED list inside a single resolved scope: resolved uses
|
|
# the first server and only rotates to the next when it fails, which is exactly the
|
|
# PiHole-primary/fallback behaviour wanted — it is NOT a race between two servers, so a
|
|
# gateway NXDOMAIN can never beat PiHole to a jackson.lan answer while PiHole is healthy.
|
|
- name: Ensure the systemd-resolved drop-in directory exists
|
|
ansible.builtin.file:
|
|
path: /etc/systemd/resolved.conf.d
|
|
state: directory
|
|
owner: root
|
|
group: root
|
|
mode: "0755"
|
|
when: node_baseline_dns_mechanism == 'systemd-resolved'
|
|
tags: [dns]
|
|
|
|
- name: Configure resilient DNS upstreams via a systemd-resolved drop-in
|
|
ansible.builtin.copy:
|
|
dest: /etc/systemd/resolved.conf.d/99-k3s-dns.conf
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
# Managed by the Ansible node_baseline role (tags: dns) — do not edit by hand.
|
|
# Ordered upstream list: the FIRST server is the local-zone/ad-blocking resolver and
|
|
# stays authoritative; the rest exist only so external resolution (containerd image
|
|
# pulls, CoreDNS `forward . /etc/resolv.conf`) survives its recursion flapping.
|
|
[Resolve]
|
|
DNS={{ node_baseline_dns_servers | join(' ') }}
|
|
Domains={{ node_baseline_dns_search | join(' ') }}
|
|
# Reset to empty deliberately. systemd ships compiled-in public fallback resolvers
|
|
# (Cloudflare/Google) that resolved would reach for if the list above ever came up
|
|
# empty — which would silently bypass BOTH the local zone and the network's
|
|
# ad-blocking. This fix adds redundancy; it never adds a bypass.
|
|
FallbackDNS=
|
|
notify: Restart systemd-resolved
|
|
when: node_baseline_dns_mechanism == 'systemd-resolved'
|
|
tags: [dns]
|
|
|
|
# --- Mechanism B: a directly-managed /etc/resolv.conf (prod's shape today) -----------
|
|
# Only reached when nothing else owns the file (the `contended` branch above already
|
|
# stopped the play otherwise). glibc walks `nameserver` lines in order and falls through
|
|
# on timeout, so ordering carries the same policy as the resolved list.
|
|
# NOTE this REPLACES the file: any nameserver currently there and NOT in
|
|
# node_baseline_dns_servers is dropped. That is the point — the list in group_vars is the
|
|
# whole declared truth — so an environment that wants its existing IPv6 upstreams kept
|
|
# must list them there too. `backup: true` leaves the pre-role copy beside it the first
|
|
# time it changes, because this file is load-bearing for reaching the node at all.
|
|
- name: Configure resilient DNS upstreams directly in /etc/resolv.conf
|
|
ansible.builtin.copy:
|
|
dest: /etc/resolv.conf
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
backup: true
|
|
content: |
|
|
# Managed by the Ansible node_baseline role (tags: dns) — do not edit by hand.
|
|
# Ordered upstream list: the FIRST server is the local-zone resolver and stays
|
|
# authoritative; the rest exist only so external resolution (containerd image pulls,
|
|
# CoreDNS `forward . /etc/resolv.conf`) survives its recursion flapping.
|
|
{% if node_baseline_dns_search | length > 0 %}
|
|
search {{ node_baseline_dns_search | join(' ') }}
|
|
{% endif %}
|
|
{% for server in node_baseline_dns_servers %}
|
|
nameserver {{ server }}
|
|
{% endfor %}
|
|
{% if node_baseline_dns_options | length > 0 %}
|
|
options {{ node_baseline_dns_options | join(' ') }}
|
|
{% endif %}
|
|
when: node_baseline_dns_mechanism == 'resolv-conf-file'
|
|
tags: [dns]
|