Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

feat(prod): temp Gitea source + wireless disabled + flux bootstrap path

Browse Source 
Autonomous prod bring-up, operator away. Three changes that together clear the last
blockers before `flux bootstrap`.

=== 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared ===
clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for
decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN
gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's
workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable
(Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT.

clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner
with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done
looks like'. Repo verified PUBLIC with anonymous read two independent ways (API
"private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef
is needed — documented so a future auth failure is not misread as a network fault.

bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED
libvirt NAT, and forbade agent use; the operator has now directed this for prod.

=== 2. WIRELESS DISABLED ON PROD (operator-reported) ===
k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting
/etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection
checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported
success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed
nothing' the contended branch exists to prevent.

Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the
node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost
nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a
wireless interface and nothing else, take links down and unload modules with NO reboot, then
report the resulting state. dhcpcd is now in the contended set without weakening fail-closed.

Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its
unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public
names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down.

=== 3. flux_bootstrap PATH PARAMETERISED ===
bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it
ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role
default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in
group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve
clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide.

DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for
both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level
canary and a deep-path content canary) confirming the harness detects change — and both
canaries proven restored. Honest scope note recorded: kustomize strips comments, so the
harness proves the built stream, not source comments.

L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0.

=== STILL BLOCKING A FULL PROD RECONCILE (operator decisions) ===
- S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at
  192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod
  backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a
  workstation container would create false confidence. Velero/CNPG backups will fail loudly.
- Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl
  SSO will not work on prod until split-horizon is solved for it (§4).
- The temp Gitea is a single container on the operator's workstation — a single point of
  failure for prod reconciliation, acceptable for bring-up only.

Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-04 18:53:17 -05:00
1 parent 05baae90ec
commit 9b8979a427
9 files changed
+659 -46
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

infra/ansible/roles/node_baseline/tasks/main.yml
+412 -14
View File
@@ -1,6 +1,6 @@
---
# TAGS — every task here carries one of `packages`, `kernel`, `system`, `disks` or `dns`. An
# untagged run is unchanged (all five run, in this order). The split exists because the PACKAGE floor
# TAGS — every task here carries one of `packages`, `kernel`, `system`, `wireless`, `disks` or
# `dns`. An untagged run is unchanged (all six run, in this order). The split exists because the PACKAGE floor
# and the DATA-DISK work have different readiness: packages are safe on any node from
# first contact, while the disk tasks need block devices that a new environment may not
# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
@@ -16,6 +16,12 @@
# is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
# it must be runnable without touching a disk decision that is still open:
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
# `wireless` (a cluster node gets exactly ONE path onto the network) is separated for a third
# reason: it must be runnable BEFORE `dns` and without touching packages, the kernel or a disk
# decision, because a node with a second DHCP-managed interface reverts the `dns` write. It is
# placed before the disk work for the same reason, and the pair is the whole remedy for the
# k3s-prod-3 incident:
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags wireless,dns
- name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
ansible.builtin.apt:
@@ -314,6 +320,304 @@
state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
tags: [system]
# =====================================================================================
# WIRELESS — tag `wireless`: a cluster node gets exactly ONE path onto the network
# =====================================================================================
# WHAT WENT WRONG (k3s-prod-3, reported by the operator 2026-09-04). The node was up on BOTH
# eno1 (192.168.2.12 — the inventory address) and wlp1s0 (192.168.2.195, DHCP, the SAME /24).
# The wifi lease carried a dhcpcd run with it, and dhcpcd's `20-resolv.conf` hook REWROTE
# /etc/resolv.conf, undoing the resolvers this role had just written: the file read
# "# Generated by dhcpcd from wlp1s0.dhcp, wlp1s0.dhcp6, wlp1s0.ra" with nameserver
# 192.168.2.1 plus two Google IPv6 servers, timestamped THREE MINUTES AFTER the role's own
# write (01:32 -> 01:35). The role had reported `changed` and green. That is exactly the
# "successful run that fixed nothing" the `contended` branch of the DNS block below exists to
# prevent, and the detection blind spot that let it through is closed there.
#
# WHY THE INTERFACE IS REMOVED RATHER THAN JUST REPORTED. Closing the detection hole alone
# would have turned a silent revert into a loud failure — an improvement, but it leaves the
# node with a second, DHCP-managed path onto the LAN, and on a k3s server that is three
# distinct problems beyond DNS:
# * a SECOND DEFAULT ROUTE, whose metric decides which source address the node uses;
# * a SECOND ADDRESS ON THE SAME L2 — the segment where kube-vip floats 192.168.2.9 and
# Cilium L2-announces 192.168.2.20-.30 (§4). One host answering ARP on one subnet from
# two NICs is the exact shape neither mechanism can reason about;
# * an UNMANAGED address: nothing in this repo knows about .195, and the DHCP server is
# free to hand out a different one after any lease expiry.
# None of that is a resolver bug, and none of it is fixed by writing resolv.conf harder.
#
# WHY THE DRIVER LAYER — the four alternatives, and why each is weaker. What has to hold is:
# survives a REBOOT, survives a LEASE RENEWAL, and cannot be undone by a routine `ifup`.
# Measured on the nodes themselves — all three are HP EliteDesk 800 G5 minis, eno1 is
# `e1000e`, the wifi NIC is an Intel AX200 on `iwlwifi`, and the OS is Debian 13 trixie with
# ifupdown + `networking.service`, NO NetworkManager, NO systemd-networkd, NO systemd-resolved:
# `ip link set <wifi> down` — NOT persistent. Any `ifup`, any hotplug event, any reboot
# undoes it. It is what had already been done by hand on node 3, and the stale dhcpcd
# /etc/resolv.conf outlived it — the link was down and the wrong resolvers were still
# in the file, across a reboot.
# mask the service that manages it — THERE IS NO SUCH SERVICE HERE. dhcpcd on these nodes
# is `dhcpcd-base`: no dhcpcd.service exists at all (`systemctl is-enabled dhcpcd` =
# not-found), it is spawned per-interface by ifupdown. The only unit that can bring
# wireless up is `networking.service` — which ALSO owns eno1's static address, so
# masking it would cut the one SSH path to a machine whose operator is not on site.
# `rfkill` soft-block — weaker on every axis. The netdev still EXISTS, so ifupdown and
# dhcpcd can still be pointed at it; persistence depends on systemd-rfkill restoring
# saved state (the kernel's own `persistent` flag reads 0 on these devices, and the
# `rfkill` CLI is not even installed); and one `rfkill unblock wlan` undoes it.
# the BIOS `M.2 WLAN/BT` toggle — genuinely the deepest layer, and it reads `Enable` on all
# three nodes (hp-bioscfg survey, §1b). NOT taken here: it needs a REBOOT to take
# effect, and a prod reboot is a serialized, supervised operator step (§0b "SECOND
# FAULT" — k3s-prod-2 hung on a warm reboot and needed the power button). It also
# disables Bluetooth on the same M.2 card. Left as an operator recommendation in §4.
# THE DRIVER MODULE — chosen. `blacklist` plus `install <module> /bin/false` in
# /etc/modprobe.d means kmod refuses to load the wifi driver on this boot and every
# future one, so THE NETDEV NEVER EXISTS. No interface means no DHCP lease, no
# resolv.conf hook, no second route, no second address — and every re-activation path
# above (udev hotplug, a hand-run `ifup`, wpa_supplicant, a hand-run dhcpcd) needs a
# device that is not there. `blacklist` alone would still allow a dependency- or
# hand-triggered `modprobe`; the `install ... /bin/false` line is what makes it refuse
# outright. It is ONE declarative file, readable on the node, and undone by deleting
# it. Paired with the runtime unload below, it needs NO reboot to take effect.
#
# SAFE AND IDEMPOTENT ON ALL THREE PROD NODES *AND* ON THE LIVE DEV CLUSTER. Nothing here is
# hostname- or hardware-specific: the block acts only on interfaces the node itself reports as
# real 802.11 devices (a `phy80211` link in sysfs) and on the driver modules behind them.
# * dev (Proxmox VMs, virtio NICs) has no such interface, so the discovery comes back empty,
# every task below is skipped, and NOT EVEN THE modprobe.d FILE IS WRITTEN. The block
# cannot touch dev's networking, which is why it is safe to have on by default.
# * k3s-prod-1 (wlp1s0) and k3s-prod-2 (wlp2s0) already have the link DOWN, so the link-down
# task is skipped by its own `when` and only the modprobe.d file and the module unload
# remain — and both are no-ops on a re-run.
# * k3s-prod-3 (wlp1s0) is the reported node.
# A re-run changes nothing on any of them: the file is content-identical, `ip link set down`
# is gated on the link not already being down, and the unload is gated on the module still
# being loaded.
#
# AND IT REFUSES TO CUT ITS OWN PATH IN. The assert below stops the play if the address
# Ansible is connected to is bound to a wireless interface, or if the default route leaves
# over one, or if the wireless driver module is also the driver of the default-route
# interface. That is the one case where doing this would lock everybody out of a machine
# nobody can walk up to, so it is checked per node, on every run, from that node's own facts.
- name: Survey this node's 802.11 devices, their drivers and any live DHCP client
ansible.builtin.shell:
executable: /bin/sh
cmd: |
set -u
# Flat KEY|value lines so the Ansible side needs one single-group regex per fact and
# no parsing cleverness. A wireless interface is one with a phy80211 link in sysfs —
# that is the kernel's own answer to "is this 802.11", not a name pattern like wlp*.
wifi=''
for d in /sys/class/net/*; do
[ -e "$d/phy80211" ] || continue
i=$(basename "$d")
wifi="$wifi $i"
m=$(basename "$(readlink -f "$d/device/driver" 2>/dev/null || echo unknown)")
state=$(cat "$d/operstate" 2>/dev/null || echo unknown)
printf 'IFACE|%s\n' "$i"
printf 'DETAIL|%s driver=%s operstate=%s\n' "$i" "$m" "$state"
# LIVE means "not already down" — the link-down task keys off this, so a node whose
# radio is already down reports no change.
[ "$state" = down ] || printf 'IFACE_LIVE|%s\n' "$i"
ip -o addr show dev "$i" 2>/dev/null | awk '{print "IFACE_ADDR|" $4}'
[ "$m" = unknown ] || printf 'MODULE|%s\n' "$m"
done
for m in $(printf '%s\n' $wifi | while read -r i; do
[ -n "$i" ] && basename "$(readlink -f "/sys/class/net/$i/device/driver" 2>/dev/null || echo unknown)"
done | sort -u); do
[ "$m" = unknown ] && continue
[ -d "/sys/module/$m" ] || continue
printf 'MODULE_LOADED|%s\n' "$m"
# Holders must be unloaded FIRST: `modprobe -r iwlwifi` fails while iwlmvm still
# holds it. Read from sysfs rather than assumed, so a different wifi driver stack
# (ath*, rtw*, mt76*) works without editing this role.
for h in $(ls "/sys/module/$m/holders" 2>/dev/null); do printf 'MODULE_HOLDER|%s\n' "$h"; done
done
dr=$(ip -o route show default 2>/dev/null | awk '{for (i = 1; i < NF; i++) if ($i == "dev") print $(i + 1)}' | head -1)
printf 'DEFROUTE_IFACE|%s\n' "${dr:-none}"
if [ -n "${dr:-}" ] && [ -e "/sys/class/net/$dr" ]; then
printf 'DEFROUTE_MODULE|%s\n' "$(basename "$(readlink -f "/sys/class/net/$dr/device/driver" 2>/dev/null || echo unknown)")"
fi
# Any DHCP client process, with the interfaces it was started for. Used only to stop a
# client that holds a WIRELESS interface and nothing else.
for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
printf 'DHCPCLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
done
exit 0
register: node_baseline_wifi_scan
changed_when: false
check_mode: false
when: node_baseline_disable_wireless | bool
tags: [wireless]
- name: Interpret the wireless survey
ansible.builtin.set_fact:
node_baseline_wifi_ifaces: "{{ scan | regex_findall('(?m)^IFACE\\|(.+)$') }}"
node_baseline_wifi_live: "{{ scan | regex_findall('(?m)^IFACE_LIVE\\|(.+)$') }}"
node_baseline_wifi_addrs: "{{ scan | regex_findall('(?m)^IFACE_ADDR\\|(\\S+)') | map('regex_replace', '/.*$', '') | list }}"
node_baseline_wifi_modules: "{{ scan | regex_findall('(?m)^MODULE\\|(.+)$') | unique }}"
node_baseline_wifi_modules_loaded: "{{ scan | regex_findall('(?m)^MODULE_LOADED\\|(.+)$') | unique }}"
node_baseline_wifi_module_holders: "{{ scan | regex_findall('(?m)^MODULE_HOLDER\\|(.+)$') | unique }}"
node_baseline_wifi_detail: "{{ scan | regex_findall('(?m)^DETAIL\\|(.+)$') }}"
node_baseline_wifi_dhcp_clients: "{{ scan | regex_findall('(?m)^DHCPCLIENT\\|(.+)$') }}"
node_baseline_defroute_iface: "{{ (scan | regex_findall('(?m)^DEFROUTE_IFACE\\|(.+)$') | first) | default('none', true) }}"
node_baseline_defroute_module: "{{ (scan | regex_findall('(?m)^DEFROUTE_MODULE\\|(.+)$') | first) | default('unknown', true) }}"
vars:
scan: "{{ node_baseline_wifi_scan.stdout | default('') }}"
when: node_baseline_disable_wireless | bool
tags: [wireless]
- name: Report this node's wireless hardware
ansible.builtin.debug:
msg: >-
wireless devices = {{ node_baseline_wifi_detail | default([]) | join('; ') | default('(none)', true) }}
| driver modules = {{ node_baseline_wifi_modules | default([]) | join(', ') | default('(none)', true) }}
(loaded: {{ node_baseline_wifi_modules_loaded | default([]) | join(', ') | default('(none)', true) }},
held by: {{ node_baseline_wifi_module_holders | default([]) | join(', ') | default('(none)', true) }})
| default route via {{ node_baseline_defroute_iface | default('?') }}
(driver {{ node_baseline_defroute_module | default('?') }})
when: node_baseline_disable_wireless | bool
tags: [wireless]
# The one way this block could do real harm is by removing the interface it is being run
# over. Checked from the node's own facts, per node, every run — not assumed from the fact
# that prod "should" be wired.
- name: Refuse to disable wireless if the path to this node runs OVER the wireless interface
ansible.builtin.assert:
that:
- node_baseline_defroute_iface not in node_baseline_wifi_ifaces
- node_baseline_defroute_module not in node_baseline_wifi_modules
- (ansible_host | default(inventory_hostname)) not in node_baseline_wifi_addrs
fail_msg: >-
REFUSING to disable wireless on this node: the connection to it appears to BE the
wireless path. ansible_host={{ ansible_host | default(inventory_hostname) }}, wireless
interfaces={{ node_baseline_wifi_ifaces | join(', ') }} carrying
{{ node_baseline_wifi_addrs | join(', ') | default('no addresses', true) }}, default route
via {{ node_baseline_defroute_iface }} (driver {{ node_baseline_defroute_module }}).
Disabling the driver here would drop this node off the network with no remote way back.
Give the node a wired address, point the inventory at it, and re-run.
quiet: true
when:
- node_baseline_disable_wireless | bool
- node_baseline_wifi_ifaces | length > 0
tags: [wireless]
# THE PERSISTENT HALF. This file is the whole reboot story: kmod consults /etc/modprobe.d on
# every load attempt, so from here on the driver cannot be loaded by udev at boot, as a
# dependency, or by hand. Written ONLY when the node actually has a wireless driver, so a
# node without wifi hardware (every dev VM) gets no file and no change at all.
# `update-initramfs` is deliberately NOT run: no wifi module is loaded from the initramfs on
# these nodes (root is a local SATA/NVMe device), the driver is autoloaded by udev after
# switch-root, and re-generating an initramfs on a prod node is a bigger risk than the one it
# would cover.
- name: Block the wireless driver module(s) from ever loading again
ansible.builtin.copy:
dest: "{{ node_baseline_wireless_modprobe_file }}"
owner: root
group: root
mode: "0644"
content: |
# Managed by the Ansible node_baseline role (tags: wireless) — do not edit by hand.
#
# WHY THIS NODE HAS NO WIRELESS. A k3s server gets exactly ONE path onto the network.
# A second, DHCP-managed interface on the same LAN gives the node a second default
# route, a second address nothing in the cluster's config knows about, and a DHCP
# client whose resolv.conf hook silently overwrites the resolvers Ansible manages
# (that is what happened on k3s-prod-3, 2026-09-04). The link is also brought down and
# the module unloaded at the same time, so no reboot is needed for this to take effect.
#
# `blacklist` stops udev's alias-driven autoload; the `install ... /bin/false` line is
# what also refuses a dependency-triggered or hand-run `modprobe`. Both are needed.
#
# TO RE-ENABLE WIRELESS ON THIS NODE: delete this file and `modprobe` the module(s)
# below (or reboot). To stop Ansible putting it back, set
# node_baseline_disable_wireless: false for this host first — otherwise the next
# node-baseline run recreates it. Full rationale: roles/node_baseline/tasks/main.yml,
# the `wireless` block, and infra/PROD-PROVISIONING.md §4.
{% for module in node_baseline_wifi_modules %}
blacklist {{ module }}
install {{ module }} /bin/false
{% endfor %}
when:
- node_baseline_disable_wireless | bool
- node_baseline_wifi_modules | length > 0
tags: [wireless]
# THE RUNTIME HALF — everything below makes the change true NOW, so the node does not have to
# be rebooted to stop carrying a second address (this role never reboots: §0b "SECOND FAULT").
#
# Stop a DHCP client that is holding a wireless interface first, so it releases its lease
# instead of being left holding a device that is about to disappear. HEAVILY GUARDED: it runs
# only for clients whose command line names a wireless interface and names NO non-wireless
# interface, because `dhcpcd -x` against a shared master process would also drop a wired
# lease. On these prod nodes eno1 is a static ifupdown address with no DHCP client at all, so
# this task has nothing to do; it exists so the ordering is correct on a node where wifi is
# still actively leasing.
- name: Stop the dhcpcd instance that holds a wireless interface (and nothing else)
ansible.builtin.command:
argv: [dhcpcd, --exit, "{{ item }}"]
loop: "{{ node_baseline_wifi_ifaces }}"
register: node_baseline_wifi_dhcpcd_exit
changed_when: rc == 0
failed_when: false
when:
- node_baseline_disable_wireless | bool
- node_baseline_wifi_dhcp_wifi_only | length > 0
- node_baseline_wifi_dhcp_wifi_only | select('search', 'dhcpcd') | list | length > 0
vars:
# A client is "wireless-only" when its argv mentions a wireless interface and mentions no
# other interface this node has. Empty interface sets are turned into a pattern that
# cannot match, so an odd node fails towards doing nothing rather than towards guessing.
node_baseline_non_wifi_ifaces: "{{ ansible_facts['interfaces'] | default([]) | difference(node_baseline_wifi_ifaces) | difference(['lo']) }}"
node_baseline_wifi_dhcp_wifi_only: >-
{{ node_baseline_wifi_dhcp_clients
| select('search', '\b(' ~ ((node_baseline_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
| reject('search', '\b(' ~ ((node_baseline_non_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
| list }}
tags: [wireless]
- name: Take each wireless link down now
ansible.builtin.command:
argv: [ip, link, set, dev, "{{ item }}", down]
loop: "{{ node_baseline_wifi_live }}"
changed_when: true
when: node_baseline_disable_wireless | bool
tags: [wireless]
# Unload the driver so the netdev is gone immediately — holders (iwlmvm on these nodes) first,
# then the driver itself, because `modprobe -r` refuses a module that is still held.
# NOT fatal on failure: the persistent block above is already in place, so a module that
# cannot be unloaded right now is gone at the next boot regardless. The report below says so
# out loud rather than letting a green play imply the radio is already off.
- name: Unload the wireless driver module(s) now (no reboot needed)
community.general.modprobe:
name: "{{ item }}"
state: absent
loop: "{{ node_baseline_wifi_module_holders + node_baseline_wifi_modules_loaded }}"
register: node_baseline_wifi_unload
failed_when: false
when: node_baseline_disable_wireless | bool
tags: [wireless]
- name: Report the wireless state this node is left in
ansible.builtin.debug:
msg: >-
{{ 'No 802.11 device on this node — nothing to disable (no modprobe.d file written).'
if (node_baseline_wifi_ifaces | length) == 0
else ((('WIRELESS DISABLED: ' ~ (node_baseline_wifi_modules | join(', ')) ~ ' blocked in '
~ node_baseline_wireless_modprobe_file ~ ' (survives reboot) and unloaded now; '
~ (node_baseline_wifi_ifaces | join(', ')) ~ ' is down and the netdev is gone.')
if (failed_unloads | length) == 0
else ('WIRELESS BLOCKED FOR THE NEXT BOOT BUT STILL LOADED NOW: could not unload '
~ (failed_unloads | join(', ')) ~ '. ' ~ node_baseline_wireless_modprobe_file
~ ' is in place, so the driver will not load again after a reboot, but the '
~ 'radio is live until then — check `ip link` and whether something still '
~ 'holds the device.'))) }}
vars:
failed_unloads: >-
{{ (node_baseline_wifi_unload.results | default([]))
| selectattr('failed', 'defined') | selectattr('failed') | map(attribute='item') | list }}
when: node_baseline_disable_wireless | bool
tags: [wireless]
# =====================================================================================
# DATA DISKS — tag `disks`, in tasks/disks.yml
# =====================================================================================
@@ -377,6 +681,11 @@
# node and the matching seam is written. Adding a third mechanism means adding a branch
# here, not rewriting the role.
#
# AND DETECTION IS ONLY AS GOOD AS THE SET OF OWNERS IT KNOWS ABOUT. It originally looked for
# resolvconf/openresolv and NetworkManager, missed dhcpcd, and therefore reported a green,
# `changed` run on k3s-prod-3 whose resolvers a wifi lease reverted three minutes later. dhcpcd
# is now part of that set — see the detection task below and the `contended` failure.
#
# All of it is `dns`-tagged, so `--tags packages` (the prod package-floor escape hatch
# documented at the top of this file) still installs nothing but packages.
@@ -396,6 +705,55 @@
check_mode: false
tags: [dns]
# THE dhcpcd BLIND SPOT, CLOSED (k3s-prod-3, 2026-09-04). dhcpcd's `20-resolv.conf` hook
# rewrites /etc/resolv.conf on every lease, renewal and carrier change, and it is a THIRD
# way to own that file — neither resolvconf nor NetworkManager. On node 3 it silently reverted
# this role's write three minutes later, and because nothing below looked for it the role had
# already reported `changed` and green: the precise failure the `contended` branch exists to
# make impossible.
#
# WHY THIS LOOKS FOR A LIVE CLIENT AND NOT FOR AN INSTALLED PACKAGE. `dhcpcd-base` is
# installed on all three prod nodes and always will be — it is the DHCP client ifupdown
# reaches for — and there is NO dhcpcd.service on them at all (`systemctl is-active dhcpcd` =
# not-found; it is spawned per-interface by ifupdown). Keying off the package, or off a unit
# that does not exist, would put every prod node permanently in `contended` and leave the DNS
# block unable to configure a node it is perfectly safe to configure. What actually matters is
# whether a DHCP client CAN still write the file, so all three real signals are checked:
# CLIENT| a dhcpcd/dhclient process running right now (with the argv that started it);
# UNIT| a dhcpcd/dhclient unit that IS active or IS enabled, on a distro that has one;
# STANZA| an UNCOMMENTED `iface <x> inet[6] dhcp` in /etc/network/interfaces{,.d} — the
# configuration that makes ifupdown spawn a client at the next boot or `ifup`, even
# when none is running at this instant.
# Any one of them means "another manager is in play" and the play stops below. None of them
# means no client can run, and the file is genuinely ours to own.
- name: Determine whether a DHCP client (dhcpcd/dhclient) can rewrite /etc/resolv.conf
ansible.builtin.shell:
executable: /bin/sh
cmd: |
set -u
for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
printf 'CLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
done
for u in dhcpcd dhcpcd5 dhclient isc-dhcp-client; do
a=$(systemctl is-active "$u" 2>/dev/null || true)
e=$(systemctl is-enabled "$u" 2>/dev/null || true)
# Compared exactly, never with a substring match: `is-active` answers "inactive",
# which contains the word "active".
if [ "$a" = active ] || [ "$e" = enabled ] || [ "$e" = enabled-runtime ]; then
printf 'UNIT|%s active=%s enabled=%s\n' "$u" "$a" "$e"
fi
done
# `^[[:space:]]*iface` cannot match a commented-out stanza, which is how the wireless
# stanza is left on these nodes.
grep -rhE '^[[:space:]]*iface[[:space:]]+[^[:space:]]+[[:space:]]+inet6?[[:space:]]+dhcp' \
/etc/network/interfaces /etc/network/interfaces.d/ 2>/dev/null \
| sed -e 's/^[[:space:]]*/STANZA|/'
exit 0
register: node_baseline_dhcp_client_scan
changed_when: false
check_mode: false
tags: [dns]
# follow: false — we need to know whether /etc/resolv.conf IS a symlink and where it
# points, which is the strongest single signal of who owns the file.
- name: Inspect /etc/resolv.conf itself (symlink or real file, and its target)
@@ -426,9 +784,10 @@
# resolved's loopback stub (127.0.0.53 / 127.0.0.54). Writing a resolved drop-in is
# then the ONLY thing that changes real behaviour; writing resolv.conf would be
# overwritten by resolved (or, worse, land in /run and vanish at reboot).
# contended — nobody's stub, but resolvconf or NetworkManager is present/running and
# would regenerate the file underneath us. We do NOT write. The role fails loudly
# below rather than leaving an operator believing a change stuck when it did not.
# contended — nobody's stub, but resolvconf, NetworkManager or a DHCP client (dhcpcd /
# dhclient) is present/running/configured and would regenerate the file underneath us.
# We do NOT write. The role fails loudly below rather than leaving an operator
# believing a change stuck when it did not.
# resolv-conf-file — a plain, directly-managed /etc/resolv.conf (prod's shape today).
# Safe to own outright.
# resolved running but NOT owning resolv.conf correctly falls through to the file branch:
@@ -438,13 +797,21 @@
node_baseline_dns_mechanism: >-
{{
'systemd-resolved' if (node_baseline_resolved_running and node_baseline_resolved_owns_resolv)
else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present)
else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present
or node_baseline_dhcp_client_live)
else 'resolv-conf-file')
}}
# Kept as facts (not just task vars) so the failure message, the report and the
# reclaim note below all quote the SAME evidence this decision was made on.
node_baseline_dhcp_client_evidence: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) | regex_findall('(?m)^(?:CLIENT|UNIT|STANZA)\\|(.+)$') }}"
# A file dhcpcd WROTE, versus a dhcpcd that can still write it — two different facts, and
# only the second one is contention. See the reclaim note below.
node_baseline_resolv_dhcpcd_written: "{{ 'Generated by dhcpcd' in (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
vars:
node_baseline_resolved_running: "{{ (node_baseline_resolved_active.stdout | default('') | trim) == 'active' }}"
node_baseline_nm_running: "{{ (node_baseline_nm_active.stdout | default('') | trim) == 'active' }}"
node_baseline_resolvconf_present: "{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}"
node_baseline_dhcp_client_live: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) is search('(?m)^(?:CLIENT|UNIT|STANZA)\\|') }}"
node_baseline_resolv_text: "{{ (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
node_baseline_resolv_ns: "{{ node_baseline_resolv_text | regex_findall('(?m)^\\s*nameserver\\s+(\\S+)') }}"
node_baseline_resolved_owns_resolv: >-
@@ -461,7 +828,9 @@
NetworkManager is-active={{ node_baseline_nm_active.stdout | default('') | trim }},
/etc/resolv.conf islnk={{ node_baseline_resolv_stat.stat.islnk | default(false) }}
target={{ node_baseline_resolv_stat.stat.lnk_source | default('(not a symlink)') }},
resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }})
resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }},
DHCP client evidence={{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }},
resolv.conf written by dhcpcd={{ node_baseline_resolv_dhcpcd_written | default(false) }})
tags: [dns]
# Same shape, and the same reason, as the data-disk assert above: this environment's
@@ -494,16 +863,45 @@
- name: Refuse to manage /etc/resolv.conf while another resolver manager owns it
ansible.builtin.fail:
msg: >-
/etc/resolv.conf on this node is not the systemd-resolved stub, but
resolvconf/openresolv (/etc/resolvconf/resolv.conf.d present:
{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}) or NetworkManager
(is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}) is in play and
would regenerate the file underneath this role. Writing it here would look like it
worked and silently revert. Add the matching branch to roles/node_baseline (the
`dns`-tagged block) for whichever manager this node really uses — do not force it.
/etc/resolv.conf on this node is not the systemd-resolved stub, but something else would
regenerate the file underneath this role: resolvconf/openresolv
(/etc/resolvconf/resolv.conf.d present:
{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}), NetworkManager
(is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}), or a DHCP client
that can still write it — dhcpcd/dhclient evidence:
{{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }}.
Writing it here would look like it worked and silently revert — which is exactly what
happened on k3s-prod-3 on 2026-09-04, where a wifi lease's dhcpcd hook reverted this
role's resolvers three minutes after a green run. If the evidence above is a DHCP client
on a WIRELESS interface, the fix is the `wireless` block earlier in this role (a cluster
node gets one network path) — run it, then re-run this. If it is a DHCP client on the
node's real interface, or another manager entirely, add the matching branch to
roles/node_baseline for the manager this node really uses — do not force it.
when: node_baseline_dns_mechanism == 'contended'
tags: [dns]
# THE CORPSE CASE, and why it is NOT a hole in the fail-closed rule above. After the
# `wireless` block disables the radio, k3s-prod-3 is left with a /etc/resolv.conf that dhcpcd
# WROTE (its "# Generated by dhcpcd from wlp1s0.dhcp..." header, the gateway as sole IPv4
# resolver, two Google IPv6 servers) and NO dhcpcd that can ever run again. `contended` means
# "another manager is live and will undo this write"; a file left behind by a client that no
# longer exists is not a manager, it is the damage — and refusing to write here would leave the
# node permanently mis-resolved, which is the very outcome the operator reported. So the file
# branch below reclaims it, and says so.
- name: Note that a dhcpcd-written /etc/resolv.conf is being reclaimed
ansible.builtin.debug:
msg: >-
/etc/resolv.conf on this node was written by dhcpcd, but no DHCP client can write it any
more (no process, no active/enabled unit, no `inet dhcp` stanza). Taking ownership and
replacing it with the declared upstreams
({{ node_baseline_dns_servers | join(', ') }}). If a DHCP client ever comes back on this
node, the `contended` check above will stop the play instead of letting it revert
silently.
when:
- node_baseline_dns_mechanism == 'resolv-conf-file'
- node_baseline_resolv_dhcpcd_written | default(false)
tags: [dns]
# --- Mechanism A: systemd-resolved (dev's shape today) -------------------------------
# A drop-in, never an edit of /etc/systemd/resolved.conf, so a package upgrade of that file
# never conflicts. `DNS=` is one ORDERED list inside a single resolved scope: resolved uses