3 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	0e92d258dd	fix(node_baseline): swap-off now survives a reboot — mask swap.target ... The role's `swapoff -a` + commented fstab line held only until the next boot, on every prod node. Measured 2026-09-07: all three came back from a reboot with 977 MiB of swap ACTIVE (k3s-prod-1 /dev/sda4, k3s-prod-3 /dev/sda4, k3s-prod-2 /dev/nvme0n1p4) while their fstab swap entry was commented out by this role. The role reported success and the setting silently expired. CAUSE. systemd-gpt-auto-generator claims any GPT partition typed 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") from the PARTITION TABLE, synthesises a .swap unit into /run/systemd/generator.late/ and links it into swap.target.wants/. It never reads /etc/fstab, so neutralising fstab cannot stop it. Re-running the generator by hand on k3s-prod-1 says so verbatim: Probed fstype 'swap' on partition /dev/disk/by-diskseq/1-part4. Adding swap: /dev/disk/by-diskseq/1-part4 MECHANISM. Mask swap.target. `systemctl show` on the generated unit says its only inbound dependency is `WantedBy=swap.target` — Requires/RequiredBy/BoundBy empty, and the matching .device unit has an empty Wants=, so a device appearing does not pull its swap unit in by itself. swap.target is the single choke point for every activation path: gpt-auto, systemd-fstab-generator, systemd-cryptsetup-generator. It is device-shape independent (no device name appears in the change, so one task covers sda4, nvme0n1p4 and dev's vda VMs), boot-safe (swap.target is Wants=/After= from sysinit.target and After= from tmp.mount; RequiredBy= is EMPTY on all three prod and all three dev nodes, and a masked unit reached only via Wants= is skipped, not failed), and inert if the device is absent or renamed. The generated units are masked too, as a second independent barrier. Rejected: `systemd.gpt_auto=0` (disables GPT auto-discovery wholesale, needs a GRUB edit + reboot to become true, and on DEV would switch off root auto-discovery — dev's vda1 carries the discoverable GUID 4f68bce3-e8cd-4db1-96e7-fbcaf984b709); clearing the partition's GPT type GUID (a partition-table write to the live boot disk of a node serving etcd and Longhorn replicas, and an irreversible decision about that 977 MiB that belongs to the operator); `systemd.swap=0` (systemd-fstab-generator only — this swap does not come from fstab). `swapoff -a` is now conditional on `swapon --show` returning something, so the task reports honestly instead of hiding a command that ran anyway behind changed_when: false. PROVEN on k3s-prod-1 (all 18 Longhorn volumes healthy before the reboot): run 1 changed=2, run 2 changed=0 after `reboot`: /proc/swaps empty, `free` Swap 0B, swap.target masked, the generated unit still emitted by the generator and still masked, `systemctl --failed` empty, `systemctl is-system-running` = running. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 22:35:16 -05:00
jakman85andClaude Opus 5	c1f24e4a6d	fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03) ... Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in node_baseline_packages). Landing it would have failed the handler with command-not-found and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole on a network they cannot reach. The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF. PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 — breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`. MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one set_fact yielding three outcomes, because the two environments no longer share a resolver stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no cloud-init at all): * systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never silently bypass the local zone and ad-blocking. * resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which would destroy PiHole-first ordering. * contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING and fails loudly: a change that silently gets reverted is worse than no change. Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch. Ordering is the policy that keeps the local zone working — the second server is only consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a jackson.lan answer. SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan); prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it. PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk devices, with a dns-tagged assert that stops the play before any file is written. A public resolver was NOT invented as a default: it would bypass the local zone, which is a decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at the now-dropped branch) to state the choice, including that prod routing to the dev PiHole is unverified across the pfSense boundary. tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch worth taking as-is. Dev-only by nature. ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed /etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent single-upstream WARN even after a correct fix. It now asks resolvectl for the real upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and not PASS. The honesty discipline is preserved exactly. Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean; --tags packages still selects ONLY the apt task (the prod package-floor escape hatch bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL detection expression and REAL file bodies extracted from the role by script (so the test cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape -> resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the assert byte-identically while prod overrides fail it by name; collector unit-tested against 8 resolver shapes. Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 00:11:49 -05:00
jakman85andClaude Opus 4.8	a2392a4a43	tofu(proxmox): API-only native cloud-init; move node baseline to Ansible node_baseline role ... Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-25 10:55:48 -05:00