home-network-k3s-impl/infra/ansible/roles/node_baseline/defaults/main.yml

---
# node_baseline — first-boot floor for k3s/Cilium/Longhorn, moved out of cloud-init
# (Proxmox provisioning is API-only). Runs before k3s (site.yml). Idempotent.
node_baseline_packages:
  # VM-only in practice (binds the virtio-serial channel Proxmox attaches) but harmless to install
  # unconditionally: the task below only *starts* it when that channel exists, so on bare metal
  # (prod) it installs and stays a dormant, disabled unit. Kept in the shared list rather than
  # split per-platform so dev and prod install from the exact same package set (R-ENV-02 reuse).
  - qemu-guest-agent
  - open-iscsi
  - nfs-common
  - curl
  - cryptsetup   # Longhorn >=1.12 RequiredPackages check needs the cryptsetup userspace tool (dm_crypt module alone isn't enough)
  # Drive health (smartctl) is a PERMANENT part of a bare-metal node's baseline, not a one-off
  # manual install: prod runs on real disks that can fail/degrade (see infra/PROD-PROVISIONING.md
  # §2/§5 — a ~15-year-old consumer SSD on node 1), where dev's virtio disks never needed it.
  # Installed here (not hand-run with apt-get) so it's declared, idempotent, and present on every
  # node from first baseline — package installation is the role's job, never a manual step
  # (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo).
  - smartmontools
# --- BACKPORTS KERNEL (EFI memattr, PROD-PROVISIONING.md §0b/§3) ---------------------
# OFF BY DEFAULT, ON FOR PROD. This is a STANDING, DELIBERATE DEVIATION from the Debian 13
# baseline — not a one-off `apt install` someone has to remember on the next node.
#
# WHY: HP's R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI Memory
# Attributes Table. Linux < 6.13 does not validate it before `memblock_reserve()`ing memory
# from its contents, so an unlucky boot reserves everything above the table and the node
# comes up with a fraction of its RAM (k3s-prod-2: 2.6 GiB of 16 GB, twice). Upstream commit
# 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" (Ard Biesheuvel,
# first released in v6.13) adds the validation. It is NOT in v6.12 and NOT in linux-6.12.y —
# no `Cc: stable`, no `Fixes:`, so no automatic feed will ever pick it up. There is also NO
# kernel cmdline workaround: efi_memattr_init() runs before any `efi=` bail-out. Booting a
# >= 6.13 kernel is the only fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
#
# WHY A VARIABLE AND NOT A HOSTNAME TEST. The dev nodes are Proxmox VMs whose firmware
# publishes no MAT at all — they are genuinely unaffected and must NOT be dragged onto a
# backports kernel as a side effect of sharing this role (R-ENV-02 is "same role, scoped
# overrides", not "same kernel"). A `when: inventory_hostname is match('k3s-prod-')` would
# encode the environment into the ROLE, would silently miss a fourth prod node named
# differently, and could not be flipped for a one-off test host. So it is a plain boolean,
# default false, turned on in inventory/group_vars/k3s_prod/vars.yml — the SAME seam
# kube_vip_address / k3s_api_host / cluster_kubeconfig_name already use.
#
# SECURITY COST, STATED PLAINLY: packages in *-backports are provided AS-IS. They are NOT
# covered by the Debian Security Team's guarantees for the main archive — no security-tracker
# commitment, no DSAs. In exchange for that the node keeps the RAM it paid for. Accepting it
# is the operator's recorded decision (PROD-PROVISIONING.md §0b/§3), not this role's.
#
# EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y (and therefore in Debian's
# 6.12 rebase), flip this back to false everywhere, `apt purge` the backports kernel, and
# delete this block. Nothing else in the repo depends on it.
node_baseline_backports_kernel: false
# The suite, mirror and components the source line is built from. Split out so a mirror
# change is a variable, not an edit to a heredoc in tasks/main.yml.
node_baseline_backports_suite: trixie-backports
node_baseline_backports_mirror: "http://deb.debian.org/debian"
# Matches /etc/apt/sources.list on these nodes (main + non-free-firmware). Deliberately NOT
# contrib/non-free: this deviation exists to get ONE fixed kernel, and widening the archive
# surface beyond what the node already trusts is a separate decision nobody has taken.
node_baseline_backports_components:
  - main
  - non-free-firmware
# The METApackage, not a pinned ABI (e.g. linux-image-6.19.14+deb13-amd64). A frozen ABI
# would stop receiving even backports' own updates the moment the ABI rolls, leaving the
# node on a kernel nothing patches — strictly worse than the risk this deviation accepts.
# The metapackage tracks whatever backports currently ships (>= 6.13 by construction, and
# far above it in practice), and `ButAutomaticUpgrades: yes` in the backports Release file
# means `apt upgrade` keeps it current once installed. The EXACT version each node landed on
# is recorded on the node itself — see node_baseline_backports_kernel_stamp.
node_baseline_backports_kernel_packages:
  - linux-image-amd64
# Provenance/rollback note written on the node. An operator standing in front of a prod box
# with no repo checkout must be able to answer "why is this machine not on the distro
# kernel, and how do I undo it?" from the machine itself.
node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel

node_baseline_kernel_modules:
  - overlay
  - br_netfilter
  - iscsi_tcp
  - dm_crypt
  - nfs

# --- NODE CLOCK / TIMEZONE (tag `system`) ---------------------------------------------
# EVERY NODE IS UTC. Measured drift that made this a knob rather than an assumption: the
# three prod nodes disagreed on their local zone (read off k3s's own etcd snapshot
# annotations, 2026-09-05) —
#   k3s-prod-1  2026-09-05T16:31:28-05:00   (Central)
#   k3s-prod-2  2026-09-05T16:29:07-05:00   (Central)
#   k3s-prod-3  2026-09-05T17:27:35-04:00   (Eastern)
# In UTC those three are ~4 minutes apart, which is the CORRECT staggering: nothing was
# functionally wrong, because Kubernetes, etcd and the container runtime are UTC internally.
# What IS wrong is that any timestamp a node renders in LOCAL time — journald's default
# output, `ls -l`, a shell prompt, an etcd snapshot filename — cannot be compared across
# nodes without knowing each node's zone. Correlating three nodes' logs during an incident
# is exactly when nobody has that knowledge to hand, and this cluster took two unplanned
# node reboots in the 24h before this was written.
#
# WHY UTC AND NOT "STANDARDISE ON CENTRAL". Picking one local zone leaves the class of
# problem intact (a fourth node, or a DST boundary, re-creates it) and adds a translation
# step to every log read, because every timestamp the PLATFORM emits — Kubernetes events,
# Prometheus, Loki, Velero, CNPG WAL, the etcd snapshot annotations above — is already UTC.
# UTC removes the class: node-local time and platform time become the same clock, and there
# is no DST discontinuity in which a repeated or skipped hour breaks log ordering. This also
# matches the only timezone convention the repo already had — catalog/plex.yaml records
# "no other service in the repo sets a timezone ... so use the safe default" and both it and
# apps/wave-0/plex.yaml set TZ=Etc/UTC. (platform/crafty sets America/Chicago, deliberately:
# that is a USER-FACING game server's schedule, not a node's log clock.)
#
# WHY A KNOB AT ALL (R-ENV-02). The value is a node-level Ansible fact, so its override seam
# is inventory/group_vars/<env>, NOT clusters/base/cluster-vars.yaml (a Flux postBuild
# ConfigMap no Ansible code reads) — the same seam node_baseline_backports_kernel and
# node_baseline_dns_servers use. NO ENVIRONMENT OVERRIDES IT TODAY, and one should not have
# to: dev and prod both want UTC and there is nothing environment-specific about a node
# clock. It is a variable rather than a literal in the task so a genuinely
# operator-facing one-off host (never a k3s node) can differ without forking the role.
#
# Must be a zoneinfo name present on the node (`timedatectl list-timezones`); the assert in
# tasks/main.yml rejects an empty or obviously-malformed value before the module runs, and
# community.general.timezone itself refuses a name /usr/share/zoneinfo does not have.
node_baseline_timezone: Etc/UTC

# --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
# The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
# a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:
#
#   node_baseline_longhorn_source:
#     device           dedicated block device at node_baseline_longhorn_device (DEV: the
#                      tofu-provisioned virtio disk /dev/vdb). mkfs + mount by label.
#     claim-mountpoint re-purpose the partition the node ALREADY has mounted at
#                      node_baseline_longhorn_claim_mountpoint (PROD: /srv). The device
#                      path is discovered ON THE NODE — it is /dev/sda5 on prod nodes 1
#                      and 3 but /dev/nvme0n1p5 on node 2, so no group-wide device value
#                      is correct and a hardcoded one would eventually mkfs the wrong
#                      partition on a future node. Destructive; heavily guarded — read
#                      tasks/disks.yml before changing anything here.
#
#   node_baseline_etcd_source:
#     device                dedicated block device at node_baseline_etcd_device (DEV:
#                           /dev/vdc, docs/plans/ssd-etcd-isolation.md — etcd WAL fsync
#                           isolated from Longhorn rebuild I/O after dev was burned by HDD
#                           contention).
#     root-filesystem       etcd lives on the OS filesystem, with NO dedicated device and
#                           NO claimed data partition under it. Kept as a valid, supported
#                           shape; NOT what prod uses any more (see below).
#     shared-data-partition etcd lives inside node_baseline_k3s_data_dir, which sits on the
#                           CLAIMED data partition (PROD). No dedicated device, but also NOT
#                           the OS filesystem: the whole point of the 2026-09-04 layout
#                           revision is that /var (11.8 GiB) can hold neither the container
#                           image store nor etcd's WAL. See infra/PROD-PROVISIONING.md §2.
#
# Every one of these is a FIRST-CLASS, explicitly-configured state — an operator decision
# recorded in infra/PROD-PROVISIONING.md §2 — and NOT the same thing as an unset device
# var, which still fails the `disks` assert.
node_baseline_longhorn_source: device
node_baseline_longhorn_device: /dev/vdb
# --- The DATA PARTITION: mountpoint + filesystem label -------------------------------
# NEUTRAL NAMES ON PURPOSE (revised 2026-09-04). On prod this ONE partition now serves TWO
# tenants — Longhorn's replica store AND k3s's whole data dir (etcd + the containerd image
# store) — so neither the mountpoint nor the label may name only one of them. Prod overrides
# both in group_vars/k3s_prod/vars.yml (`/data`, label `k3s-data`).
#
# THE DEFAULTS BELOW ARE DEV'S LIVE VALUES AND MUST NOT CHANGE. The dev cluster has 18
# Longhorn volumes whose replicas live on the partition labelled `longhorn` mounted at
# /var/lib/longhorn; dev's etcd has its own dedicated device and dev's k3s data dir stays
# at k3s's own default. Dev's mountpoint IS Longhorn's dataPath there (one tenant, one
# partition), which is why node_baseline_longhorn_data_path below defaults to the mount.
node_baseline_data_mount: /var/lib/longhorn
node_baseline_data_fs_label: longhorn
# Longhorn's dataPath — the directory Longhorn puts replicas in. On dev that IS the mount
# root (single tenant). On prod it is a SUBDIRECTORY of the mount, because k3s owns a
# sibling subdirectory of the same partition. Must agree with the LONGHORN_DATA_PATH knob
# in clusters/base/cluster-vars.yaml (patched per overlay) — that is the Helm side of the
# same fact, and the two are checked against each other by nothing but this comment.
node_baseline_longhorn_data_path: "{{ node_baseline_data_mount }}"
# k3s's --data-dir. Defers to the CLUSTER-WIDE `k3s_data_dir` when an environment declares
# one (group_vars), and otherwise restates k3s's own built-in default. It is deliberately
# NOT a second independent variable: roles/k3s_server reads `k3s_data_dir` to build the
# install flags, this role reads the same value to create the directory and to place etcd,
# and one line in group_vars/k3s_prod/vars.yml moves BOTH. Lazy Jinja means the group_vars
# value (higher precedence than any role default) is what actually resolves here, even
# though roles/k3s_server is not in the node-baseline play; the default() keeps
# node-baseline.yml runnable standalone.
node_baseline_k3s_data_dir: "{{ k3s_data_dir | default('/var/lib/rancher/k3s') }}"
# Only consulted when node_baseline_longhorn_source is `claim-mountpoint`. Empty by
# default so a dev host cannot enter the claim path even by accident.
node_baseline_longhorn_claim_mountpoint: ""
# --- ONE-TIME TRANSITION KNOBS (delete once the fleet is converged) ------------------
# A node claimed by an EARLIER version of this role carries the OLD label at the OLD
# mountpoint, and the idempotency pivot (`blkid -L <current label>`) would therefore read
# "not claimed yet" and walk into the greenfield claim path — which then fails closed at
# "the claim mountpoint must actually be a mountpoint", because /srv is long gone. These
# two vars declare the state to migrate FROM. Empty by default: dev's label and mountpoint
# are NOT changing, so dev has nothing to transition and cannot enter the transition path.
node_baseline_data_retired_fs_label: ""
node_baseline_data_retired_mount: ""
# Sanity floor for the claim path: a data partition smaller than this is not "a small
# store", it is the wrong partition. Prod's /srv is 415-425 GiB.
node_baseline_longhorn_min_size_gb: 100
# Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
# so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
# See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
# device renames (matches the data partition's LABEL= approach).
node_baseline_etcd_source: device
node_baseline_etcd_device: /dev/vdc
# DERIVED, never restated: etcd's db always lives at <k3s data dir>/server/db, so moving
# the data dir moves etcd with it and the two cannot drift. Dev resolves to the unchanged
# /var/lib/rancher/k3s/server/db (where dev's dedicated etcd SSD is mounted); prod resolves
# to /data/k3s/server/db, inside the claimed partition.
node_baseline_etcd_mount: "{{ node_baseline_k3s_data_dir }}/server/db"
node_baseline_etcd_fs_label: k3s-etcd
# Mounts whose backing devices the claim path must NEVER touch. Used to build the
# protected-device set from the node's own gathered facts (plus live swap), so a node
# partitioned differently from the fleet is judged on its own layout rather than against
# a hardcoded list of partition names.
node_baseline_protected_mounts:
  - /
  - /boot
  - /boot/efi
  - /usr
  - /var
  - /home
  - /tmp

# --- WIRELESS (tasks/main.yml, tag `wireless`) ---------------------------------------
# A CLUSTER NODE GETS EXACTLY ONE PATH ONTO THE NETWORK. ON by default — unlike the
# backports kernel above, which is a prod-only deviation, this is a cluster-wide policy and
# it is a provable NO-OP where it does not apply: the tasks act only on interfaces the node
# reports as real 802.11 devices (a `phy80211` link in sysfs), and dev's Proxmox VMs have
# none, so on dev nothing is written, nothing is unloaded and nothing changes. Defaulting it
# true is what makes a FOURTH node with a wifi card inherit the policy from being a cluster
# node, rather than from someone remembering to set a flag.
#
# WHY IT EXISTS: k3s-prod-3 came up on both eno1 (192.168.2.12, the inventory address) and
# wlp1s0 (192.168.2.195, DHCP on the same /24). The wifi lease's dhcpcd run rewrote
# /etc/resolv.conf three minutes after this role wrote it, and a second address on the LAN
# where kube-vip floats 192.168.2.9 and Cilium L2-announces .20-.30 is a problem in its own
# right. See the `wireless` block in tasks/main.yml for the full reasoning, including why the
# DRIVER MODULE is the layer that gets blocked rather than the link, a service, or rfkill.
#
# SET IT FALSE for a host that must genuinely keep wifi (there is none today). That does NOT
# undo an already-applied block — delete the file below on the node as well.
node_baseline_disable_wireless: true
# Where the persistent block lives. 99- so it is read last and overrides any earlier
# permissive rule; the name says what it does, on a node with no repo checkout to hand.
node_baseline_wireless_modprobe_file: /etc/modprobe.d/99-k3s-no-wireless.conf

# --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
# These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml
# (the same seam kube_vip_address / k3s_api_host / cluster_kubeconfig_name use) — NOT in
# clusters/base/cluster-vars.yaml, which is a Flux postBuild ConfigMap that no Ansible
# code reads.
#
# WHY: the nodes shipped with ONE upstream resolver. PiHole answers the local jackson.lan
# zone instantly but its external recursion flaps (measured 0/25 for registry-1.docker.io /
# quay.io / ghcr.io while gitea.jackson.lan resolved 5/5), and one upstream is a hard SPOF:
# it broke containerd image pulls ("lookup registry-1.docker.io: Try again") AND CoreDNS,
# which forwards to /etc/resolv.conf and flooded its log with i/o timeouts to 192.168.1.18.
#
# ORDER IS THE POLICY. This is a RESILIENCE fix, not a bypass: the first entry stays PiHole,
# so *.jackson.lan and network-wide ad-blocking keep working exactly as before, and the
# second is only ever consulted when the first fails. Both write paths preserve that order
# (systemd-resolved tries the servers of one scope in order; glibc walks `nameserver` lines
# in order). Never put a public resolver first, and never enable `rotate` — either would
# silently take the local zone and the ad-block policy out of the path.
#
# Keep in parity with infra/tofu (var.dns_server, var.network_gateway), which sets the same
# two servers via cloud-init so a freshly rebuilt dev node boots with this list already.
node_baseline_dns_servers:
  - 192.168.1.18   # PiHole — local jackson.lan zone + ad-blocking. PRIMARY, keep first.
  - 192.168.1.1    # pfSense gateway — reliable external recursion. FALLBACK only.
# Search domain(s). An EMPTY list means "configure no search domain" and is a legitimate
# value (prod uses it — its local domain is not a decided thing), not a missing one.
node_baseline_dns_search:
  - jackson.lan
# resolv.conf `options` for the DIRECT-file mechanism only (systemd-resolved does its own
# per-server failover timing and ignores these). glibc's defaults are timeout:5 attempts:2,
# i.e. up to 20s of stalled lookups before it reaches the second server — long enough for
# containerd to give up on a pull. These shorten the failover, and deliberately do NOT
# include `rotate`, which would round-robin the servers and break PiHole-first ordering.
node_baseline_dns_options:
  - timeout:2
  - attempts:2