jakman85andClaude Opus 5
f5353cbd30
A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
holds itself: the announcer does not answer its own ARP request, so on the
lease holder the neighbour entry settles at FAILED and packets to that VIP die
before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes
x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced.
platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and
MASQUERADEs remote clients to the node's own address, so a VPN client takes
the node's LAN path and inherits its ARP failures. The router's udp/51820
forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16
Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering
(its VIP has two Services and k3s-prod-2 held the other lease). The tunnel
handshook, names resolved, every app hung, and nothing reported it.
on k3s-prod-1: ip neigh show 192.168.2.20 -> FAILED (its own VIP)
on prod-2/3: ip neigh show 192.168.2.20 -> lladdr ... (prod-1), 443 OK
Mark the node and exclude it from announcing:
* node_baseline labels the forwarded node (node_baseline_vpn_ingress_node,
k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes
the label from any node that no longer matches so moving it MOVES the label.
* l2-announcement-policy.yaml excludes that label, so the node ARP-resolves
every VIP. Fixes all VIPs for VPN clients, not just the one that collided.
Cost: announcement candidacy drops from 3 nodes to 2.
* VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels
(label added to the kube-state-metrics allowlist), with
VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches,
so a missing label would silently re-arm the whole failure AND leave the
critical alert inert. Four promtool cases, including that inertness.
A node LABEL rather than a cluster-vars knob because the alert must read it
too, and extract_rules.py rightly refuses a ${...} placeholder in an alert
expr -- the rule unit-tested would not be the rule deployed. One source of
truth, and it touches no sensitive path.
Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and
"failover is this one line". Both false; the first caused this.
UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the
handoff entry's third section.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
289 lines
19 KiB
YAML
289 lines
19 KiB
YAML
---
|
|
# node_baseline — first-boot floor for k3s/Cilium/Longhorn, moved out of cloud-init
|
|
# (Proxmox provisioning is API-only). Runs before k3s (site.yml). Idempotent.
|
|
node_baseline_packages:
|
|
# VM-only in practice (binds the virtio-serial channel Proxmox attaches) but harmless to install
|
|
# unconditionally: the task below only *starts* it when that channel exists, so on bare metal
|
|
# (prod) it installs and stays a dormant, disabled unit. Kept in the shared list rather than
|
|
# split per-platform so dev and prod install from the exact same package set (R-ENV-02 reuse).
|
|
- qemu-guest-agent
|
|
- open-iscsi
|
|
- nfs-common
|
|
- curl
|
|
- cryptsetup # Longhorn >=1.12 RequiredPackages check needs the cryptsetup userspace tool (dm_crypt module alone isn't enough)
|
|
# Drive health (smartctl) is a PERMANENT part of a bare-metal node's baseline, not a one-off
|
|
# manual install: prod runs on real disks that can fail/degrade (see infra/PROD-PROVISIONING.md
|
|
# §2/§5 — a ~15-year-old consumer SSD on node 1), where dev's virtio disks never needed it.
|
|
# Installed here (not hand-run with apt-get) so it's declared, idempotent, and present on every
|
|
# node from first baseline — package installation is the role's job, never a manual step
|
|
# (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo).
|
|
- smartmontools
|
|
# --- BACKPORTS KERNEL (EFI memattr, PROD-PROVISIONING.md §0b/§3) ---------------------
|
|
# OFF BY DEFAULT, ON FOR PROD. This is a STANDING, DELIBERATE DEVIATION from the Debian 13
|
|
# baseline — not a one-off `apt install` someone has to remember on the next node.
|
|
#
|
|
# WHY: HP's R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI Memory
|
|
# Attributes Table. Linux < 6.13 does not validate it before `memblock_reserve()`ing memory
|
|
# from its contents, so an unlucky boot reserves everything above the table and the node
|
|
# comes up with a fraction of its RAM (k3s-prod-2: 2.6 GiB of 16 GB, twice). Upstream commit
|
|
# 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" (Ard Biesheuvel,
|
|
# first released in v6.13) adds the validation. It is NOT in v6.12 and NOT in linux-6.12.y —
|
|
# no `Cc: stable`, no `Fixes:`, so no automatic feed will ever pick it up. There is also NO
|
|
# kernel cmdline workaround: efi_memattr_init() runs before any `efi=` bail-out. Booting a
|
|
# >= 6.13 kernel is the only fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
|
|
#
|
|
# WHY A VARIABLE AND NOT A HOSTNAME TEST. The dev nodes are Proxmox VMs whose firmware
|
|
# publishes no MAT at all — they are genuinely unaffected and must NOT be dragged onto a
|
|
# backports kernel as a side effect of sharing this role (R-ENV-02 is "same role, scoped
|
|
# overrides", not "same kernel"). A `when: inventory_hostname is match('k3s-prod-')` would
|
|
# encode the environment into the ROLE, would silently miss a fourth prod node named
|
|
# differently, and could not be flipped for a one-off test host. So it is a plain boolean,
|
|
# default false, turned on in inventory/group_vars/k3s_prod/vars.yml — the SAME seam
|
|
# kube_vip_address / k3s_api_host / cluster_kubeconfig_name already use.
|
|
#
|
|
# SECURITY COST, STATED PLAINLY: packages in *-backports are provided AS-IS. They are NOT
|
|
# covered by the Debian Security Team's guarantees for the main archive — no security-tracker
|
|
# commitment, no DSAs. In exchange for that the node keeps the RAM it paid for. Accepting it
|
|
# is the operator's recorded decision (PROD-PROVISIONING.md §0b/§3), not this role's.
|
|
#
|
|
# EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y (and therefore in Debian's
|
|
# 6.12 rebase), flip this back to false everywhere, `apt purge` the backports kernel, and
|
|
# delete this block. Nothing else in the repo depends on it.
|
|
node_baseline_backports_kernel: false
|
|
# The suite, mirror and components the source line is built from. Split out so a mirror
|
|
# change is a variable, not an edit to a heredoc in tasks/main.yml.
|
|
node_baseline_backports_suite: trixie-backports
|
|
node_baseline_backports_mirror: "http://deb.debian.org/debian"
|
|
# Matches /etc/apt/sources.list on these nodes (main + non-free-firmware). Deliberately NOT
|
|
# contrib/non-free: this deviation exists to get ONE fixed kernel, and widening the archive
|
|
# surface beyond what the node already trusts is a separate decision nobody has taken.
|
|
node_baseline_backports_components:
|
|
- main
|
|
- non-free-firmware
|
|
# The METApackage, not a pinned ABI (e.g. linux-image-6.19.14+deb13-amd64). A frozen ABI
|
|
# would stop receiving even backports' own updates the moment the ABI rolls, leaving the
|
|
# node on a kernel nothing patches — strictly worse than the risk this deviation accepts.
|
|
# The metapackage tracks whatever backports currently ships (>= 6.13 by construction, and
|
|
# far above it in practice), and `ButAutomaticUpgrades: yes` in the backports Release file
|
|
# means `apt upgrade` keeps it current once installed. The EXACT version each node landed on
|
|
# is recorded on the node itself — see node_baseline_backports_kernel_stamp.
|
|
node_baseline_backports_kernel_packages:
|
|
- linux-image-amd64
|
|
# Provenance/rollback note written on the node. An operator standing in front of a prod box
|
|
# with no repo checkout must be able to answer "why is this machine not on the distro
|
|
# kernel, and how do I undo it?" from the machine itself.
|
|
node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel
|
|
|
|
# Modules written to /etc/modules-load.d/k3s-cilium-longhorn.conf (persistent) AND modprobe'd now.
|
|
node_baseline_kernel_modules:
|
|
- overlay
|
|
- br_netfilter
|
|
- iscsi_tcp
|
|
- dm_crypt
|
|
- nfs
|
|
# Remote-access VPN (platform/vpn-ingress). The WireGuard DaemonSet creates its wg0 link in the
|
|
# HOST netns with CAP_NET_ADMIN only -- it is deliberately NOT granted CAP_SYS_MODULE, so it
|
|
# cannot autoload this itself: `ip link add wg0 type wireguard` fails outright when the module is
|
|
# absent, and the pod CrashLoops. Loading it here is therefore a PREREQUISITE of that component,
|
|
# not a nicety. In-tree since kernel 5.6, so Debian 13's 7.1.8 ships it and there is nothing to
|
|
# install. Naming it here also means the kernel does the crypto: without the module a WireGuard
|
|
# container's only fallback is slow userspace wireguard-go, which we do not use (and which would
|
|
# need /dev/net/tun -- a device a hostPath mount cannot actually grant, as the /dev/dri attempt
|
|
# proved: visible but EPERM from the device cgroup).
|
|
- wireguard
|
|
|
|
# --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
|
|
# The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
|
|
# a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:
|
|
#
|
|
# node_baseline_longhorn_source:
|
|
# device dedicated block device at node_baseline_longhorn_device (DEV: the
|
|
# tofu-provisioned virtio disk /dev/vdb). mkfs + mount by label.
|
|
# claim-mountpoint re-purpose the partition the node ALREADY has mounted at
|
|
# node_baseline_longhorn_claim_mountpoint (PROD: /srv). The device
|
|
# path is discovered ON THE NODE — it is /dev/sda5 on prod nodes 1
|
|
# and 3 but /dev/nvme0n1p5 on node 2, so no group-wide device value
|
|
# is correct and a hardcoded one would eventually mkfs the wrong
|
|
# partition on a future node. Destructive; heavily guarded — read
|
|
# tasks/disks.yml before changing anything here.
|
|
#
|
|
# node_baseline_etcd_source:
|
|
# device dedicated block device at node_baseline_etcd_device (DEV:
|
|
# /dev/vdc, docs/plans/ssd-etcd-isolation.md — etcd WAL fsync
|
|
# isolated from Longhorn rebuild I/O after dev was burned by HDD
|
|
# contention).
|
|
# root-filesystem etcd lives on the OS filesystem, with NO dedicated device and
|
|
# NO claimed data partition under it. Kept as a valid, supported
|
|
# shape; NOT what prod uses any more (see below).
|
|
# shared-data-partition etcd lives inside node_baseline_k3s_data_dir, which sits on the
|
|
# CLAIMED data partition (PROD). No dedicated device, but also NOT
|
|
# the OS filesystem: the whole point of the 2026-09-04 layout
|
|
# revision is that /var (11.8 GiB) can hold neither the container
|
|
# image store nor etcd's WAL. See infra/PROD-PROVISIONING.md §2.
|
|
#
|
|
# Every one of these is a FIRST-CLASS, explicitly-configured state — an operator decision
|
|
# recorded in infra/PROD-PROVISIONING.md §2 — and NOT the same thing as an unset device
|
|
# var, which still fails the `disks` assert.
|
|
node_baseline_longhorn_source: device
|
|
node_baseline_longhorn_device: /dev/vdb
|
|
# --- The DATA PARTITION: mountpoint + filesystem label -------------------------------
|
|
# NEUTRAL NAMES ON PURPOSE (revised 2026-09-04). On prod this ONE partition now serves TWO
|
|
# tenants — Longhorn's replica store AND k3s's whole data dir (etcd + the containerd image
|
|
# store) — so neither the mountpoint nor the label may name only one of them. Prod overrides
|
|
# both in group_vars/k3s_prod/vars.yml (`/data`, label `k3s-data`).
|
|
#
|
|
# THE DEFAULTS BELOW ARE DEV'S LIVE VALUES AND MUST NOT CHANGE. The dev cluster has 18
|
|
# Longhorn volumes whose replicas live on the partition labelled `longhorn` mounted at
|
|
# /var/lib/longhorn; dev's etcd has its own dedicated device and dev's k3s data dir stays
|
|
# at k3s's own default. Dev's mountpoint IS Longhorn's dataPath there (one tenant, one
|
|
# partition), which is why node_baseline_longhorn_data_path below defaults to the mount.
|
|
node_baseline_data_mount: /var/lib/longhorn
|
|
node_baseline_data_fs_label: longhorn
|
|
# Longhorn's dataPath — the directory Longhorn puts replicas in. On dev that IS the mount
|
|
# root (single tenant). On prod it is a SUBDIRECTORY of the mount, because k3s owns a
|
|
# sibling subdirectory of the same partition. Must agree with the LONGHORN_DATA_PATH knob
|
|
# in clusters/base/cluster-vars.yaml (patched per overlay) — that is the Helm side of the
|
|
# same fact, and the two are checked against each other by nothing but this comment.
|
|
node_baseline_longhorn_data_path: "{{ node_baseline_data_mount }}"
|
|
# k3s's --data-dir. Defers to the CLUSTER-WIDE `k3s_data_dir` when an environment declares
|
|
# one (group_vars), and otherwise restates k3s's own built-in default. It is deliberately
|
|
# NOT a second independent variable: roles/k3s_server reads `k3s_data_dir` to build the
|
|
# install flags, this role reads the same value to create the directory and to place etcd,
|
|
# and one line in group_vars/k3s_prod/vars.yml moves BOTH. Lazy Jinja means the group_vars
|
|
# value (higher precedence than any role default) is what actually resolves here, even
|
|
# though roles/k3s_server is not in the node-baseline play; the default() keeps
|
|
# node-baseline.yml runnable standalone.
|
|
node_baseline_k3s_data_dir: "{{ k3s_data_dir | default('/var/lib/rancher/k3s') }}"
|
|
# Only consulted when node_baseline_longhorn_source is `claim-mountpoint`. Empty by
|
|
# default so a dev host cannot enter the claim path even by accident.
|
|
node_baseline_longhorn_claim_mountpoint: ""
|
|
# --- ONE-TIME TRANSITION KNOBS (delete once the fleet is converged) ------------------
|
|
# A node claimed by an EARLIER version of this role carries the OLD label at the OLD
|
|
# mountpoint, and the idempotency pivot (`blkid -L <current label>`) would therefore read
|
|
# "not claimed yet" and walk into the greenfield claim path — which then fails closed at
|
|
# "the claim mountpoint must actually be a mountpoint", because /srv is long gone. These
|
|
# two vars declare the state to migrate FROM. Empty by default: dev's label and mountpoint
|
|
# are NOT changing, so dev has nothing to transition and cannot enter the transition path.
|
|
node_baseline_data_retired_fs_label: ""
|
|
node_baseline_data_retired_mount: ""
|
|
# Sanity floor for the claim path: a data partition smaller than this is not "a small
|
|
# store", it is the wrong partition. Prod's /srv is 415-425 GiB.
|
|
node_baseline_longhorn_min_size_gb: 100
|
|
# Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
|
|
# so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
|
|
# See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
|
|
# device renames (matches the data partition's LABEL= approach).
|
|
node_baseline_etcd_source: device
|
|
node_baseline_etcd_device: /dev/vdc
|
|
# DERIVED, never restated: etcd's db always lives at <k3s data dir>/server/db, so moving
|
|
# the data dir moves etcd with it and the two cannot drift. Dev resolves to the unchanged
|
|
# /var/lib/rancher/k3s/server/db (where dev's dedicated etcd SSD is mounted); prod resolves
|
|
# to /data/k3s/server/db, inside the claimed partition.
|
|
node_baseline_etcd_mount: "{{ node_baseline_k3s_data_dir }}/server/db"
|
|
node_baseline_etcd_fs_label: k3s-etcd
|
|
# Mounts whose backing devices the claim path must NEVER touch. Used to build the
|
|
# protected-device set from the node's own gathered facts (plus live swap), so a node
|
|
# partitioned differently from the fleet is judged on its own layout rather than against
|
|
# a hardcoded list of partition names.
|
|
node_baseline_protected_mounts:
|
|
- /
|
|
- /boot
|
|
- /boot/efi
|
|
- /usr
|
|
- /var
|
|
- /home
|
|
- /tmp
|
|
|
|
# --- WIRELESS (tasks/main.yml, tag `wireless`) ---------------------------------------
|
|
# A CLUSTER NODE GETS EXACTLY ONE PATH ONTO THE NETWORK. ON by default — unlike the
|
|
# backports kernel above, which is a prod-only deviation, this is a cluster-wide policy and
|
|
# it is a provable NO-OP where it does not apply: the tasks act only on interfaces the node
|
|
# reports as real 802.11 devices (a `phy80211` link in sysfs), and dev's Proxmox VMs have
|
|
# none, so on dev nothing is written, nothing is unloaded and nothing changes. Defaulting it
|
|
# true is what makes a FOURTH node with a wifi card inherit the policy from being a cluster
|
|
# node, rather than from someone remembering to set a flag.
|
|
#
|
|
# WHY IT EXISTS: k3s-prod-3 came up on both eno1 (192.168.2.12, the inventory address) and
|
|
# wlp1s0 (192.168.2.195, DHCP on the same /24). The wifi lease's dhcpcd run rewrote
|
|
# /etc/resolv.conf three minutes after this role wrote it, and a second address on the LAN
|
|
# where kube-vip floats 192.168.2.9 and Cilium L2-announces .20-.30 is a problem in its own
|
|
# right. See the `wireless` block in tasks/main.yml for the full reasoning, including why the
|
|
# DRIVER MODULE is the layer that gets blocked rather than the link, a service, or rfkill.
|
|
#
|
|
# SET IT FALSE for a host that must genuinely keep wifi (there is none today). That does NOT
|
|
# undo an already-applied block — delete the file below on the node as well.
|
|
node_baseline_disable_wireless: true
|
|
# Where the persistent block lives. 99- so it is read last and overrides any earlier
|
|
# permissive rule; the name says what it does, on a node with no repo checkout to hand.
|
|
node_baseline_wireless_modprobe_file: /etc/modprobe.d/99-k3s-no-wireless.conf
|
|
|
|
# --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
|
|
# These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml
|
|
# (the same seam kube_vip_address / k3s_api_host / cluster_kubeconfig_name use) — NOT in
|
|
# clusters/base/cluster-vars.yaml, which is a Flux postBuild ConfigMap that no Ansible
|
|
# code reads.
|
|
#
|
|
# WHY: the nodes shipped with ONE upstream resolver. PiHole answers the local jackson.lan
|
|
# zone instantly but its external recursion flaps (measured 0/25 for registry-1.docker.io /
|
|
# quay.io / ghcr.io while gitea.jackson.lan resolved 5/5), and one upstream is a hard SPOF:
|
|
# it broke containerd image pulls ("lookup registry-1.docker.io: Try again") AND CoreDNS,
|
|
# which forwards to /etc/resolv.conf and flooded its log with i/o timeouts to 192.168.1.18.
|
|
#
|
|
# ORDER IS THE POLICY. This is a RESILIENCE fix, not a bypass: the first entry stays PiHole,
|
|
# so *.jackson.lan and network-wide ad-blocking keep working exactly as before, and the
|
|
# second is only ever consulted when the first fails. Both write paths preserve that order
|
|
# (systemd-resolved tries the servers of one scope in order; glibc walks `nameserver` lines
|
|
# in order). Never put a public resolver first, and never enable `rotate` — either would
|
|
# silently take the local zone and the ad-block policy out of the path.
|
|
#
|
|
# Keep in parity with infra/tofu (var.dns_server, var.network_gateway), which sets the same
|
|
# two servers via cloud-init so a freshly rebuilt dev node boots with this list already.
|
|
node_baseline_dns_servers:
|
|
- 192.168.1.18 # PiHole — local jackson.lan zone + ad-blocking. PRIMARY, keep first.
|
|
- 192.168.1.1 # pfSense gateway — reliable external recursion. FALLBACK only.
|
|
# Search domain(s). An EMPTY list means "configure no search domain" and is a legitimate
|
|
# value (prod uses it — its local domain is not a decided thing), not a missing one.
|
|
node_baseline_dns_search:
|
|
- jackson.lan
|
|
# resolv.conf `options` for the DIRECT-file mechanism only (systemd-resolved does its own
|
|
# per-server failover timing and ignores these). glibc's defaults are timeout:5 attempts:2,
|
|
# i.e. up to 20s of stalled lookups before it reaches the second server — long enough for
|
|
# containerd to give up on a pull. These shorten the failover, and deliberately do NOT
|
|
# include `rotate`, which would round-robin the servers and break PiHole-first ordering.
|
|
node_baseline_dns_options:
|
|
- timeout:2
|
|
- attempts:2
|
|
# Static /etc/hosts entries (tags: dns, hosts), as a list of {ip, names}. EMPTY by default:
|
|
# dev needs none. Prod pins its OIDC issuer host to the Gateway VIP because the public
|
|
# answer for it is the non-hairpinned WAN IP (inventory/group_vars/k3s_prod/vars.yml).
|
|
# Written as ONE marked block, so removing an entry here removes it from the node, and
|
|
# an empty list removes the block entirely.
|
|
node_baseline_hosts_pins: []
|
|
|
|
# --- The node that terminates remote-access WireGuard (tags: vpn-ingress) ----------------------
|
|
# Hostname of the node the SOHO router forwards WAN udp/51820 to, i.e. the one platform/vpn-ingress
|
|
# actually serves remote clients from. Set it in group_vars for the environment; EMPTY by default,
|
|
# because dev has no such forward and because a wrong value is worse than none.
|
|
#
|
|
# WHAT IT IS FOR, AND WHY IT IS NOT COSMETIC. The matching node gets
|
|
# `homelab.keithjackson.dev/vpn-ingress=true`, and
|
|
# platform-config/cilium/l2-announcement-policy.yaml EXCLUDES that label from announcing any
|
|
# LoadBalancer VIP. MEASURED on prod 2026-10-08: a node cannot reach a VIP whose Cilium L2
|
|
# announcement lease it holds itself (the announcer does not answer its own ARP request -- the
|
|
# neighbour entry goes to `FAILED`). Because vpn-ingress runs on `hostNetwork` and MASQUERADEs
|
|
# clients to the node's own address, a VPN client inherits that: with the forward on the node
|
|
# holding the `shared` Gateway lease, all 16 Gateway hostnames were dead over the VPN while PiHole
|
|
# DNS still resolved. Names worked, every app hung, nothing alerted.
|
|
#
|
|
# [parity] MUST name the node the router's forward actually points at. The forward is off-repo and
|
|
# no check can read it (runbooks/vpn-remote-access.md step 3). Changing the forward is therefore a
|
|
# two-part change: this value (then re-run the role), and the router line.
|
|
#
|
|
# Only ONE node should ever carry the label: it removes that node from VIP announcement candidacy,
|
|
# so labelling all three would leave nothing announcing any VIP at all. The task below removes the
|
|
# label from a node that no longer matches, so moving the value moves the label.
|
|
node_baseline_vpn_ingress_node: ""
|
|
node_baseline_vpn_ingress_label_key: homelab.keithjackson.dev/vpn-ingress
|
|
# k3s ships its kubeconfig here; every prod node is a k3s server, so `k3s kubectl` works locally.
|
|
node_baseline_vpn_ingress_kubeconfig: /etc/rancher/k3s/k3s.yaml
|