jakman85andClaude Opus 4.8
af5fc5b25f
Interim bridge to stop Longhorn replica-rebuild I/O storms on the shared HDD
Storage pool from starving etcd WAL fsync (crash-looping k3s / flapping the
kube-vip VIP). Adds a small dedicated SSD disk per node for etcd only; Longhorn
stays on HDD. See docs/plans/ssd-etcd-isolation.md.
tofu (infra/tofu):
- New vars etcd_ssd_datastore (default "local-zfs" = rpool/data, Micron 5100
SATA SSD mirror w/ PLP) and etcd_disk_gib (default 16).
- New virtio2 disk block on the SSD datastore per VM: discard=on, cache=none
(honor guest fsync straight to the ZFS ZIL — safe on PLP SSD), iothread=false.
Kept last (ascending interface) so bpg does not reorder virtio0/virtio1.
- Revert the debian_cloud_image resource to the long type name
proxmox_virtual_environment_download_file to match the R2 state: bpg >= 0.111
dropped the cross-type moved{} linkage to the short proxmox_download_file, so
the stale moved block failed EVERY plan with "Resource type mismatch" (broken
on main too). Aligning code to state yields a clean plan with no image
re-download. Short-name migration is deferred to a deliberate `tofu state mv`.
ansible (node_baseline):
- Format /dev/vdc (label k3s-etcd) and mount it at
/var/lib/rancher/k3s/server/db (defaults,noatime; no nofail so a missing etcd
disk fails loudly rather than silently landing etcd on the root HDD) before
k3s starts, so a rebuilt node comes up with etcd isolated on SSD. Data-safe:
filesystem module never reformats an existing fs.
Phase 1 only: non-disruptive. No tofu apply, no node/VM/etcd/Longhorn changes.
tofu plan verified: 0 to add, 3 to change, 0 to destroy — every VM an in-place
disk ADD (virtio2), dev-3 also an in-place memory 16384->24576; no VM replace.
Merge gate green (render==golden included).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
109 lines
4.1 KiB
YAML
109 lines
4.1 KiB
YAML
---
|
|
- name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
|
|
ansible.builtin.apt:
|
|
name: "{{ node_baseline_packages }}"
|
|
state: present
|
|
update_cache: true
|
|
cache_valid_time: 3600
|
|
|
|
- name: Persist kernel modules for k3s/Cilium/Longhorn
|
|
ansible.builtin.copy:
|
|
dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: "{{ node_baseline_kernel_modules | join('\n') }}\n"
|
|
|
|
- name: Load kernel modules now
|
|
community.general.modprobe:
|
|
name: "{{ item }}"
|
|
state: present
|
|
loop: "{{ node_baseline_kernel_modules }}"
|
|
|
|
- name: Install sysctl tuning for k3s/Cilium
|
|
ansible.builtin.copy:
|
|
dest: /etc/sysctl.d/99-k3s-cilium.conf
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
net.bridge.bridge-nf-call-iptables = 1
|
|
net.bridge.bridge-nf-call-ip6tables = 1
|
|
net.ipv4.ip_forward = 1
|
|
net.ipv6.conf.all.forwarding = 1
|
|
fs.inotify.max_user_instances = 8192
|
|
fs.inotify.max_user_watches = 524288
|
|
notify: Reload sysctl
|
|
|
|
- name: Enable and start iscsid (Longhorn prerequisite)
|
|
ansible.builtin.systemd:
|
|
name: iscsid
|
|
enabled: true
|
|
state: started
|
|
|
|
# qemu-guest-agent binds to the virtio-serial channel /dev/virtio-ports/org.qemu.guest_agent.0,
|
|
# which Proxmox attaches only when the VM's `agent` option is enabled. Our tofu keeps agent
|
|
# DISABLED on purpose (enabling it makes `tofu apply` block up to 15m waiting for an agent the
|
|
# stock Debian cloud image hasn't installed yet — bpg gotcha). So we ENABLE the unit for the day
|
|
# a channel exists, but only START it when the channel is actually present — never fail the
|
|
# baseline over a missing hypervisor channel.
|
|
- name: Check whether the qemu guest-agent virtio channel is attached
|
|
ansible.builtin.stat:
|
|
path: /dev/virtio-ports/org.qemu.guest_agent.0
|
|
register: node_baseline_qga_channel
|
|
|
|
- name: Enable qemu-guest-agent (start only when its virtio channel is present)
|
|
ansible.builtin.systemd:
|
|
name: qemu-guest-agent
|
|
enabled: true
|
|
state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
|
|
|
|
- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
|
|
community.general.filesystem:
|
|
fstype: ext4
|
|
dev: "{{ node_baseline_longhorn_device }}"
|
|
opts: "-L {{ node_baseline_longhorn_fs_label }}"
|
|
|
|
- name: Mount the Longhorn data disk
|
|
ansible.posix.mount:
|
|
path: "{{ node_baseline_longhorn_mount }}"
|
|
src: "LABEL={{ node_baseline_longhorn_fs_label }}"
|
|
fstype: ext4
|
|
opts: "defaults,nofail,x-systemd.device-timeout=5s"
|
|
state: mounted
|
|
|
|
# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
|
|
# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
|
|
# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
|
|
# a disk that already carries a filesystem, so re-running against an already-migrated
|
|
# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
|
|
# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
|
|
# role would mount over a populated directory.
|
|
- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
|
|
community.general.filesystem:
|
|
fstype: ext4
|
|
dev: "{{ node_baseline_etcd_device }}"
|
|
opts: "-L {{ node_baseline_etcd_fs_label }}"
|
|
|
|
- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
|
|
ansible.posix.mount:
|
|
path: "{{ node_baseline_etcd_mount }}"
|
|
src: "LABEL={{ node_baseline_etcd_fs_label }}"
|
|
fstype: ext4
|
|
# noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
|
|
# missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
|
|
opts: "defaults,noatime"
|
|
dump: "0"
|
|
passno: "2"
|
|
state: mounted
|
|
|
|
- name: Disable swap now
|
|
ansible.builtin.command: swapoff -a
|
|
changed_when: false
|
|
|
|
- name: Neutralize swap entries in fstab (kubelet requires swap off)
|
|
ansible.builtin.replace:
|
|
path: /etc/fstab
|
|
regexp: '^([^#].*\sswap\s.*)$'
|
|
replace: '# \1 # disabled for k3s/kubelet'
|