home-network-k3s-impl/docs/plans/cluster-upgrade-plan.md

Cluster upgrade plan — k3s 1.31.5 → 1.36.x + node OS (dev cluster)

✅ COMPLETED / historical — executed 2026-08-30, see git log

This plan has been carried out. The dev cluster now runs k3s v1.36.4+k3s1 on Debian 13 (trixie) with Cilium 1.20.1 and Longhorn 1.12.1. The steps, gates, and rollbacks below are retained as the record of how the upgrade was performed — do NOT re-execute them against the live cluster. Current state is tracked in infra/PHASE1-STATUS.md and the 2026-08-30 DECISIONS-LOG entry.

Scope: the home-network-k3s-impl dev cluster (k3s-dev-1/2/3 @ 192.168.1.50-52), 3× HA embedded-etcd servers behind a kube-vip VIP, Debian 12 bookworm. Upgrade Kubernetes/k3s from v1.31.5+k3s1 → v1.36.x+k3s1 (five minors) and the node OS (Debian 12 → 13), with an etcd-snapshot / Velero / CNPG backup gate and explicit PASS/STOP criteria at every step, and a defined rollback for each hop.

Status: PLAN ONLY. Nothing here has been executed. All version pins marked "confirm at execution" must be re-checked against the upstream release page on the day of the hop — this doc was written 2026-08-29 and homelab component versions move weekly (Renovate).

Companion runbooks that this plan reuses wholesale:

• docs/plans/ssd-etcd-execution-runbook.md — the proven safe-bounce procedure (Longhorn rebuild-freeze + one-node-at-a-time + leader-last + per-node verify gates). Every k3s hop below is a parameterised repeat of that runbook.
• docs/plans/ssd-etcd-isolation.md — why etcd is fragile on this cluster (slow HDD; etcd now on a dedicated SSD virtio2) and how a single member can be reset+rejoined from the other two.

---

0. TL;DR / operator decisions needed

#	Decision	Recommendation	Needs operator sign-off
D1	k3s upgrade method	Scripted in-place per-node binary upgrade driven by Ansible, one node at a time, leader-last, with the Longhorn rebuild-freeze. NOT system-upgrade-controller (its auto-drain fights this cluster's slow-HDD Longhorn and removes the human gate between nodes).	Yes — SUC vs scripted-in-place
D2	Node OS upgrade	Immutable re-provision via tofu (bump the cloud image to Debian 13 trixie, replace one node at a time, rejoin etcd), run as a separate campaign from the k3s hops. Not in-place apt dist-upgrade.	Yes — Debian 13 now, or stay on 12
D3	Do OS + k3s on the same node bounce?	No. Never combine. Finish the k3s ladder to a stable target, then run the OS campaign (or vice-versa), one variable per bounce.	Confirm
D4	Sequencing granularity	One k3s minor per maintenance window (5 windows). Cilium + Longhorn + Gateway-API CRDs advance in lockstep inside the same window (they are the pacing blockers).	Confirm
D5	Flux image.toolkit.fluxcd.io/v1beta2 → /v1 migration	Must be done before Flux is upgraded to v2.7+ (which is required for k8s 1.36). Do it early, decoupled.	Confirm timing

Headline blockers (do these or the hop fails):

1. Cilium 1.16.5 is already at/near its ceiling — Cilium 1.16 tops out around k8s 1.30-1.31. Cilium must step 1.16→1.17→1.18→1.19→1.20 in lockstep with the k8s minors. It is the single most urgent item.
2. Longhorn 1.7.3 must step 1.7→1.8→1.9→1.10→1.11→1.12 (one minor at a time) to reach a 1.36-validated release (1.12.x).
3. Flux v2.4.0 → v2.7+ is required for k8s 1.36, and v2.7 removed the image-automation v1beta2 API the repo still uses.
4. KubeVirt v1.9.0 / CDI v1.66.0 are already 1.36-ready — a pleasant surprise; no VM-stack upgrade needed, just keep them where they are.

---

1. Confirmed current-state inventory

1.1 Nodes (live, verified 2026-08-29 via dev.sh run kubectl get nodes -o wide)

Node	IP	Roles	k8s	OS	Kernel	Runtime
k3s-dev-1	192.168.1.50	control-plane,etcd,master	v1.31.5+k3s1	Debian 12 bookworm	6.1.0-52-cloud	containerd 1.7.23-k3s2
k3s-dev-2	192.168.1.51	control-plane,etcd,master	v1.31.5+k3s1	Debian 12 bookworm	6.1.0-52-cloud	containerd 1.7.23-k3s2
k3s-dev-3	192.168.1.52	control-plane,etcd,master	v1.31.5+k3s1	Debian 12 bookworm	6.1.0-52-cloud	containerd 1.7.23-k3s2

• 3 servers, embedded etcd, all control-plane (no dedicated workers). Quorum = 2/3.
• kube-vip control-plane VIP 192.168.1.49 (fronted through pfSense NAT at 192.168.2.50:6443).
• etcd datastore is on the dedicated SSD virtio2 (/dev/vdc, LABEL=k3s-etcd) mounted at /var/lib/rancher/k3s/server/db (migrated this session). Longhorn data disk = virtio1 (/dev/vdb, /var/lib/longhorn). OS = virtio0 (/dev/vda).
• 18 Longhorn volumes, all healthy (the count is the post-step invariant).

1.2 Where versions are pinned in this repo

Thing	Pin location	Current value
k3s version	infra/ansible/roles/k3s_server/defaults/main.yml → k3s_version; applied via INSTALL_K3S_VERSION in roles/k3s_server/tasks/main.yml	v1.31.5+k3s1
k3s install guard	roles/k3s_server/tasks/main.yml — install task has creates: /etc/rancher/k3s/k3s.yaml	idempotent → will NOT upgrade a running node on a version bump (see §4)
kubectl/helm/flux (toolbox)	infra/devcontainer/versions.env, ci/versions.env	kubectl 1.31.4, flux 2.4.0
Cilium (ongoing)	platform/cilium/helmrelease.yaml → version: "1.16.5"	1.16.5
Cilium (bootstrap) + Gateway API CRDs	infra/ansible/roles/cilium/defaults/main.yml → gateway_api_version: "v1.2.1", gateway_api_crd_urls	v1.2.1
Longhorn	platform/longhorn/helmrelease.yaml → version: ">=1.7.0 <1.8.0" (live 1.7.3)	1.7.3
KubeVirt + CDI	platform/kubevirt/kustomization.yaml (remote release manifests)	KubeVirt v1.9.0, CDI v1.66.0
kube-vip	infra/ansible/roles/kube_vip/defaults/main.yml → kube_vip_version	v0.8.9
external-snapshotter	platform/external-snapshotter/kustomization.yaml (ref=v8.2.0)	v8.2.0
cert-manager	platform/cert-manager/helmrelease.yaml → ">=1.16.0 <1.17.0" (live 1.16.5)	1.16.5
CNPG	platform/cnpg/helmrelease.yaml → ">=0.22.0 <1.0.0" (live chart 0.29.0)	operator ≈1.26/1.27 era
kube-prometheus-stack	platform/monitoring/kube-prometheus-stack.yaml → ">=65.0.0 <66.0.0" (live 65.8.1)	65.8.1
Loki	platform/monitoring/loki.yaml → ">=6.0.0 <7.0.0" (live 6.55.0)	6.55.0
Alloy	platform/monitoring/alloy.yaml → ">=0.10.0 <1.0.0" (live 0.12.6)	0.12.6
Velero	platform/velero/helmrelease.yaml → ">=8.0.0 <9.0.0" (live 8.7.2, Velero v1.15.x)	8.7.2
Kyverno	platform/kyverno/admission/helmrelease.yaml → version: "3.3.x" (live 3.3.9)	3.3.9
oauth2-proxy	platform/oauth2-proxy/helmrelease.yaml → ">=7.0.0 <8.0.0" (live 7.18.0)	7.18.0
external-dns	platform/externaldns/helmrelease.yaml → ">=1.15.0 <1.16.0" (live 1.15.2)	1.15.2
headlamp	platform/headlamp/helmrelease.yaml → ">=0.41.0 <0.42.0" (live 0.41.0)	0.41.0
Flux	installed by flux bootstrap (toolbox v2.4.0); controllers live in flux-system	v2.4.0
tofu cloud image	infra/tofu/variables.tf → image_source, image_file_name	debian-12-generic-amd64.qcow2 (bookworm)

1.3 Live platform component versions (verified 2026-08-29)

Flux 2.4.0 (source-controller v1.4.1, kustomize-controller v1.4.0, helm-controller v1.1.0, image-automation v0.39.0, image-reflector v0.33.0, notification v1.4.0) · cert-manager v1.16.5 · CNPG chart 0.29.0 (PostgreSQL 16.4, 3 single-instance clusters: keycloak-db, nextcloud-db, semaphore-db) · Cilium 1.16.5 · Kyverno 3.3.9 · Longhorn 1.7.3 · Alloy 0.12.6 · kube-prometheus-stack 65.8.1 · Loki 6.55.0 · oauth2-proxy 7.18.0 · Velero 8.7.2 (+plugin-for-aws v1.11.1) · KubeVirt v1.9.0 · CDI v1.66.0 · snapshot-controller v8.2.0 · kube-vip v0.8.9 · Gateway API CRDs v1 (v1.2.1 line).

1.4 CRD / API surface the repo actually uses (grep of platform/ apps/ catalog/ clusters/)

v1, apps/v1, networking.k8s.io/v1, rbac.authorization.k8s.io/v1, storage.k8s.io/v1, gateway.networking.k8s.io/v1, cilium.io/v2, monitoring.coreos.com/v1, helm.toolkit.fluxcd.io/v2, source.toolkit.fluxcd.io/v1, kustomize.toolkit.fluxcd.io/v1, image.toolkit.fluxcd.io/v1beta2 ⚠, postgresql.cnpg.io/v1, kubevirt.io/v1, cdi.kubevirt.io/v1beta1, cert-manager.io/v1, kyverno.io/v1, k8s.keycloak.org/v2alpha1, snapshot.storage.k8s.io/v1, client.authentication.k8s.io/v1beta1 (kubeconfig exec plugin).

None of these are Kubernetes core APIs removed in 1.32-1.36. The only removal that bites is a Flux CRD (image.toolkit.fluxcd.io/v1beta2) — see §3.

---

2. k3s upgrade path (sequential, one minor at a time)

2.1 The path and why skipping is unsafe

1.31.5+k3s1 → 1.32.z → 1.33.z → 1.34.z → 1.35.z → 1.36.z    (+k3s1 each)

Skipping minors is not supported. Kubernetes' version-skew policy allows the kube-apiserver and components to differ by at most one minor during an upgrade, and a downgrade of the etcd datastore is not supported. Jumping >1 minor risks an apiserver that can't read objects written by an intermediate storage version and control-plane components outside the skew window. On k3s (all three nodes are servers/apiserver), the whole control plane moves together per hop, so the rule is: one minor per campaign, all three nodes brought to the same minor before the next. Source: https://kubernetes.io/releases/version-skew-policy/.

2.2 Target patch per minor (confirm at execution)

Pick the latest patch of each minor from https://github.com/k3s-io/k3s/releases (or the k3s channel server https://update.k3s.io/v1-release/channels, where channel v1.3X resolves to the latest patch of that minor). Known-latest as of writing:

Hop	Target (min)	Notes / source
→ 1.32	latest v1.32.z+k3s1	confirm on releases page
→ 1.33	latest v1.33.z+k3s1	k3s 1.33 release notes: https://docs.k3s.io/release-notes/v1.33.X
→ 1.34	latest v1.34.z+k3s1	confirm
→ 1.35	latest v1.35.z+k3s1	confirm
→ 1.36	v1.36.3+k3s1 (Aug 4 2026) or newer (v1.36.4+k3s1 reported)	https://docs.k3s.io/release-notes/v1.36.X

2.3 k3s-specific breaking changes across the path

• containerd major bump 1.7 → 2.x. k3s 1.36.3 ships containerd v2.3.2-k3s2 (current node is 1.7.23-k3s2). The containerd v1→v2 migration happens somewhere in the 1.32-1.34 window. It is transparent for standard workloads, but: (a) config.toml schema changed — this repo sets device_ownership_from_security_context via --nonroot-devices (k3s config.yaml), which k3s regenerates into the new containerd config on restart; verify it survives (KubeVirt/CDI Block-mode PVCs depend on it — see roles/k3s_server/tasks/main.yml). (b) Any node with custom /etc/rancher/k3s/config.toml.tmpl would need review — this cluster has none.
• etcd 3.5 → 3.6. k3s 1.36.3 ships etcd v3.6.14-k3s1. etcd datastore version bumps are one-way (no downgrade). This is the single strongest reason for the pre-hop etcd snapshot gate (§6) and defines the point-of-no-return (§7).
• Traefik chart v40 breaking change in k3s ≥1.36.2 (ingress-nginx provider rename). N/A — this cluster disables Traefik (k3s_disable: "traefik,servicelb,local-storage"); Cilium Gateway API owns ingress.
• kube-proxy / flannel / network-policy already disabled (--disable-kube-proxy, --flannel-backend=none, --disable-network-policy) so k8s 1.36's IPVS-mode kube-proxy deprecation is N/A — Cilium is the kube-proxy replacement.
• Always read the Kubernetes urgent-upgrade notes for each minor before the hop: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/ (CHANGELOG-1.3X.md → "Urgent Upgrade Notes").

---

3. Removed / deprecated API audit (1.32 → 1.36)

3.1 Kubernetes core removals/deprecations per minor

k8s	Removed / deprecated	Affects this repo?
1.32	flowcontrol.apiserver.k8s.io/v1beta3 (FlowSchema, PriorityLevelConfiguration) removed — use flowcontrol.apiserver.k8s.io/v1 (GA since 1.29). Field rename assuredConcurrencyShares→nominalConcurrencyShares.	No — repo defines no FlowSchema/PriorityLevelConfiguration CRs.
1.33	status.nodeInfo.kubeProxyVersion field removed from Node.	No — nothing reads it.
1.34	AppArmor annotations path deprecations continue; AppArmor GA'd earlier.	No repo AppArmor annotations.
1.35	Minor graduations; no removal that touches this repo's API set.	No.
1.36	gitRepo volume permanently disabled; Service externalIPs deprecated; kube-proxy IPVS on a deprecation clock.	No — no gitRepo volumes, no externalIPs, kube-proxy already replaced by Cilium.

Sources: https://kubernetes.io/docs/reference/using-api/deprecation-guide/ · https://www.kubernetes.io/blog/2026/03/30/kubernetes-v1-36-sneak-peek/ · per-minor CHANGELOG-1.3X.md.

Conclusion: the repo's manifests use no Kubernetes core API removed in 1.32-1.36. (Grep for flowcontrol.apiserver, gitRepo:, externalIPs:, policy/v1beta1, batch/v1beta1, autoscaling/v2beta*, extensions/v1beta1, PodSecurityPolicy returns nothing.)

3.2 The one real removal that bites — Flux image.toolkit.fluxcd.io/v1beta2

Flux v2.7+ removed image.toolkit.fluxcd.io/v1beta2 (ImageRepository, ImagePolicy, ImageUpdateAutomation → graduated to image.toolkit.fluxcd.io/v1). Flux v2.7+ is required for k8s 1.36 (§4.1). The repo still uses v1beta2 in:

• platform/image-automation/image-update-automation.yaml
• platform/image-automation/per-service/pihole.example.yaml (plus any per-service ImageRepository/ImagePolicy generated from that template.)

Required migration (do BEFORE upgrading Flux to v2.7+): follow the official "Upgrade Procedure for Flux v2.7+" — migrate stored objects off v1beta2, then update every manifest in Git from image.toolkit.fluxcd.io/v1beta2 → image.toolkit.fluxcd.io/v1. Source: https://github.com/fluxcd/flux2/discussions/5572 · https://fluxcd.io/flux/components/image/imagerepositories/.

3.3 Detection tooling — run BEFORE each hop

Add a read-only pre-hop gate using both a live-cluster scan and a manifest scan:

# pluto — scans live cluster AND Helm releases for deprecated/removed apiVersions, target minor
./infra/devcontainer/dev.sh run bash -lc 'pluto detect-all-in-cluster --target-versions k8s=v1.3X.0 -owide'
# or scan the rendered manifests in git:
./infra/devcontainer/dev.sh run bash -lc 'pluto detect-files -d platform -d apps --target-versions k8s=v1.3X.0'

# kubent (kube-no-trouble) — quick live scan
./infra/devcontainer/dev.sh run bash -lc 'kubent --target-version 1.3X'

• pluto and kubent detect built-in deprecated APIs from their bundled version data; they do not know about third-party CRD removals (e.g. the Flux v1beta2 above) unless given a custom deprecation list. So pair the tool pass with the manual §3.2 check and a grep of the repo for each component's deprecated CRD versions before its own upgrade.
• Neither tool is currently in the toolbox image; add to infra/devcontainer/Containerfile + versions.env, or run once via go install/release binary inside the devcontainer.
• Sources: https://github.com/FairwindsOps/pluto · https://github.com/doitintl/kube-no-trouble.

---

4. Component compatibility matrix & upgrade sequencing

4.1 Kubernetes-version support per component (target = k8s 1.36)

Component	Current	k8s-1.36 target (confirm patch)	k8s support fact	Must move…	Source
Cilium	1.16.5	1.20.x	1.16:≤~1.30 · 1.17:1.29-1.32 · 1.18:1.30-1.33 · 1.19:1.32-1.35 · 1.20:1.33-1.36	in lockstep, every hop (pacing blocker #1)	compatibility.fyi/cilium; docs.cilium.io/…/compatibility
Gateway API CRDs	v1.2.1	v1.6.x (Cilium 1.20 targets v1.6.1 — verify)	too old for Cilium 1.20	with each Cilium hop	docs.cilium.io/en/v1.20/…/gateway-api
Longhorn	1.7.3	1.12.x (1.12.1)	1.7:≤~1.31 · 1.11.1:1.32-1.35 · 1.12:adds 1.36; min k8s stays v1.21	one minor at a time (pacing blocker #2)	endoflife.date/longhorn; longhorn.io best-practices; SUSE support matrix
KubeVirt	v1.9.0	v1.9.x (already there)	v1.9 validates 1.34/1.35/1.36 (N-2)	no upgrade needed	github.com/kubevirt/sig-release k8s-support-matrix
CDI	v1.66.0	keep paired w/ KubeVirt v1.9	same window as KubeVirt	no upgrade needed (keep pair)	kubevirt/containerized-data-importer releases
external-snapshotter / snapshot-controller	v8.2.0	latest v8.x (≥8.6)	v8 tracks current k8s; snapshot CRDs fwd/back-compatible	loose — opportunistic	github.com/kubernetes-csi/external-snapshotter releases
cert-manager	1.16.5	1.21.x	1.21 supports 1.33-1.36	before/with the later hops	cert-manager.io/docs/releases
CNPG	chart 0.29.0 (op ≈1.26/1.27)	1.30.x	1.30 supports 1.34/1.35/1.36; 1.29 supports 1.33-1.35	before the 1.36 hop	cloudnative-pg.io/docs/devel/supported_releases
Flux	v2.4.0	v2.7+ (latest, ~v2.8.x)	Flux N-2; v2.7+ needed for 1.36	after §3.2 v1beta2→v1 migration	fluxcd.io/flux/releases
kube-prometheus-stack	65.8.1	~88.x+ (supports kubectl/k8s 1.36)	chart 88.x tested on 1.36	before/with later hops; apply updated CRDs separately	github.com/prometheus-community/helm-charts
Loki (chart)	6.55.0	latest 6.x/newer	uses stable core APIs — loose	opportunistic	grafana/helm-charts
Alloy (chart)	0.12.6	latest	loose (DaemonSet, stable APIs)	opportunistic	grafana/alloy
Velero	chart 8.7.2 (v1.15)	latest (v1.17+/chart ~10.x)	v1.16 tested→1.33, v1.17 tested→1.34; "1.18-latest" expected	before the 1.36 hop	velero.io support-matrix; compatibility.fyi/velero
Kyverno	3.3.9	latest 3.x (N-2 for 1.36)	Kyverno follows k8s N-2; 3.3.x too old for 1.36	before/with later hops	kyverno.io/docs/installation/releases
oauth2-proxy	7.18.0	latest 7.x	loose	opportunistic	oauth2-proxy chart
external-dns	1.15.2	latest 1.x	loose	opportunistic	kubernetes-sigs/external-dns
kube-vip	v0.8.9	latest (k8s.io 0.36.x-based)	leader-election uses coordination.k8s.io/v1 (stable); latest bumps client-go to 0.36	before/with the 1.36 hop	github.com/kube-vip/kube-vip releases
headlamp	0.41.0	latest	loose UI	opportunistic	headlamp chart

4.2 The pacing blockers, and the per-hop lockstep

Cilium and Longhorn are the two items that force the pace — each supports only ~3-4 k8s minors and each upgrades one minor at a time (no skipping, pre-flight required). KubeVirt/CDI are already at target, so the VM stack is not a blocker this cycle. The mapping:

Window	k8s target	Cilium (HelmRelease)	Gateway API CRDs	Longhorn (HelmRelease)	Other components landed in this window
W1	1.32	1.17 (covers 1.29-1.32)	bump toward Cilium 1.17's required version	1.8	—
W2	1.33	1.18 (1.30-1.33)	bump	1.9	cert-manager→1.18/1.19; kube-prometheus-stack CRDs refresh
W3	1.34	1.19 (1.32-1.35)	bump	1.10	CNPG→1.28/1.29; Velero→latest
W4	1.35	1.19 still covers 1.35 (optional →1.20 early)	as needed	1.11	Kyverno→latest; Flux v1beta2→v1 migration + Flux→v2.7+
W5	1.36	1.20 (1.33-1.36)	v1.6.x	1.12	cert-manager→1.21; CNPG→1.30; kube-vip→latest; snapshot-controller→latest v8.x

Ordering rule inside each window (critical): CNI/storage compatibility first, then the k3s control-plane hop, then verify. Concretely per window:

1. Land the component upgrades that must precede the new k8s minor (Cilium minor, Gateway API CRDs, Longhorn minor, and any component whose current version won't run on the new k8s). Let Flux reconcile; verify healthy on the still-current k8s.
2. Run the k3s minor hop (§5) node-by-node.
3. Verify the components again on the new k8s; then land the "opportunistic/loose" bumps.

Rationale for "Cilium before the apiserver hop": if the apiserver moves to a minor the running Cilium doesn't support, CNI/agent may fail to start on that node and the node never goes Ready. Cilium's own guidance is to run its pre-flight check DaemonSet before each Cilium minor bump and to upgrade Cilium before or alongside the k8s minor within its supported window.

---

5. Upgrade method — evaluation & recommendation (D1)

5.1 Options

(a) Rancher system-upgrade-controller (SUC) with Plans. Canonical k3s upgrade tool. A Plan targets nodes by label, concurrency 1, HA-aware, drains → replaces the k3s binary → uncordons.

• Pros: official, declarative, handles server-vs-agent ordering; could live in GitOps.
• Cons for THIS cluster: SUC's default drain evicts pods, which on this slow-HDD Longhorn triggers replica rebuild storms that starve etcd WAL fsync and flap the VIP — the exact failure ssd-etcd-isolation.md documents. You'd have to disable drain (drain: {enabled:false}) and still manually apply the Longhorn freeze, at which point SUC's main value (drain automation) is gone. Worse, SUC automates away the human gate between nodes — the user's hard requirement is an explicit backup + PASS/STOP verification between every node, which SUC does not pause for.

(b) Scripted in-place per-node binary upgrade (Ansible-driven). Re-run the k3s install script with the new INSTALL_K3S_VERSION on one node at a time; k3s replaces its binary and restarts, etcd re-joins at the new version. This is exactly the ssd-etcd-execution-runbook.md bounce shape.

• Pros: reuses the proven freeze + leader-last + per-node-verify pattern; no drain (no rebuild storm — replicas reattach); the human gate sits naturally between nodes; minimal moving parts. Note the creates: guard: the existing install task self-skips on a running node, so the upgrade path must invoke the install script explicitly (a dedicated play/tag, or an ad-hoc curl … | INSTALL_K3S_VERSION=<tag> sh -s - server), not rely on bumping the var alone.
• Cons: does not touch the OS (handled separately, §6/D2); slightly more manual than SUC.

(c) Immutable re-provision via tofu. Bump image_source + k3s_version, tofu replace one VM at a time, node rejoins etcd fresh and Longhorn rebuilds that node's replicas.

• Pros: also solves the OS upgrade in the same motion; clean, reproducible cattle nodes.
• Cons: destroys the node's etcd member + all its Longhorn replicas → full rebuild (not reattach) on the slow HDD per node (throttled but heavy, ~hours for 18 volumes' replicas across three rebuilds); the etcd SSD virtio2 is wiped on replace (fine for a 3-node HA etcd — the new member syncs from the other two). Heavier and slower than (b) for a pure k8s hop.

5.2 Recommendation

• k3s minor hops (W1-W5): choose (b), scripted in-place per-node. It matches the cluster's proven safe-bounce, avoids rebuild storms, and preserves the mandatory per-node human gate. Pin the new tag in roles/k3s_server/defaults/main.yml (k3s_version) and drive the actual upgrade with an explicit install-script invocation (bypassing the creates: guard), serial: 1, leader-last.
• Node OS (Debian 12→13): choose (c), immutable re-provision, as a separate campaign (§8). A clean trixie image beats an in-place apt dist-upgrade for reproducibility, and the nodes are already cattle (cloud-init + node_baseline rebuild them). Accept the per-node Longhorn rebuild there (that campaign wants rebuild, throttled at limit=1, since replicas are genuinely gone).
• SUC remains a reasonable future option once the cluster is on fast disks (the prod/mini-PC Phase-7 cluster) where drain-driven rebuild is cheap; note it for later.

---

6. Data-validation gates at EVERY step (hard requirement)

Every node bounce (there are 3 per k3s window; 3 per OS window) is wrapped by a pre-step backup gate and a post-step verification gate. Every window additionally takes a full cluster backup before the first node.

6.1 PRE-STEP backup gate (before the first node of a window, and re-confirm before each node)

# 1) etcd on-demand snapshot (k3s native) — the primary rollback artifact. Run on the leader.
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible <leader> -b -m shell \
  -a "k3s etcd-snapshot save --name pre-1.3X-$(date +%Y%m%d%H%M)"'
# verify it exists + is copied off-node (snapshots land in /var/lib/rancher/k3s/server/db/snapshots;
# also archived to S3 if configured). Confirm size > 0 and note the name.

# 2) Velero backup (cluster resources + PVC data via kopia FSB → MinIO 192.168.1.9:9000)
./infra/devcontainer/dev.sh run bash -lc 'velero backup create pre-1.3X-$(date +%Y%m%d%H%M) --wait'
./infra/devcontainer/dev.sh run bash -lc 'velero backup describe <name> --details'   # Phase=Completed, 0 errors

# 3) CNPG logical backup + verify, for each of keycloak-db / nextcloud-db / semaphore-db
#    (they already run continuous WAL + daily base backups to MinIO via barmanObjectStore).
#    Trigger an on-demand base backup and confirm it completes:
./infra/devcontainer/dev.sh run kubectl -n <ns> create -f - <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata: { name: pre-upg-<ns>, namespace: <ns> }
spec: { method: barmanObjectStore, cluster: { name: <cluster> } }
EOF
./infra/devcontainer/dev.sh run kubectl -n <ns> get backup pre-upg-<ns> -o jsonpath='{.status.phase}'  # completed
# capture an integrity baseline to compare after:
./infra/devcontainer/dev.sh run kubectl -n <ns> exec <cluster>-1 -- psql -tAc \
  "select (select count(*) from pg_database), (select sum(('x'||substr(md5(t.oid::text),1,8))::bit(32)::bigint) from pg_class t)"

PASS to proceed: etcd snapshot present (>0 bytes, name recorded) AND Velero backup Completed with 0 errors AND all three CNPG on-demand backups completed with baseline captured. STOP (do not start the hop): any backup fails, errors, or cannot be verified restorable.

6.2 The per-node bounce itself (reuse ssd-etcd-execution-runbook.md)

Per node NODE=k3s-dev-N, VMID, leader last (re-verify leader before each node):

# leader check
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell \
  -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_is_leader"'

# 1) FREEZE Longhorn rebuilds (reattach, not rebuild) — the proven safe-bounce
./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io \
  concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"0"}'

# 2) Upgrade k3s ON THIS NODE ONLY (explicit install, bypassing the creates: guard):
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible $NODE -b -m shell \
  -a "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.3X.z+k3s1 sh -s - server"'
#    (k3s replaces its binary and restarts; embedded etcd re-joins at the new version)

# 3) VERIFY this node fully before touching the next (see §6.3), THEN
# 4) THAW rebuilds only after 18/18 healthy:
./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io \
  concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"1"}'

6.3 POST-STEP verification gate (after each node; full set after the last node of a window)

./infra/devcontainer/dev.sh run kubectl get nodes -o wide            # NODE Ready, VERSION = new minor
./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd       # ok
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell \
  -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader"'   # all = 1
./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=R:.status.robustness --no-headers | grep -c healthy       # 18
./infra/devcontainer/dev.sh run kubectl -n kube-system rollout status ds/cilium --timeout=300s
./infra/devcontainer/dev.sh run bash -lc 'flux get kustomizations -A && flux get helmreleases -A'  # all Ready
# etcd snapshot restorability spot-check (list; a scratch restore is the W-level drill, §7)
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible <leader> -b -m shell \
  -a "k3s etcd-snapshot ls"'
# CNPG integrity vs the §6.1 baseline
./infra/devcontainer/dev.sh run kubectl -n <ns> exec <cluster>-1 -- pg_isready         # accepting connections
# re-run the row-count/checksum query and diff against baseline (must match)

App smoke tests (the QA-sweep URL set) — after each window completes, expect HTTP 200/302 (or the app's healthy code) from: auth · crafty · gitea · grafana · headlamp · jackett · keycloak (/realms/homelab/.well-known/openid-configuration) · nextcloud · pihole · plex · qbittorrent · radarr · seerr · semaphore · sonarr · unifi — all under *.home.keithjackson.dev. Plus one KubeVirt VM check (a VMI Running / console reachable).

PASS to advance to the next node/window: all nodes Ready at the expected version · etcd /readyz/etcd=ok and 3/3 have a leader · 18/18 Longhorn healthy (reattached, no rebuild storm) · Cilium DaemonSet rolled out · all Flux Kustomizations + HelmReleases Ready · CNPG pg_isready + row-count/checksum unchanged vs baseline · QA-sweep all green. STOP (halt, leave freeze=0, do not advance): etcd /readyz not ok · a member without a leader · VIP 192.168.2.50:6443 flapping · a node not rejoining/not Ready · any Longhorn volume Faulting · k3s crash-looping · a CNPG DB failing pg_isready or checksum drift · a Flux HR stuck not-Ready · any QA-sweep URL down. Recovery per §7.

---

7. Rollback per step & point-of-no-return

7.1 Per-node (before etcd majority is on the new minor)

The safest rollback for a single bad node is to re-pin the old k3s version on that node and re-run the install script for the old tag, then verify it rejoins at the old minor; or reset the member and rejoin from the other two (2/3 quorum holds), per ssd-etcd-isolation.md §4.5. Because you go leader-last and one node at a time, at any moment ≤1 node is on the new minor until the 2nd node upgrades.

7.2 Point-of-no-return per hop

When the second server reaches the new minor, etcd majority (2/3) is on the new datastore version (etcd 3.5→3.6 across the path is one-way). Past that point, "roll back the node" is no longer enough — a true rollback is an etcd snapshot restore of the whole cluster to the pre-hop snapshot:

# cluster-wide restore from the pre-hop snapshot (DISRUPTIVE — full control-plane stop):
# on ONE server, with k3s stopped on all three:
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -b -m shell -a "systemctl stop k3s"'
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible <node1> -b -m shell \
  -a "k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/<pre-1.3X-snap>"'
# then wipe db on the other two and rejoin them (standard k3s cluster-reset-restore flow),
# and PIN k3s_version back to the old tag before they re-join.

After an etcd restore, Longhorn volume data is recovered by Velero (kopia FSB) / CNPG PITR for anything written between snapshot and restore — this is why the §6.1 triad (etcd + Velero + CNPG) is mandatory, not just etcd.

7.3 Window-level DR drill (do once, in W1, to prove the path)

Before the first real hop, prove restorability end-to-end: velero restore create --from-backup <pre-snap> --namespace-mappings <ns>:<ns>-scratch into a scratch namespace and confirm data lands (this is the existing Phase-4 R-STATE-03 gate). If the drill fails, do not begin the upgrade.

7.4 OS-campaign rollback

Re-provision is inherently reversible per node: if a re-provisioned trixie node misbehaves, tofu re-apply with image_source reverted to bookworm re-creates it on Debian 12; the node rejoins etcd and rebuilds replicas as with any replacement. Point-of-no-return is the same etcd majority rule if OS re-provision is (against advice) combined with a k3s bump — which is why D3 says keep them separate.

---

8. Node OS upgrade — Debian 12 → 13 (trixie)

• Debian 13 "trixie" released 2025-08-09, ships Linux 6.12 LTS (up from bookworm's 6.1). Support to 2028-08 (LTS 2030-08). Sources: https://www.debian.org/News/2025/20250809 · https://www.debian.org/releases/trixie/.
• Kernel implications (positive): 6.12 is newer than Cilium's/Longhorn's minimums and improves eBPF/BPF feature coverage for Cilium and has current iscsi_tcp/nfs/dm_crypt/overlay/ br_netfilter modules Longhorn needs (all already pinned in roles/node_baseline/defaults/main.yml → node_baseline_kernel_modules). No known regression for this stack; open-iscsi/nfs-common packages exist on trixie.
• Method (D2): immutable re-provision. Bump infra/tofu/variables.tf:
  • image_source → https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
  • image_file_name → debian-13-generic-amd64.qcow2 Then tofu apply -replace=proxmox_virtual_environment_vm.node["k3s-dev-N"] one node at a time, let node_baseline + k3s_server (secondary/join) rebuild it, and let it rejoin etcd. Because a replaced node loses its Longhorn replicas, run this campaign with rebuild throttled at 1 (not frozen at 0 — you want the rebuild here), wait for 18/18 healthy before the next node, and keep the same per-node backup + verify gates (§6).
• Coordinate with k3s (D3): run the OS campaign after the cluster is stable on a chosen k3s minor (either finish the whole 1.36 ladder first, or interleave one full campaign between k3s windows) — never re-provision to trixie and bump k3s in the same node bounce; that entangles two failure domains and defeats per-step rollback.
• In-place apt dist-upgrade is the fallback if re-provision is undesirable, but it is discarded here: it mutates cattle nodes outside the tofu/cloud-init source of truth, risks half-migrated states across three nodes, and violates the repo's "changes go through IaC" principle.

---

9. Sequencing & HA safety

• Quorum: 3 servers, embedded etcd, quorum 2/3. Exactly one node down at a time keeps quorum. Never bounce two simultaneously.
• Leader last: re-verify the etcd leader before each node (etcd_server_is_leader on :2381) and upgrade followers first, leader last, so leadership only moves once and late.
• VIP stability: kube-vip floats 192.168.1.49 to the current leader; with one node down the VIP fails over to a surviving leader and the API stays reachable through the pfSense NAT (192.168.2.50:6443). Watch for VIP flapping as a STOP signal.
• Rebuild-freeze (reattach, not rebuild): for k3s in-place hops set concurrent-replica-rebuild-per-node-limit=0 around each bounce so a briefly-down node's replicas reattach instead of triggering a rebuild storm that starves etcd WAL fsync (the whole reason etcd is on a dedicated SSD). Thaw to 1 only after 18/18 healthy. (For the OS re-provision campaign use 1, not 0, since replicas are genuinely gone and must rebuild.)
• CNPG note: all three DBs are single-instance (instances: 1) — each incurs a brief outage when its pod reschedules during a node bounce. Order/announce accordingly; the app smoke-tests (§6.3) will blip while a DB pod moves. (Consider instances: 3 for the prod cluster.)
• KubeVirt/CDI: already 1.36-ready; VMs without evictionStrategy: LiveMigrate simply Evict+restart on a node bounce (we do not drain in method (b), so VMs are only affected on the node actually bouncing).

Estimated maintenance windows: per k3s minor ≈ 60-120 min (3 nodes × ~15-30 min bounce + verify + freeze/thaw + component reconcile), so 5 windows ≈ 5-10 h total spread across days, plus the W-level backups. OS re-provision campaign ≈ 2-4 h per node (dominated by Longhorn rebuild of that node's share of 18 volumes on HDD) → ~1-1.5 days, one node per session.

---

10. Dev-first, then prod (Phase-7 carry-over)

This dev run is the rehearsal for the eventual prod / mini-PC cluster. What carries over:

• The method (scripted in-place per-node for k3s; immutable re-provision for OS), the leader-last + freeze + per-node-gate discipline, and the backup triad gate are cluster-agnostic and become the prod runbook verbatim.
• The component lockstep table (§4.2) is the reusable dependency map; re-confirm exact patch numbers at prod time (they will have moved).
• Differences to expect on prod: fast disks (NVMe/SSD) make Longhorn rebuild cheap → SUC becomes viable there (drain no longer dangerous), and multi-instance CNPG (instances: 3) removes the single-DB outage blip. Prod may also run dedicated workers, changing SUC Plan server-vs-agent ordering.
• The Flux v1beta2→v1 migration (§3.2) is a one-time repo change shared by both clusters — do it once, both benefit.

---

Appendix A — pre-flight checklist per window (copy/paste)

[ ] Read k3s release notes for the target minor + k8s CHANGELOG "Urgent Upgrade Notes"
[ ] pluto/kubent scan (live + manifests) against target minor → clean (or Flux v1beta2 handled)
[ ] Component pre-reqs landed & Ready on CURRENT k8s (Cilium minor, Gateway CRDs, Longhorn minor, …)
[ ] Cilium pre-flight check DaemonSet passed for the Cilium minor bump
[ ] PRE-STEP backup gate PASS: etcd snapshot + Velero Completed + 3× CNPG completed + baselines captured
[ ] (W1 only) Velero restore DR drill into scratch ns PASS
[ ] Leader identified; node order = followers → leader
[ ] For each node: freeze(0) → upgrade → §6.3 verify PASS → thaw(1) → 18/18 healthy → next
[ ] After last node: full §6.3 + QA-sweep all green + KubeVirt VM check
[ ] Land opportunistic/loose component bumps; Flux all Ready
[ ] Pin the new k3s_version in roles/k3s_server/defaults/main.yml; PR → main (never commit to main)
[ ] Record outcome in DECISIONS-LOG.md

Appendix B — sources

• k3s releases & notes: https://github.com/k3s-io/k3s/releases · https://docs.k3s.io/release-notes/v1.36.X · https://docs.k3s.io/release-notes/v1.33.X · channels https://update.k3s.io/v1-release/channels
• k8s version-skew: https://kubernetes.io/releases/version-skew-policy/ · deprecation guide https://kubernetes.io/docs/reference/using-api/deprecation-guide/ · v1.36 sneak peek https://www.kubernetes.io/blog/2026/03/30/kubernetes-v1-36-sneak-peek/
• Cilium compat: https://docs.cilium.io/en/stable/network/kubernetes/compatibility/ · https://compatibility.fyi/projects/cilium/ · Gateway API https://docs.cilium.io/en/v1.20/network/servicemesh/gateway-api/gateway-api/
• Longhorn: https://endoflife.date/longhorn · https://longhorn.io/docs/ · https://www.suse.com/suse-longhorn/support-matrix/ · https://github.com/longhorn/longhorn/wiki/Version-Update-Policy
• KubeVirt/CDI: https://github.com/kubevirt/sig-release/blob/main/releases/k8s-support-matrix.md · https://github.com/kubevirt/kubevirt/blob/main/docs/kubernetes-compatibility.md
• external-snapshotter: https://github.com/kubernetes-csi/external-snapshotter/releases
• cert-manager: https://cert-manager.io/docs/releases/
• CNPG: https://cloudnative-pg.io/docs/devel/supported_releases/
• Flux: https://fluxcd.io/flux/releases/ · v2.7 upgrade https://github.com/fluxcd/flux2/discussions/5572 · image API https://fluxcd.io/flux/components/image/imagerepositories/
• kube-prometheus-stack: https://github.com/prometheus-community/helm-charts
• Velero: https://velero.io/docs/main/ · https://compatibility.fyi/projects/velero/
• Kyverno: https://kyverno.io/docs/installation/releases/
• kube-vip: https://github.com/kube-vip/kube-vip/releases · https://kube-vip.io/docs/upgrade/
• Debian 13: https://www.debian.org/News/2025/20250809 · https://www.debian.org/releases/trixie/
• Tools: pluto https://github.com/FairwindsOps/pluto · kubent https://github.com/doitintl/kube-no-trouble