home-network-k3s-impl/docs/plans/cluster-upgrade-plan.md

# Cluster upgrade plan — k3s 1.31.5 → 1.36.x + node OS (dev cluster)

**Scope:** the `home-network-k3s-impl` **dev** cluster (`k3s-dev-1/2/3` @ 192.168.1.50-52), 3× HA
embedded-etcd servers behind a kube-vip VIP, Debian 12 bookworm. Upgrade Kubernetes/k3s from
**v1.31.5+k3s1 → v1.36.x+k3s1** (five minors) and the node OS (Debian 12 → 13), with an
etcd-snapshot / Velero / CNPG backup gate and explicit PASS/STOP criteria at **every** step, and a
defined rollback for each hop.

> Status: **PLAN ONLY.** Nothing here has been executed. All version pins marked "confirm at
> execution" must be re-checked against the upstream release page on the day of the hop — this doc
> was written 2026-08-29 and homelab component versions move weekly (Renovate).

> Companion runbooks that this plan reuses wholesale:
> - `docs/plans/ssd-etcd-execution-runbook.md` — the **proven safe-bounce** procedure (Longhorn
>   rebuild-freeze + one-node-at-a-time + leader-last + per-node verify gates). Every k3s hop below
>   is a parameterised repeat of that runbook.
> - `docs/plans/ssd-etcd-isolation.md` — why etcd is fragile on this cluster (slow HDD; etcd now on
>   a dedicated SSD `virtio2`) and how a single member can be reset+rejoined from the other two.

---

## 0. TL;DR / operator decisions needed

| # | Decision | Recommendation | Needs operator sign-off |
|---|---|---|---|
| D1 | k3s upgrade **method** | Scripted **in-place per-node** binary upgrade driven by Ansible, one node at a time, leader-last, with the Longhorn rebuild-freeze. NOT system-upgrade-controller (its auto-drain fights this cluster's slow-HDD Longhorn and removes the human gate between nodes). | **Yes** — SUC vs scripted-in-place |
| D2 | Node **OS** upgrade | **Immutable re-provision** via tofu (bump the cloud image to Debian 13 trixie, replace one node at a time, rejoin etcd), run as a **separate campaign** from the k3s hops. Not in-place `apt` dist-upgrade. | **Yes** — Debian 13 now, or stay on 12 |
| D3 | Do OS + k3s on the **same** node bounce? | **No.** Never combine. Finish the k3s ladder to a stable target, then run the OS campaign (or vice-versa), one variable per bounce. | Confirm |
| D4 | Sequencing granularity | One k3s **minor** per maintenance window (5 windows). Cilium + Longhorn + Gateway-API CRDs advance in **lockstep** inside the same window (they are the pacing blockers). | Confirm |
| D5 | Flux `image.toolkit.fluxcd.io/v1beta2` → `/v1` migration | **Must** be done before Flux is upgraded to v2.7+ (which is required for k8s 1.36). Do it early, decoupled. | Confirm timing |

**Headline blockers (do these or the hop fails):**
1. **Cilium 1.16.5 is already at/near its ceiling** — Cilium 1.16 tops out around k8s 1.30-1.31.
   Cilium **must** step 1.16→1.17→1.18→1.19→1.20 in lockstep with the k8s minors. It is the single
   most urgent item.
2. **Longhorn 1.7.3** must step 1.7→1.8→1.9→1.10→1.11→1.12 (one minor at a time) to reach a
   1.36-validated release (1.12.x).
3. **Flux v2.4.0 → v2.7+** is required for k8s 1.36, and v2.7 **removed** the image-automation
   `v1beta2` API the repo still uses.
4. **KubeVirt v1.9.0 / CDI v1.66.0 are already 1.36-ready** — a pleasant surprise; no VM-stack
   upgrade needed, just keep them where they are.

---

## 1. Confirmed current-state inventory

### 1.1 Nodes (live, verified 2026-08-29 via `dev.sh run kubectl get nodes -o wide`)

| Node | IP | Roles | k8s | OS | Kernel | Runtime |
|---|---|---|---|---|---|---|
| k3s-dev-1 | 192.168.1.50 | control-plane,etcd,master | v1.31.5+k3s1 | Debian 12 bookworm | 6.1.0-52-cloud | containerd 1.7.23-k3s2 |
| k3s-dev-2 | 192.168.1.51 | control-plane,etcd,master | v1.31.5+k3s1 | Debian 12 bookworm | 6.1.0-52-cloud | containerd 1.7.23-k3s2 |
| k3s-dev-3 | 192.168.1.52 | control-plane,etcd,master | v1.31.5+k3s1 | Debian 12 bookworm | 6.1.0-52-cloud | containerd 1.7.23-k3s2 |

- 3 servers, embedded etcd, **all** control-plane (no dedicated workers). Quorum = 2/3.
- kube-vip control-plane VIP **192.168.1.49** (fronted through pfSense NAT at 192.168.2.50:6443).
- etcd datastore is on the **dedicated SSD `virtio2` (`/dev/vdc`, LABEL=k3s-etcd)** mounted at
  `/var/lib/rancher/k3s/server/db` (migrated this session). Longhorn data disk = `virtio1`
  (`/dev/vdb`, `/var/lib/longhorn`). OS = `virtio0` (`/dev/vda`).
- **18** Longhorn volumes, all `healthy` (the count is the post-step invariant).

### 1.2 Where versions are pinned in this repo

| Thing | Pin location | Current value |
|---|---|---|
| **k3s version** | `infra/ansible/roles/k3s_server/defaults/main.yml` → `k3s_version`; applied via `INSTALL_K3S_VERSION` in `roles/k3s_server/tasks/main.yml` | `v1.31.5+k3s1` |
| k3s install guard | `roles/k3s_server/tasks/main.yml` — install task has `creates: /etc/rancher/k3s/k3s.yaml` | **idempotent → will NOT upgrade a running node on a version bump** (see §4) |
| kubectl/helm/flux (toolbox) | `infra/devcontainer/versions.env`, `ci/versions.env` | kubectl 1.31.4, flux 2.4.0 |
| **Cilium (ongoing)** | `platform/cilium/helmrelease.yaml` → `version: "1.16.5"` | 1.16.5 |
| Cilium (bootstrap) + **Gateway API CRDs** | `infra/ansible/roles/cilium/defaults/main.yml` → `gateway_api_version: "v1.2.1"`, `gateway_api_crd_urls` | v1.2.1 |
| **Longhorn** | `platform/longhorn/helmrelease.yaml` → `version: ">=1.7.0 <1.8.0"` (live 1.7.3) | 1.7.3 |
| **KubeVirt + CDI** | `platform/kubevirt/kustomization.yaml` (remote release manifests) | KubeVirt v1.9.0, CDI v1.66.0 |
| **kube-vip** | `infra/ansible/roles/kube_vip/defaults/main.yml` → `kube_vip_version` | v0.8.9 |
| **external-snapshotter** | `platform/external-snapshotter/kustomization.yaml` (`ref=v8.2.0`) | v8.2.0 |
| **cert-manager** | `platform/cert-manager/helmrelease.yaml` → `">=1.16.0 <1.17.0"` (live 1.16.5) | 1.16.5 |
| **CNPG** | `platform/cnpg/helmrelease.yaml` → `">=0.22.0 <1.0.0"` (live chart 0.29.0) | operator ≈1.26/1.27 era |
| **kube-prometheus-stack** | `platform/monitoring/kube-prometheus-stack.yaml` → `">=65.0.0 <66.0.0"` (live 65.8.1) | 65.8.1 |
| **Loki** | `platform/monitoring/loki.yaml` → `">=6.0.0 <7.0.0"` (live 6.55.0) | 6.55.0 |
| **Alloy** | `platform/monitoring/alloy.yaml` → `">=0.10.0 <1.0.0"` (live 0.12.6) | 0.12.6 |
| **Velero** | `platform/velero/helmrelease.yaml` → `">=8.0.0 <9.0.0"` (live 8.7.2, Velero v1.15.x) | 8.7.2 |
| **Kyverno** | `platform/kyverno/admission/helmrelease.yaml` → `version: "3.3.x"` (live 3.3.9) | 3.3.9 |
| **oauth2-proxy** | `platform/oauth2-proxy/helmrelease.yaml` → `">=7.0.0 <8.0.0"` (live 7.18.0) | 7.18.0 |
| **external-dns** | `platform/externaldns/helmrelease.yaml` → `">=1.15.0 <1.16.0"` (live 1.15.2) | 1.15.2 |
| **headlamp** | `platform/headlamp/helmrelease.yaml` → `">=0.41.0 <0.42.0"` (live 0.41.0) | 0.41.0 |
| **Flux** | installed by `flux bootstrap` (toolbox v2.4.0); controllers live in `flux-system` | v2.4.0 |
| **tofu cloud image** | `infra/tofu/variables.tf` → `image_source`, `image_file_name` | `debian-12-generic-amd64.qcow2` (bookworm) |

### 1.3 Live platform component versions (verified 2026-08-29)

Flux 2.4.0 (source-controller v1.4.1, kustomize-controller v1.4.0, helm-controller v1.1.0,
image-automation v0.39.0, image-reflector v0.33.0, notification v1.4.0) · cert-manager v1.16.5 ·
CNPG chart 0.29.0 (PostgreSQL 16.4, 3 single-instance clusters: keycloak-db, nextcloud-db,
semaphore-db) · Cilium 1.16.5 · Kyverno 3.3.9 · Longhorn 1.7.3 · Alloy 0.12.6 ·
kube-prometheus-stack 65.8.1 · Loki 6.55.0 · oauth2-proxy 7.18.0 · Velero 8.7.2 (+plugin-for-aws
v1.11.1) · KubeVirt v1.9.0 · CDI v1.66.0 · snapshot-controller v8.2.0 · kube-vip v0.8.9 ·
Gateway API CRDs v1 (v1.2.1 line).

### 1.4 CRD / API surface the repo actually uses (grep of `platform/ apps/ catalog/ clusters/`)

`v1`, `apps/v1`, `networking.k8s.io/v1`, `rbac.authorization.k8s.io/v1`, `storage.k8s.io/v1`,
`gateway.networking.k8s.io/v1`, `cilium.io/v2`, `monitoring.coreos.com/v1`,
`helm.toolkit.fluxcd.io/v2`, `source.toolkit.fluxcd.io/v1`, `kustomize.toolkit.fluxcd.io/v1`,
`image.toolkit.fluxcd.io/v1beta2` ⚠, `postgresql.cnpg.io/v1`, `kubevirt.io/v1`,
`cdi.kubevirt.io/v1beta1`, `cert-manager.io/v1`, `kyverno.io/v1`, `k8s.keycloak.org/v2alpha1`,
`snapshot.storage.k8s.io/v1`, `client.authentication.k8s.io/v1beta1` (kubeconfig exec plugin).

**None of these are Kubernetes *core* APIs removed in 1.32-1.36.** The only removal that bites is a
**Flux** CRD (`image.toolkit.fluxcd.io/v1beta2`) — see §3.

---

## 2. k3s upgrade path (sequential, one minor at a time)

### 2.1 The path and why skipping is unsafe

```
1.31.5+k3s1 → 1.32.z → 1.33.z → 1.34.z → 1.35.z → 1.36.z    (+k3s1 each)
```

**Skipping minors is not supported.** Kubernetes' version-skew policy allows the kube-apiserver and
components to differ by at most **one minor** during an upgrade, and a downgrade of the etcd
datastore is not supported. Jumping >1 minor risks an apiserver that can't read objects written by
an intermediate storage version and control-plane components outside the skew window. On k3s (all
three nodes are servers/apiserver), the whole control plane moves together per hop, so the rule is:
**one minor per campaign, all three nodes brought to the same minor before the next.**
Source: <https://kubernetes.io/releases/version-skew-policy/>.

### 2.2 Target patch per minor (confirm at execution)

Pick the **latest patch of each minor** from <https://github.com/k3s-io/k3s/releases> (or the k3s
channel server `https://update.k3s.io/v1-release/channels`, where channel `v1.3X` resolves to the
latest patch of that minor). Known-latest as of writing:

| Hop | Target (min) | Notes / source |
|---|---|---|
| → 1.32 | latest `v1.32.z+k3s1` | confirm on releases page |
| → 1.33 | latest `v1.33.z+k3s1` | k3s 1.33 release notes: <https://docs.k3s.io/release-notes/v1.33.X> |
| → 1.34 | latest `v1.34.z+k3s1` | confirm |
| → 1.35 | latest `v1.35.z+k3s1` | confirm |
| → 1.36 | **`v1.36.3+k3s1`** (Aug 4 2026) or newer (`v1.36.4+k3s1` reported) | <https://docs.k3s.io/release-notes/v1.36.X> |

### 2.3 k3s-specific breaking changes across the path

- **containerd major bump 1.7 → 2.x.** k3s 1.36.3 ships **containerd v2.3.2-k3s2** (current node is
  1.7.23-k3s2). The containerd v1→v2 migration happens somewhere in the 1.32-1.34 window. It is
  transparent for standard workloads, but: (a) `config.toml` schema changed — this repo sets
  `device_ownership_from_security_context` via `--nonroot-devices` (k3s config.yaml), which k3s
  regenerates into the new containerd config on restart; verify it survives (KubeVirt/CDI Block-mode
  PVCs depend on it — see `roles/k3s_server/tasks/main.yml`). (b) Any node with custom
  `/etc/rancher/k3s/config.toml.tmpl` would need review — this cluster has none.
- **etcd 3.5 → 3.6.** k3s 1.36.3 ships **etcd v3.6.14-k3s1**. etcd datastore version bumps are
  **one-way** (no downgrade). This is the single strongest reason for the pre-hop etcd snapshot gate
  (§6) and defines the point-of-no-return (§7).
- **Traefik chart v40 breaking change** in k3s ≥1.36.2 (ingress-nginx provider rename). **N/A** —
  this cluster disables Traefik (`k3s_disable: "traefik,servicelb,local-storage"`); Cilium Gateway
  API owns ingress.
- **kube-proxy / flannel / network-policy already disabled** (`--disable-kube-proxy`,
  `--flannel-backend=none`, `--disable-network-policy`) so k8s 1.36's **IPVS-mode kube-proxy
  deprecation is N/A** — Cilium is the kube-proxy replacement.
- Always read the **Kubernetes urgent-upgrade notes** for each minor before the hop:
  <https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/> (`CHANGELOG-1.3X.md` →
  "Urgent Upgrade Notes").

---

## 3. Removed / deprecated API audit (1.32 → 1.36)

### 3.1 Kubernetes core removals/deprecations per minor

| k8s | Removed / deprecated | Affects this repo? |
|---|---|---|
| **1.32** | **`flowcontrol.apiserver.k8s.io/v1beta3`** (FlowSchema, PriorityLevelConfiguration) **removed** — use `flowcontrol.apiserver.k8s.io/v1` (GA since 1.29). Field rename `assuredConcurrencyShares`→`nominalConcurrencyShares`. | **No** — repo defines no FlowSchema/PriorityLevelConfiguration CRs. |
| **1.33** | `status.nodeInfo.kubeProxyVersion` **field removed** from Node. | **No** — nothing reads it. |
| **1.34** | **AppArmor** annotations path deprecations continue; `AppArmor` GA'd earlier. | No repo AppArmor annotations. |
| **1.35** | Minor graduations; no removal that touches this repo's API set. | No. |
| **1.36** | **`gitRepo` volume permanently disabled**; **Service `externalIPs` deprecated**; **kube-proxy IPVS** on a deprecation clock. | **No** — no gitRepo volumes, no `externalIPs`, kube-proxy already replaced by Cilium. |

Sources: <https://kubernetes.io/docs/reference/using-api/deprecation-guide/> ·
<https://www.kubernetes.io/blog/2026/03/30/kubernetes-v1-36-sneak-peek/> ·
per-minor `CHANGELOG-1.3X.md`.

**Conclusion: the repo's manifests use no Kubernetes core API removed in 1.32-1.36.** (Grep for
`flowcontrol.apiserver`, `gitRepo:`, `externalIPs:`, `policy/v1beta1`, `batch/v1beta1`,
`autoscaling/v2beta*`, `extensions/v1beta1`, `PodSecurityPolicy` returns **nothing**.)

### 3.2 The one real removal that bites — Flux `image.toolkit.fluxcd.io/v1beta2`

**Flux v2.7+ removed `image.toolkit.fluxcd.io/v1beta2`** (ImageRepository, ImagePolicy,
ImageUpdateAutomation → graduated to `image.toolkit.fluxcd.io/v1`). Flux v2.7+ is required for k8s
1.36 (§4.1). The repo still uses `v1beta2` in:
- `platform/image-automation/image-update-automation.yaml`
- `platform/image-automation/per-service/pihole.example.yaml`
(plus any per-service ImageRepository/ImagePolicy generated from that template.)

**Required migration (do BEFORE upgrading Flux to v2.7+):** follow the official
"Upgrade Procedure for Flux v2.7+" — migrate stored objects off v1beta2, then update every manifest
in Git from `image.toolkit.fluxcd.io/v1beta2` → `image.toolkit.fluxcd.io/v1`.
Source: <https://github.com/fluxcd/flux2/discussions/5572> ·
<https://fluxcd.io/flux/components/image/imagerepositories/>.

### 3.3 Detection tooling — run BEFORE each hop

Add a read-only pre-hop gate using **both** a live-cluster scan and a manifest scan:

```bash
# pluto — scans live cluster AND Helm releases for deprecated/removed apiVersions, target minor
./infra/devcontainer/dev.sh run bash -lc 'pluto detect-all-in-cluster --target-versions k8s=v1.3X.0 -owide'
# or scan the rendered manifests in git:
./infra/devcontainer/dev.sh run bash -lc 'pluto detect-files -d platform -d apps --target-versions k8s=v1.3X.0'

# kubent (kube-no-trouble) — quick live scan
./infra/devcontainer/dev.sh run bash -lc 'kubent --target-version 1.3X'
```

- `pluto` and `kubent` detect **built-in** deprecated APIs from their bundled version data; they do
  **not** know about third-party CRD removals (e.g. the Flux `v1beta2` above) unless given a custom
  deprecation list. So pair the tool pass with the manual §3.2 check and a `grep` of the repo for
  each component's deprecated CRD versions before its own upgrade.
- Neither tool is currently in the toolbox image; add to `infra/devcontainer/Containerfile` +
  `versions.env`, or run once via `go install`/release binary inside the devcontainer.
- Sources: <https://github.com/FairwindsOps/pluto> · <https://github.com/doitintl/kube-no-trouble>.

---

## 4. Component compatibility matrix & upgrade sequencing

### 4.1 Kubernetes-version support per component (target = k8s 1.36)

| Component | Current | k8s-1.36 target (confirm patch) | k8s support fact | Must move… | Source |
|---|---|---|---|---|---|
| **Cilium** | 1.16.5 | **1.20.x** | 1.16:≤~1.30 · 1.17:1.29-1.32 · 1.18:1.30-1.33 · 1.19:1.32-1.35 · 1.20:1.33-1.36 | **in lockstep, every hop** (pacing blocker #1) | compatibility.fyi/cilium; docs.cilium.io/…/compatibility |
| **Gateway API CRDs** | v1.2.1 | **v1.6.x** (Cilium 1.20 targets v1.6.1 — verify) | too old for Cilium 1.20 | with each Cilium hop | docs.cilium.io/en/v1.20/…/gateway-api |
| **Longhorn** | 1.7.3 | **1.12.x** (1.12.1) | 1.7:≤~1.31 · 1.11.1:1.32-1.35 · 1.12:adds 1.36; min k8s stays v1.21 | one minor at a time (pacing blocker #2) | endoflife.date/longhorn; longhorn.io best-practices; SUSE support matrix |
| **KubeVirt** | **v1.9.0** | **v1.9.x (already there)** | v1.9 validates 1.34/1.35/1.36 (N-2) | **no upgrade needed** | github.com/kubevirt/sig-release k8s-support-matrix |
| **CDI** | **v1.66.0** | keep paired w/ KubeVirt v1.9 | same window as KubeVirt | no upgrade needed (keep pair) | kubevirt/containerized-data-importer releases |
| **external-snapshotter / snapshot-controller** | v8.2.0 | **latest v8.x (≥8.6)** | v8 tracks current k8s; snapshot CRDs fwd/back-compatible | loose — opportunistic | github.com/kubernetes-csi/external-snapshotter releases |
| **cert-manager** | 1.16.5 | **1.21.x** | 1.21 supports 1.33-1.36 | before/with the later hops | cert-manager.io/docs/releases |
| **CNPG** | chart 0.29.0 (op ≈1.26/1.27) | **1.30.x** | 1.30 supports 1.34/1.35/1.36; 1.29 supports 1.33-1.35 | before the 1.36 hop | cloudnative-pg.io/docs/devel/supported_releases |
| **Flux** | v2.4.0 | **v2.7+ (latest, ~v2.8.x)** | Flux N-2; v2.7+ needed for 1.36 | after §3.2 v1beta2→v1 migration | fluxcd.io/flux/releases |
| **kube-prometheus-stack** | 65.8.1 | **~88.x+** (supports kubectl/k8s 1.36) | chart 88.x tested on 1.36 | before/with later hops; **apply updated CRDs separately** | github.com/prometheus-community/helm-charts |
| **Loki (chart)** | 6.55.0 | **latest 6.x/newer** | uses stable core APIs — loose | opportunistic | grafana/helm-charts |
| **Alloy (chart)** | 0.12.6 | **latest** | loose (DaemonSet, stable APIs) | opportunistic | grafana/alloy |
| **Velero** | chart 8.7.2 (v1.15) | **latest (v1.17+/chart ~10.x)** | v1.16 tested→1.33, v1.17 tested→1.34; "1.18-latest" expected | before the 1.36 hop | velero.io support-matrix; compatibility.fyi/velero |
| **Kyverno** | 3.3.9 | **latest 3.x (N-2 for 1.36)** | Kyverno follows k8s N-2; 3.3.x too old for 1.36 | before/with later hops | kyverno.io/docs/installation/releases |
| **oauth2-proxy** | 7.18.0 | **latest 7.x** | loose | opportunistic | oauth2-proxy chart |
| **external-dns** | 1.15.2 | **latest 1.x** | loose | opportunistic | kubernetes-sigs/external-dns |
| **kube-vip** | v0.8.9 | **latest (k8s.io 0.36.x-based)** | leader-election uses coordination.k8s.io/v1 (stable); latest bumps client-go to 0.36 | before/with the 1.36 hop | github.com/kube-vip/kube-vip releases |
| **headlamp** | 0.41.0 | latest | loose UI | opportunistic | headlamp chart |

### 4.2 The pacing blockers, and the per-hop lockstep

**Cilium and Longhorn are the two items that force the pace** — each supports only ~3-4 k8s minors
and each upgrades **one minor at a time (no skipping, pre-flight required)**. KubeVirt/CDI are
already at target, so the VM stack is *not* a blocker this cycle. The mapping:

| Window | k8s target | Cilium (HelmRelease) | Gateway API CRDs | Longhorn (HelmRelease) | Other components landed in this window |
|---|---|---|---|---|---|
| **W1** | 1.32 | **1.17** (covers 1.29-1.32) | bump toward Cilium 1.17's required version | **1.8** | — |
| **W2** | 1.33 | **1.18** (1.30-1.33) | bump | **1.9** | cert-manager→1.18/1.19; kube-prometheus-stack CRDs refresh |
| **W3** | 1.34 | **1.19** (1.32-1.35) | bump | **1.10** | CNPG→1.28/1.29; Velero→latest |
| **W4** | 1.35 | 1.19 still covers 1.35 (optional →1.20 early) | as needed | **1.11** | Kyverno→latest; **Flux v1beta2→v1 migration + Flux→v2.7+** |
| **W5** | 1.36 | **1.20** (1.33-1.36) | **v1.6.x** | **1.12** | cert-manager→1.21; CNPG→1.30; kube-vip→latest; snapshot-controller→latest v8.x |

Ordering rule **inside** each window (critical): **CNI/storage compatibility first, then the k3s
control-plane hop, then verify.** Concretely per window:
1. Land the **component** upgrades that must *precede* the new k8s minor (Cilium minor, Gateway API
   CRDs, Longhorn minor, and any component whose *current* version won't run on the new k8s). Let
   Flux reconcile; verify healthy on the **still-current** k8s.
2. Run the **k3s minor hop** (§5) node-by-node.
3. Verify the components again on the **new** k8s; then land the "opportunistic/loose" bumps.

> Rationale for "Cilium before the apiserver hop": if the apiserver moves to a minor the running
> Cilium doesn't support, CNI/agent may fail to start on that node and the node never goes Ready.
> Cilium's own guidance is to run its **pre-flight check** DaemonSet before each Cilium minor bump
> and to upgrade Cilium **before or alongside** the k8s minor within its supported window.

---

## 5. Upgrade method — evaluation & recommendation (D1)

### 5.1 Options

**(a) Rancher system-upgrade-controller (SUC) with Plans.** Canonical k3s upgrade tool. A `Plan`
targets nodes by label, concurrency 1, HA-aware, drains → replaces the k3s binary → uncordons.
- **Pros:** official, declarative, handles server-vs-agent ordering; could live in GitOps.
- **Cons for THIS cluster:** SUC's default **drain evicts pods**, which on this slow-HDD Longhorn
  triggers replica **rebuild** storms that starve etcd WAL fsync and flap the VIP — the exact
  failure `ssd-etcd-isolation.md` documents. You'd have to disable drain (`drain: {enabled:false}`)
  and still manually apply the Longhorn freeze, at which point SUC's main value (drain automation) is
  gone. Worse, SUC **automates away the human gate between nodes** — the user's hard requirement is
  an explicit backup + PASS/STOP verification *between* every node, which SUC does not pause for.

**(b) Scripted in-place per-node binary upgrade (Ansible-driven).** Re-run the k3s install script
with the new `INSTALL_K3S_VERSION` on one node at a time; k3s replaces its binary and restarts,
etcd re-joins at the new version. This is exactly the `ssd-etcd-execution-runbook.md` bounce shape.
- **Pros:** reuses the **proven** freeze + leader-last + per-node-verify pattern; no drain (no
  rebuild storm — replicas *reattach*); the human gate sits naturally between nodes; minimal moving
  parts. **Note the `creates:` guard:** the existing install task self-skips on a running node, so
  the upgrade path must invoke the install script *explicitly* (a dedicated play/tag, or an ad-hoc
  `curl … | INSTALL_K3S_VERSION=<tag> sh -s - server`), not rely on bumping the var alone.
- **Cons:** does not touch the OS (handled separately, §6/D2); slightly more manual than SUC.

**(c) Immutable re-provision via tofu.** Bump `image_source` + `k3s_version`, `tofu replace` one VM
at a time, node rejoins etcd fresh and Longhorn rebuilds that node's replicas.
- **Pros:** **also solves the OS upgrade** in the same motion; clean, reproducible cattle nodes.
- **Cons:** destroys the node's etcd member + **all** its Longhorn replicas → full **rebuild** (not
  reattach) on the slow HDD per node (throttled but heavy, ~hours for 18 volumes' replicas across
  three rebuilds); the etcd SSD `virtio2` is wiped on replace (fine for a 3-node HA etcd — the new
  member syncs from the other two). Heavier and slower than (b) for a *pure k8s* hop.

### 5.2 Recommendation

- **k3s minor hops (W1-W5): choose (b), scripted in-place per-node.** It matches the cluster's
  proven safe-bounce, avoids rebuild storms, and preserves the mandatory per-node human gate. Pin
  the new tag in `roles/k3s_server/defaults/main.yml` (`k3s_version`) **and** drive the actual
  upgrade with an explicit install-script invocation (bypassing the `creates:` guard), `serial: 1`,
  leader-last.
- **Node OS (Debian 12→13): choose (c), immutable re-provision**, as a **separate** campaign (§8).
  A clean trixie image beats an in-place `apt` dist-upgrade for reproducibility, and the nodes are
  already cattle (cloud-init + `node_baseline` rebuild them). Accept the per-node Longhorn rebuild
  there (that campaign *wants* rebuild, throttled at limit=1, since replicas are genuinely gone).
- SUC remains a reasonable future option **once the cluster is on fast disks** (the prod/mini-PC
  Phase-7 cluster) where drain-driven rebuild is cheap; note it for later.

---

## 6. Data-validation gates at EVERY step (hard requirement)

Every **node bounce** (there are 3 per k3s window; 3 per OS window) is wrapped by a pre-step backup
gate and a post-step verification gate. Every **window** additionally takes a full cluster backup
before the first node.

### 6.1 PRE-STEP backup gate (before the first node of a window, and re-confirm before each node)

```bash
# 1) etcd on-demand snapshot (k3s native) — the primary rollback artifact. Run on the leader.
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible <leader> -b -m shell \
  -a "k3s etcd-snapshot save --name pre-1.3X-$(date +%Y%m%d%H%M)"'
# verify it exists + is copied off-node (snapshots land in /var/lib/rancher/k3s/server/db/snapshots;
# also archived to S3 if configured). Confirm size > 0 and note the name.

# 2) Velero backup (cluster resources + PVC data via kopia FSB → MinIO 192.168.1.9:9000)
./infra/devcontainer/dev.sh run bash -lc 'velero backup create pre-1.3X-$(date +%Y%m%d%H%M) --wait'
./infra/devcontainer/dev.sh run bash -lc 'velero backup describe <name> --details'   # Phase=Completed, 0 errors

# 3) CNPG logical backup + verify, for each of keycloak-db / nextcloud-db / semaphore-db
#    (they already run continuous WAL + daily base backups to MinIO via barmanObjectStore).
#    Trigger an on-demand base backup and confirm it completes:
./infra/devcontainer/dev.sh run kubectl -n <ns> create -f - <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata: { name: pre-upg-<ns>, namespace: <ns> }
spec: { method: barmanObjectStore, cluster: { name: <cluster> } }
EOF
./infra/devcontainer/dev.sh run kubectl -n <ns> get backup pre-upg-<ns> -o jsonpath='{.status.phase}'  # completed
# capture an integrity baseline to compare after:
./infra/devcontainer/dev.sh run kubectl -n <ns> exec <cluster>-1 -- psql -tAc \
  "select (select count(*) from pg_database), (select sum(('x'||substr(md5(t.oid::text),1,8))::bit(32)::bigint) from pg_class t)"
```

**PASS to proceed:** etcd snapshot present (>0 bytes, name recorded) **AND** Velero backup
`Completed` with 0 errors **AND** all three CNPG on-demand backups `completed` with baseline
captured.
**STOP (do not start the hop):** any backup fails, errors, or cannot be verified restorable.

### 6.2 The per-node bounce itself (reuse `ssd-etcd-execution-runbook.md`)

Per node `NODE=k3s-dev-N`, `VMID`, **leader last** (re-verify leader before each node):

```bash
# leader check
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell \
  -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_is_leader"'

# 1) FREEZE Longhorn rebuilds (reattach, not rebuild) — the proven safe-bounce
./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io \
  concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"0"}'

# 2) Upgrade k3s ON THIS NODE ONLY (explicit install, bypassing the creates: guard):
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible $NODE -b -m shell \
  -a "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.3X.z+k3s1 sh -s - server"'
#    (k3s replaces its binary and restarts; embedded etcd re-joins at the new version)

# 3) VERIFY this node fully before touching the next (see §6.3), THEN
# 4) THAW rebuilds only after 18/18 healthy:
./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io \
  concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"1"}'
```

### 6.3 POST-STEP verification gate (after each node; full set after the last node of a window)

```bash
./infra/devcontainer/dev.sh run kubectl get nodes -o wide            # NODE Ready, VERSION = new minor
./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd       # ok
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell \
  -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader"'   # all = 1
./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=R:.status.robustness --no-headers | grep -c healthy       # 18
./infra/devcontainer/dev.sh run kubectl -n kube-system rollout status ds/cilium --timeout=300s
./infra/devcontainer/dev.sh run bash -lc 'flux get kustomizations -A && flux get helmreleases -A'  # all Ready
# etcd snapshot restorability spot-check (list; a scratch restore is the W-level drill, §7)
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible <leader> -b -m shell \
  -a "k3s etcd-snapshot ls"'
# CNPG integrity vs the §6.1 baseline
./infra/devcontainer/dev.sh run kubectl -n <ns> exec <cluster>-1 -- pg_isready         # accepting connections
# re-run the row-count/checksum query and diff against baseline (must match)
```

**App smoke tests (the QA-sweep URL set)** — after each *window* completes, expect HTTP 200/302
(or the app's healthy code) from:
`auth` · `crafty` · `gitea` · `grafana` · `headlamp` · `jackett` · `keycloak`
(`/realms/homelab/.well-known/openid-configuration`) · `nextcloud` · `pihole` · `plex` ·
`qbittorrent` · `radarr` · `seerr` · `semaphore` · `sonarr` · `unifi` — all under
`*.home.keithjackson.dev`. Plus one KubeVirt VM check (a VMI Running / console reachable).

**PASS to advance to the next node/window:** all nodes Ready at the expected version · etcd
`/readyz/etcd`=ok and 3/3 have a leader · **18/18** Longhorn healthy (reattached, no rebuild storm)
· Cilium DaemonSet rolled out · all Flux Kustomizations + HelmReleases Ready · CNPG `pg_isready` +
row-count/checksum unchanged vs baseline · QA-sweep all green.
**STOP (halt, leave freeze=0, do not advance):** etcd `/readyz` not ok · a member without a leader ·
VIP `192.168.2.50:6443` flapping · a node not rejoining/not Ready · any Longhorn volume Faulting ·
k3s crash-looping · a CNPG DB failing `pg_isready` or checksum drift · a Flux HR stuck not-Ready ·
any QA-sweep URL down. Recovery per §7.

---

## 7. Rollback per step & point-of-no-return

### 7.1 Per-node (before etcd majority is on the new minor)

The safest rollback for a single bad node is to **re-pin the old k3s version on that node** and
re-run the install script for the old tag, then verify it rejoins at the old minor; or reset the
member and rejoin from the other two (2/3 quorum holds), per `ssd-etcd-isolation.md` §4.5. Because
you go **leader-last and one node at a time**, at any moment ≤1 node is on the new minor until the
2nd node upgrades.

### 7.2 Point-of-no-return per hop

**When the *second* server reaches the new minor, etcd majority (2/3) is on the new datastore
version** (etcd 3.5→3.6 across the path is one-way). Past that point, "roll back the node" is no
longer enough — a true rollback is an **etcd snapshot restore** of the whole cluster to the pre-hop
snapshot:

```bash
# cluster-wide restore from the pre-hop snapshot (DISRUPTIVE — full control-plane stop):
# on ONE server, with k3s stopped on all three:
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -b -m shell -a "systemctl stop k3s"'
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible <node1> -b -m shell \
  -a "k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/<pre-1.3X-snap>"'
# then wipe db on the other two and rejoin them (standard k3s cluster-reset-restore flow),
# and PIN k3s_version back to the old tag before they re-join.
```

After an etcd restore, **Longhorn volume data is recovered by Velero (kopia FSB) / CNPG PITR** for
anything written between snapshot and restore — this is why the §6.1 triad (etcd + Velero + CNPG)
is mandatory, not just etcd.

### 7.3 Window-level DR drill (do once, in W1, to prove the path)

Before the first real hop, prove restorability end-to-end: `velero restore create --from-backup
<pre-snap> --namespace-mappings <ns>:<ns>-scratch` into a scratch namespace and confirm data lands
(this is the existing Phase-4 R-STATE-03 gate). If the drill fails, **do not begin the upgrade.**

### 7.4 OS-campaign rollback

Re-provision is inherently reversible per node: if a re-provisioned trixie node misbehaves,
`tofu` re-apply with `image_source` reverted to bookworm re-creates it on Debian 12; the node
rejoins etcd and rebuilds replicas as with any replacement. Point-of-no-return is the same etcd
majority rule if OS re-provision is (against advice) combined with a k3s bump — which is why D3 says
keep them separate.

---

## 8. Node OS upgrade — Debian 12 → 13 (trixie)

- **Debian 13 "trixie" released 2025-08-09, ships Linux 6.12 LTS** (up from bookworm's 6.1).
  Support to 2028-08 (LTS 2030-08). Sources: <https://www.debian.org/News/2025/20250809> ·
  <https://www.debian.org/releases/trixie/>.
- **Kernel implications (positive):** 6.12 is newer than Cilium's/Longhorn's minimums and improves
  eBPF/BPF feature coverage for Cilium and has current `iscsi_tcp`/`nfs`/`dm_crypt`/`overlay`/
  `br_netfilter` modules Longhorn needs (all already pinned in
  `roles/node_baseline/defaults/main.yml` → `node_baseline_kernel_modules`). No known regression for
  this stack; `open-iscsi`/`nfs-common` packages exist on trixie.
- **Method (D2): immutable re-provision.** Bump `infra/tofu/variables.tf`:
  - `image_source` → `https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2`
  - `image_file_name` → `debian-13-generic-amd64.qcow2`
  Then `tofu apply -replace=proxmox_virtual_environment_vm.node["k3s-dev-N"]` **one node at a time**,
  let `node_baseline` + `k3s_server` (secondary/join) rebuild it, and let it rejoin etcd. Because a
  replaced node loses its Longhorn replicas, run this campaign with rebuild **throttled at 1** (not
  frozen at 0 — you *want* the rebuild here), wait for 18/18 healthy before the next node, and keep
  the same per-node backup + verify gates (§6).
- **Coordinate with k3s (D3):** run the OS campaign **after** the cluster is stable on a chosen k3s
  minor (either finish the whole 1.36 ladder first, or interleave one full campaign between k3s
  windows) — **never** re-provision to trixie *and* bump k3s in the same node bounce; that entangles
  two failure domains and defeats per-step rollback.
- **In-place `apt` dist-upgrade** is the fallback if re-provision is undesirable, but it is discarded
  here: it mutates cattle nodes outside the tofu/cloud-init source of truth, risks half-migrated
  states across three nodes, and violates the repo's "changes go through IaC" principle.

---

## 9. Sequencing & HA safety

- **Quorum:** 3 servers, embedded etcd, quorum 2/3. Exactly **one** node down at a time keeps
  quorum. Never bounce two simultaneously.
- **Leader last:** re-verify the etcd leader before each node (`etcd_server_is_leader` on :2381) and
  upgrade followers first, leader last, so leadership only moves once and late.
- **VIP stability:** kube-vip floats 192.168.1.49 to the current leader; with one node down the VIP
  fails over to a surviving leader and the API stays reachable through the pfSense NAT
  (192.168.2.50:6443). Watch for VIP flapping as a STOP signal.
- **Rebuild-freeze (reattach, not rebuild):** for k3s in-place hops set
  `concurrent-replica-rebuild-per-node-limit=0` around each bounce so a briefly-down node's replicas
  **reattach** instead of triggering a rebuild storm that starves etcd WAL fsync (the whole reason
  etcd is on a dedicated SSD). Thaw to `1` only after 18/18 healthy. (For the OS re-provision
  campaign use `1`, not `0`, since replicas are genuinely gone and must rebuild.)
- **CNPG note:** all three DBs are **single-instance** (`instances: 1`) — each incurs a brief
  outage when its pod reschedules during a node bounce. Order/announce accordingly; the app
  smoke-tests (§6.3) will blip while a DB pod moves. (Consider `instances: 3` for the prod cluster.)
- **KubeVirt/CDI:** already 1.36-ready; VMs without `evictionStrategy: LiveMigrate` simply
  Evict+restart on a node bounce (we do **not** drain in method (b), so VMs are only affected on
  the node actually bouncing).

**Estimated maintenance windows:** per k3s minor ≈ **60-120 min** (3 nodes × ~15-30 min bounce +
verify + freeze/thaw + component reconcile), so **5 windows ≈ 5-10 h** total spread across days,
plus the W-level backups. OS re-provision campaign ≈ **2-4 h per node** (dominated by Longhorn
rebuild of that node's share of 18 volumes on HDD) → ~1-1.5 days, one node per session.

---

## 10. Dev-first, then prod (Phase-7 carry-over)

This dev run is the rehearsal for the eventual **prod / mini-PC** cluster. What carries over:
- The **method** (scripted in-place per-node for k3s; immutable re-provision for OS), the
  **leader-last + freeze + per-node-gate** discipline, and the **backup triad** gate are
  cluster-agnostic and become the prod runbook verbatim.
- The **component lockstep table** (§4.2) is the reusable dependency map; re-confirm exact patch
  numbers at prod time (they will have moved).
- **Differences to expect on prod:** fast disks (NVMe/SSD) make Longhorn rebuild cheap → **SUC
  becomes viable** there (drain no longer dangerous), and multi-instance CNPG (`instances: 3`)
  removes the single-DB outage blip. Prod may also run dedicated workers, changing SUC Plan
  server-vs-agent ordering.
- The **Flux v1beta2→v1 migration** (§3.2) is a one-time repo change shared by both clusters — do it
  once, both benefit.

---

## Appendix A — pre-flight checklist per window (copy/paste)

```
[ ] Read k3s release notes for the target minor + k8s CHANGELOG "Urgent Upgrade Notes"
[ ] pluto/kubent scan (live + manifests) against target minor → clean (or Flux v1beta2 handled)
[ ] Component pre-reqs landed & Ready on CURRENT k8s (Cilium minor, Gateway CRDs, Longhorn minor, …)
[ ] Cilium pre-flight check DaemonSet passed for the Cilium minor bump
[ ] PRE-STEP backup gate PASS: etcd snapshot + Velero Completed + 3× CNPG completed + baselines captured
[ ] (W1 only) Velero restore DR drill into scratch ns PASS
[ ] Leader identified; node order = followers → leader
[ ] For each node: freeze(0) → upgrade → §6.3 verify PASS → thaw(1) → 18/18 healthy → next
[ ] After last node: full §6.3 + QA-sweep all green + KubeVirt VM check
[ ] Land opportunistic/loose component bumps; Flux all Ready
[ ] Pin the new k3s_version in roles/k3s_server/defaults/main.yml; PR → main (never commit to main)
[ ] Record outcome in DECISIONS-LOG.md
```

## Appendix B — sources

- k3s releases & notes: <https://github.com/k3s-io/k3s/releases> · <https://docs.k3s.io/release-notes/v1.36.X> · <https://docs.k3s.io/release-notes/v1.33.X> · channels <https://update.k3s.io/v1-release/channels>
- k8s version-skew: <https://kubernetes.io/releases/version-skew-policy/> · deprecation guide <https://kubernetes.io/docs/reference/using-api/deprecation-guide/> · v1.36 sneak peek <https://www.kubernetes.io/blog/2026/03/30/kubernetes-v1-36-sneak-peek/>
- Cilium compat: <https://docs.cilium.io/en/stable/network/kubernetes/compatibility/> · <https://compatibility.fyi/projects/cilium/> · Gateway API <https://docs.cilium.io/en/v1.20/network/servicemesh/gateway-api/gateway-api/>
- Longhorn: <https://endoflife.date/longhorn> · <https://longhorn.io/docs/> · <https://www.suse.com/suse-longhorn/support-matrix/> · <https://github.com/longhorn/longhorn/wiki/Version-Update-Policy>
- KubeVirt/CDI: <https://github.com/kubevirt/sig-release/blob/main/releases/k8s-support-matrix.md> · <https://github.com/kubevirt/kubevirt/blob/main/docs/kubernetes-compatibility.md>
- external-snapshotter: <https://github.com/kubernetes-csi/external-snapshotter/releases>
- cert-manager: <https://cert-manager.io/docs/releases/>
- CNPG: <https://cloudnative-pg.io/docs/devel/supported_releases/>
- Flux: <https://fluxcd.io/flux/releases/> · v2.7 upgrade <https://github.com/fluxcd/flux2/discussions/5572> · image API <https://fluxcd.io/flux/components/image/imagerepositories/>
- kube-prometheus-stack: <https://github.com/prometheus-community/helm-charts>
- Velero: <https://velero.io/docs/main/> · <https://compatibility.fyi/projects/velero/>
- Kyverno: <https://kyverno.io/docs/installation/releases/>
- kube-vip: <https://github.com/kube-vip/kube-vip/releases> · <https://kube-vip.io/docs/upgrade/>
- Debian 13: <https://www.debian.org/News/2025/20250809> · <https://www.debian.org/releases/trixie/>
- Tools: pluto <https://github.com/FairwindsOps/pluto> · kubent <https://github.com/doitintl/kube-no-trouble>
```