From 5efd98323987acf080c21c8ee50e29e5ae5aa935 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 29 Aug 2026 23:23:34 -0500 Subject: [PATCH] W1: Cilium 1.16.5->1.17.18, Longhorn 1.7->1.8 (pre-k8s-1.32 component landing) Land the two pacing components on current k8s 1.31 before the 1.32 k3s hop. Cilium 1.17 supports k8s 1.29-1.32; Longhorn 1.8 supports 1.25+; Gateway API CRDs stay v1.2.1 (Cilium 1.17 targets v1.2.x). Also commits the full upgrade plan doc. Co-Authored-By: Claude Opus 4.8 --- docs/plans/cluster-upgrade-plan.md | 581 +++++++++++++++++++ infra/ansible/roles/cilium/defaults/main.yml | 2 +- platform/cilium/helmrelease.yaml | 2 +- platform/longhorn/helmrelease.yaml | 2 +- 4 files changed, 584 insertions(+), 3 deletions(-) create mode 100644 docs/plans/cluster-upgrade-plan.md diff --git a/docs/plans/cluster-upgrade-plan.md b/docs/plans/cluster-upgrade-plan.md new file mode 100644 index 0000000..2be61f9 --- /dev/null +++ b/docs/plans/cluster-upgrade-plan.md @@ -0,0 +1,581 @@ +# Cluster upgrade plan — k3s 1.31.5 → 1.36.x + node OS (dev cluster) + +**Scope:** the `home-network-k3s-impl` **dev** cluster (`k3s-dev-1/2/3` @ 192.168.1.50-52), 3× HA +embedded-etcd servers behind a kube-vip VIP, Debian 12 bookworm. Upgrade Kubernetes/k3s from +**v1.31.5+k3s1 → v1.36.x+k3s1** (five minors) and the node OS (Debian 12 → 13), with an +etcd-snapshot / Velero / CNPG backup gate and explicit PASS/STOP criteria at **every** step, and a +defined rollback for each hop. + +> Status: **PLAN ONLY.** Nothing here has been executed. All version pins marked "confirm at +> execution" must be re-checked against the upstream release page on the day of the hop — this doc +> was written 2026-08-29 and homelab component versions move weekly (Renovate). + +> Companion runbooks that this plan reuses wholesale: +> - `docs/plans/ssd-etcd-execution-runbook.md` — the **proven safe-bounce** procedure (Longhorn +> rebuild-freeze + one-node-at-a-time + leader-last + per-node verify gates). Every k3s hop below +> is a parameterised repeat of that runbook. +> - `docs/plans/ssd-etcd-isolation.md` — why etcd is fragile on this cluster (slow HDD; etcd now on +> a dedicated SSD `virtio2`) and how a single member can be reset+rejoined from the other two. + +--- + +## 0. TL;DR / operator decisions needed + +| # | Decision | Recommendation | Needs operator sign-off | +|---|---|---|---| +| D1 | k3s upgrade **method** | Scripted **in-place per-node** binary upgrade driven by Ansible, one node at a time, leader-last, with the Longhorn rebuild-freeze. NOT system-upgrade-controller (its auto-drain fights this cluster's slow-HDD Longhorn and removes the human gate between nodes). | **Yes** — SUC vs scripted-in-place | +| D2 | Node **OS** upgrade | **Immutable re-provision** via tofu (bump the cloud image to Debian 13 trixie, replace one node at a time, rejoin etcd), run as a **separate campaign** from the k3s hops. Not in-place `apt` dist-upgrade. | **Yes** — Debian 13 now, or stay on 12 | +| D3 | Do OS + k3s on the **same** node bounce? | **No.** Never combine. Finish the k3s ladder to a stable target, then run the OS campaign (or vice-versa), one variable per bounce. | Confirm | +| D4 | Sequencing granularity | One k3s **minor** per maintenance window (5 windows). Cilium + Longhorn + Gateway-API CRDs advance in **lockstep** inside the same window (they are the pacing blockers). | Confirm | +| D5 | Flux `image.toolkit.fluxcd.io/v1beta2` → `/v1` migration | **Must** be done before Flux is upgraded to v2.7+ (which is required for k8s 1.36). Do it early, decoupled. | Confirm timing | + +**Headline blockers (do these or the hop fails):** +1. **Cilium 1.16.5 is already at/near its ceiling** — Cilium 1.16 tops out around k8s 1.30-1.31. + Cilium **must** step 1.16→1.17→1.18→1.19→1.20 in lockstep with the k8s minors. It is the single + most urgent item. +2. **Longhorn 1.7.3** must step 1.7→1.8→1.9→1.10→1.11→1.12 (one minor at a time) to reach a + 1.36-validated release (1.12.x). +3. **Flux v2.4.0 → v2.7+** is required for k8s 1.36, and v2.7 **removed** the image-automation + `v1beta2` API the repo still uses. +4. **KubeVirt v1.9.0 / CDI v1.66.0 are already 1.36-ready** — a pleasant surprise; no VM-stack + upgrade needed, just keep them where they are. + +--- + +## 1. Confirmed current-state inventory + +### 1.1 Nodes (live, verified 2026-08-29 via `dev.sh run kubectl get nodes -o wide`) + +| Node | IP | Roles | k8s | OS | Kernel | Runtime | +|---|---|---|---|---|---|---| +| k3s-dev-1 | 192.168.1.50 | control-plane,etcd,master | v1.31.5+k3s1 | Debian 12 bookworm | 6.1.0-52-cloud | containerd 1.7.23-k3s2 | +| k3s-dev-2 | 192.168.1.51 | control-plane,etcd,master | v1.31.5+k3s1 | Debian 12 bookworm | 6.1.0-52-cloud | containerd 1.7.23-k3s2 | +| k3s-dev-3 | 192.168.1.52 | control-plane,etcd,master | v1.31.5+k3s1 | Debian 12 bookworm | 6.1.0-52-cloud | containerd 1.7.23-k3s2 | + +- 3 servers, embedded etcd, **all** control-plane (no dedicated workers). Quorum = 2/3. +- kube-vip control-plane VIP **192.168.1.49** (fronted through pfSense NAT at 192.168.2.50:6443). +- etcd datastore is on the **dedicated SSD `virtio2` (`/dev/vdc`, LABEL=k3s-etcd)** mounted at + `/var/lib/rancher/k3s/server/db` (migrated this session). Longhorn data disk = `virtio1` + (`/dev/vdb`, `/var/lib/longhorn`). OS = `virtio0` (`/dev/vda`). +- **18** Longhorn volumes, all `healthy` (the count is the post-step invariant). + +### 1.2 Where versions are pinned in this repo + +| Thing | Pin location | Current value | +|---|---|---| +| **k3s version** | `infra/ansible/roles/k3s_server/defaults/main.yml` → `k3s_version`; applied via `INSTALL_K3S_VERSION` in `roles/k3s_server/tasks/main.yml` | `v1.31.5+k3s1` | +| k3s install guard | `roles/k3s_server/tasks/main.yml` — install task has `creates: /etc/rancher/k3s/k3s.yaml` | **idempotent → will NOT upgrade a running node on a version bump** (see §4) | +| kubectl/helm/flux (toolbox) | `infra/devcontainer/versions.env`, `ci/versions.env` | kubectl 1.31.4, flux 2.4.0 | +| **Cilium (ongoing)** | `platform/cilium/helmrelease.yaml` → `version: "1.16.5"` | 1.16.5 | +| Cilium (bootstrap) + **Gateway API CRDs** | `infra/ansible/roles/cilium/defaults/main.yml` → `gateway_api_version: "v1.2.1"`, `gateway_api_crd_urls` | v1.2.1 | +| **Longhorn** | `platform/longhorn/helmrelease.yaml` → `version: ">=1.7.0 <1.8.0"` (live 1.7.3) | 1.7.3 | +| **KubeVirt + CDI** | `platform/kubevirt/kustomization.yaml` (remote release manifests) | KubeVirt v1.9.0, CDI v1.66.0 | +| **kube-vip** | `infra/ansible/roles/kube_vip/defaults/main.yml` → `kube_vip_version` | v0.8.9 | +| **external-snapshotter** | `platform/external-snapshotter/kustomization.yaml` (`ref=v8.2.0`) | v8.2.0 | +| **cert-manager** | `platform/cert-manager/helmrelease.yaml` → `">=1.16.0 <1.17.0"` (live 1.16.5) | 1.16.5 | +| **CNPG** | `platform/cnpg/helmrelease.yaml` → `">=0.22.0 <1.0.0"` (live chart 0.29.0) | operator ≈1.26/1.27 era | +| **kube-prometheus-stack** | `platform/monitoring/kube-prometheus-stack.yaml` → `">=65.0.0 <66.0.0"` (live 65.8.1) | 65.8.1 | +| **Loki** | `platform/monitoring/loki.yaml` → `">=6.0.0 <7.0.0"` (live 6.55.0) | 6.55.0 | +| **Alloy** | `platform/monitoring/alloy.yaml` → `">=0.10.0 <1.0.0"` (live 0.12.6) | 0.12.6 | +| **Velero** | `platform/velero/helmrelease.yaml` → `">=8.0.0 <9.0.0"` (live 8.7.2, Velero v1.15.x) | 8.7.2 | +| **Kyverno** | `platform/kyverno/admission/helmrelease.yaml` → `version: "3.3.x"` (live 3.3.9) | 3.3.9 | +| **oauth2-proxy** | `platform/oauth2-proxy/helmrelease.yaml` → `">=7.0.0 <8.0.0"` (live 7.18.0) | 7.18.0 | +| **external-dns** | `platform/externaldns/helmrelease.yaml` → `">=1.15.0 <1.16.0"` (live 1.15.2) | 1.15.2 | +| **headlamp** | `platform/headlamp/helmrelease.yaml` → `">=0.41.0 <0.42.0"` (live 0.41.0) | 0.41.0 | +| **Flux** | installed by `flux bootstrap` (toolbox v2.4.0); controllers live in `flux-system` | v2.4.0 | +| **tofu cloud image** | `infra/tofu/variables.tf` → `image_source`, `image_file_name` | `debian-12-generic-amd64.qcow2` (bookworm) | + +### 1.3 Live platform component versions (verified 2026-08-29) + +Flux 2.4.0 (source-controller v1.4.1, kustomize-controller v1.4.0, helm-controller v1.1.0, +image-automation v0.39.0, image-reflector v0.33.0, notification v1.4.0) · cert-manager v1.16.5 · +CNPG chart 0.29.0 (PostgreSQL 16.4, 3 single-instance clusters: keycloak-db, nextcloud-db, +semaphore-db) · Cilium 1.16.5 · Kyverno 3.3.9 · Longhorn 1.7.3 · Alloy 0.12.6 · +kube-prometheus-stack 65.8.1 · Loki 6.55.0 · oauth2-proxy 7.18.0 · Velero 8.7.2 (+plugin-for-aws +v1.11.1) · KubeVirt v1.9.0 · CDI v1.66.0 · snapshot-controller v8.2.0 · kube-vip v0.8.9 · +Gateway API CRDs v1 (v1.2.1 line). + +### 1.4 CRD / API surface the repo actually uses (grep of `platform/ apps/ catalog/ clusters/`) + +`v1`, `apps/v1`, `networking.k8s.io/v1`, `rbac.authorization.k8s.io/v1`, `storage.k8s.io/v1`, +`gateway.networking.k8s.io/v1`, `cilium.io/v2`, `monitoring.coreos.com/v1`, +`helm.toolkit.fluxcd.io/v2`, `source.toolkit.fluxcd.io/v1`, `kustomize.toolkit.fluxcd.io/v1`, +`image.toolkit.fluxcd.io/v1beta2` ⚠, `postgresql.cnpg.io/v1`, `kubevirt.io/v1`, +`cdi.kubevirt.io/v1beta1`, `cert-manager.io/v1`, `kyverno.io/v1`, `k8s.keycloak.org/v2alpha1`, +`snapshot.storage.k8s.io/v1`, `client.authentication.k8s.io/v1beta1` (kubeconfig exec plugin). + +**None of these are Kubernetes *core* APIs removed in 1.32-1.36.** The only removal that bites is a +**Flux** CRD (`image.toolkit.fluxcd.io/v1beta2`) — see §3. + +--- + +## 2. k3s upgrade path (sequential, one minor at a time) + +### 2.1 The path and why skipping is unsafe + +``` +1.31.5+k3s1 → 1.32.z → 1.33.z → 1.34.z → 1.35.z → 1.36.z (+k3s1 each) +``` + +**Skipping minors is not supported.** Kubernetes' version-skew policy allows the kube-apiserver and +components to differ by at most **one minor** during an upgrade, and a downgrade of the etcd +datastore is not supported. Jumping >1 minor risks an apiserver that can't read objects written by +an intermediate storage version and control-plane components outside the skew window. On k3s (all +three nodes are servers/apiserver), the whole control plane moves together per hop, so the rule is: +**one minor per campaign, all three nodes brought to the same minor before the next.** +Source: . + +### 2.2 Target patch per minor (confirm at execution) + +Pick the **latest patch of each minor** from (or the k3s +channel server `https://update.k3s.io/v1-release/channels`, where channel `v1.3X` resolves to the +latest patch of that minor). Known-latest as of writing: + +| Hop | Target (min) | Notes / source | +|---|---|---| +| → 1.32 | latest `v1.32.z+k3s1` | confirm on releases page | +| → 1.33 | latest `v1.33.z+k3s1` | k3s 1.33 release notes: | +| → 1.34 | latest `v1.34.z+k3s1` | confirm | +| → 1.35 | latest `v1.35.z+k3s1` | confirm | +| → 1.36 | **`v1.36.3+k3s1`** (Aug 4 2026) or newer (`v1.36.4+k3s1` reported) | | + +### 2.3 k3s-specific breaking changes across the path + +- **containerd major bump 1.7 → 2.x.** k3s 1.36.3 ships **containerd v2.3.2-k3s2** (current node is + 1.7.23-k3s2). The containerd v1→v2 migration happens somewhere in the 1.32-1.34 window. It is + transparent for standard workloads, but: (a) `config.toml` schema changed — this repo sets + `device_ownership_from_security_context` via `--nonroot-devices` (k3s config.yaml), which k3s + regenerates into the new containerd config on restart; verify it survives (KubeVirt/CDI Block-mode + PVCs depend on it — see `roles/k3s_server/tasks/main.yml`). (b) Any node with custom + `/etc/rancher/k3s/config.toml.tmpl` would need review — this cluster has none. +- **etcd 3.5 → 3.6.** k3s 1.36.3 ships **etcd v3.6.14-k3s1**. etcd datastore version bumps are + **one-way** (no downgrade). This is the single strongest reason for the pre-hop etcd snapshot gate + (§6) and defines the point-of-no-return (§7). +- **Traefik chart v40 breaking change** in k3s ≥1.36.2 (ingress-nginx provider rename). **N/A** — + this cluster disables Traefik (`k3s_disable: "traefik,servicelb,local-storage"`); Cilium Gateway + API owns ingress. +- **kube-proxy / flannel / network-policy already disabled** (`--disable-kube-proxy`, + `--flannel-backend=none`, `--disable-network-policy`) so k8s 1.36's **IPVS-mode kube-proxy + deprecation is N/A** — Cilium is the kube-proxy replacement. +- Always read the **Kubernetes urgent-upgrade notes** for each minor before the hop: + (`CHANGELOG-1.3X.md` → + "Urgent Upgrade Notes"). + +--- + +## 3. Removed / deprecated API audit (1.32 → 1.36) + +### 3.1 Kubernetes core removals/deprecations per minor + +| k8s | Removed / deprecated | Affects this repo? | +|---|---|---| +| **1.32** | **`flowcontrol.apiserver.k8s.io/v1beta3`** (FlowSchema, PriorityLevelConfiguration) **removed** — use `flowcontrol.apiserver.k8s.io/v1` (GA since 1.29). Field rename `assuredConcurrencyShares`→`nominalConcurrencyShares`. | **No** — repo defines no FlowSchema/PriorityLevelConfiguration CRs. | +| **1.33** | `status.nodeInfo.kubeProxyVersion` **field removed** from Node. | **No** — nothing reads it. | +| **1.34** | **AppArmor** annotations path deprecations continue; `AppArmor` GA'd earlier. | No repo AppArmor annotations. | +| **1.35** | Minor graduations; no removal that touches this repo's API set. | No. | +| **1.36** | **`gitRepo` volume permanently disabled**; **Service `externalIPs` deprecated**; **kube-proxy IPVS** on a deprecation clock. | **No** — no gitRepo volumes, no `externalIPs`, kube-proxy already replaced by Cilium. | + +Sources: · + · +per-minor `CHANGELOG-1.3X.md`. + +**Conclusion: the repo's manifests use no Kubernetes core API removed in 1.32-1.36.** (Grep for +`flowcontrol.apiserver`, `gitRepo:`, `externalIPs:`, `policy/v1beta1`, `batch/v1beta1`, +`autoscaling/v2beta*`, `extensions/v1beta1`, `PodSecurityPolicy` returns **nothing**.) + +### 3.2 The one real removal that bites — Flux `image.toolkit.fluxcd.io/v1beta2` + +**Flux v2.7+ removed `image.toolkit.fluxcd.io/v1beta2`** (ImageRepository, ImagePolicy, +ImageUpdateAutomation → graduated to `image.toolkit.fluxcd.io/v1`). Flux v2.7+ is required for k8s +1.36 (§4.1). The repo still uses `v1beta2` in: +- `platform/image-automation/image-update-automation.yaml` +- `platform/image-automation/per-service/pihole.example.yaml` +(plus any per-service ImageRepository/ImagePolicy generated from that template.) + +**Required migration (do BEFORE upgrading Flux to v2.7+):** follow the official +"Upgrade Procedure for Flux v2.7+" — migrate stored objects off v1beta2, then update every manifest +in Git from `image.toolkit.fluxcd.io/v1beta2` → `image.toolkit.fluxcd.io/v1`. +Source: · +. + +### 3.3 Detection tooling — run BEFORE each hop + +Add a read-only pre-hop gate using **both** a live-cluster scan and a manifest scan: + +```bash +# pluto — scans live cluster AND Helm releases for deprecated/removed apiVersions, target minor +./infra/devcontainer/dev.sh run bash -lc 'pluto detect-all-in-cluster --target-versions k8s=v1.3X.0 -owide' +# or scan the rendered manifests in git: +./infra/devcontainer/dev.sh run bash -lc 'pluto detect-files -d platform -d apps --target-versions k8s=v1.3X.0' + +# kubent (kube-no-trouble) — quick live scan +./infra/devcontainer/dev.sh run bash -lc 'kubent --target-version 1.3X' +``` + +- `pluto` and `kubent` detect **built-in** deprecated APIs from their bundled version data; they do + **not** know about third-party CRD removals (e.g. the Flux `v1beta2` above) unless given a custom + deprecation list. So pair the tool pass with the manual §3.2 check and a `grep` of the repo for + each component's deprecated CRD versions before its own upgrade. +- Neither tool is currently in the toolbox image; add to `infra/devcontainer/Containerfile` + + `versions.env`, or run once via `go install`/release binary inside the devcontainer. +- Sources: · . + +--- + +## 4. Component compatibility matrix & upgrade sequencing + +### 4.1 Kubernetes-version support per component (target = k8s 1.36) + +| Component | Current | k8s-1.36 target (confirm patch) | k8s support fact | Must move… | Source | +|---|---|---|---|---|---| +| **Cilium** | 1.16.5 | **1.20.x** | 1.16:≤~1.30 · 1.17:1.29-1.32 · 1.18:1.30-1.33 · 1.19:1.32-1.35 · 1.20:1.33-1.36 | **in lockstep, every hop** (pacing blocker #1) | compatibility.fyi/cilium; docs.cilium.io/…/compatibility | +| **Gateway API CRDs** | v1.2.1 | **v1.6.x** (Cilium 1.20 targets v1.6.1 — verify) | too old for Cilium 1.20 | with each Cilium hop | docs.cilium.io/en/v1.20/…/gateway-api | +| **Longhorn** | 1.7.3 | **1.12.x** (1.12.1) | 1.7:≤~1.31 · 1.11.1:1.32-1.35 · 1.12:adds 1.36; min k8s stays v1.21 | one minor at a time (pacing blocker #2) | endoflife.date/longhorn; longhorn.io best-practices; SUSE support matrix | +| **KubeVirt** | **v1.9.0** | **v1.9.x (already there)** | v1.9 validates 1.34/1.35/1.36 (N-2) | **no upgrade needed** | github.com/kubevirt/sig-release k8s-support-matrix | +| **CDI** | **v1.66.0** | keep paired w/ KubeVirt v1.9 | same window as KubeVirt | no upgrade needed (keep pair) | kubevirt/containerized-data-importer releases | +| **external-snapshotter / snapshot-controller** | v8.2.0 | **latest v8.x (≥8.6)** | v8 tracks current k8s; snapshot CRDs fwd/back-compatible | loose — opportunistic | github.com/kubernetes-csi/external-snapshotter releases | +| **cert-manager** | 1.16.5 | **1.21.x** | 1.21 supports 1.33-1.36 | before/with the later hops | cert-manager.io/docs/releases | +| **CNPG** | chart 0.29.0 (op ≈1.26/1.27) | **1.30.x** | 1.30 supports 1.34/1.35/1.36; 1.29 supports 1.33-1.35 | before the 1.36 hop | cloudnative-pg.io/docs/devel/supported_releases | +| **Flux** | v2.4.0 | **v2.7+ (latest, ~v2.8.x)** | Flux N-2; v2.7+ needed for 1.36 | after §3.2 v1beta2→v1 migration | fluxcd.io/flux/releases | +| **kube-prometheus-stack** | 65.8.1 | **~88.x+** (supports kubectl/k8s 1.36) | chart 88.x tested on 1.36 | before/with later hops; **apply updated CRDs separately** | github.com/prometheus-community/helm-charts | +| **Loki (chart)** | 6.55.0 | **latest 6.x/newer** | uses stable core APIs — loose | opportunistic | grafana/helm-charts | +| **Alloy (chart)** | 0.12.6 | **latest** | loose (DaemonSet, stable APIs) | opportunistic | grafana/alloy | +| **Velero** | chart 8.7.2 (v1.15) | **latest (v1.17+/chart ~10.x)** | v1.16 tested→1.33, v1.17 tested→1.34; "1.18-latest" expected | before the 1.36 hop | velero.io support-matrix; compatibility.fyi/velero | +| **Kyverno** | 3.3.9 | **latest 3.x (N-2 for 1.36)** | Kyverno follows k8s N-2; 3.3.x too old for 1.36 | before/with later hops | kyverno.io/docs/installation/releases | +| **oauth2-proxy** | 7.18.0 | **latest 7.x** | loose | opportunistic | oauth2-proxy chart | +| **external-dns** | 1.15.2 | **latest 1.x** | loose | opportunistic | kubernetes-sigs/external-dns | +| **kube-vip** | v0.8.9 | **latest (k8s.io 0.36.x-based)** | leader-election uses coordination.k8s.io/v1 (stable); latest bumps client-go to 0.36 | before/with the 1.36 hop | github.com/kube-vip/kube-vip releases | +| **headlamp** | 0.41.0 | latest | loose UI | opportunistic | headlamp chart | + +### 4.2 The pacing blockers, and the per-hop lockstep + +**Cilium and Longhorn are the two items that force the pace** — each supports only ~3-4 k8s minors +and each upgrades **one minor at a time (no skipping, pre-flight required)**. KubeVirt/CDI are +already at target, so the VM stack is *not* a blocker this cycle. The mapping: + +| Window | k8s target | Cilium (HelmRelease) | Gateway API CRDs | Longhorn (HelmRelease) | Other components landed in this window | +|---|---|---|---|---|---| +| **W1** | 1.32 | **1.17** (covers 1.29-1.32) | bump toward Cilium 1.17's required version | **1.8** | — | +| **W2** | 1.33 | **1.18** (1.30-1.33) | bump | **1.9** | cert-manager→1.18/1.19; kube-prometheus-stack CRDs refresh | +| **W3** | 1.34 | **1.19** (1.32-1.35) | bump | **1.10** | CNPG→1.28/1.29; Velero→latest | +| **W4** | 1.35 | 1.19 still covers 1.35 (optional →1.20 early) | as needed | **1.11** | Kyverno→latest; **Flux v1beta2→v1 migration + Flux→v2.7+** | +| **W5** | 1.36 | **1.20** (1.33-1.36) | **v1.6.x** | **1.12** | cert-manager→1.21; CNPG→1.30; kube-vip→latest; snapshot-controller→latest v8.x | + +Ordering rule **inside** each window (critical): **CNI/storage compatibility first, then the k3s +control-plane hop, then verify.** Concretely per window: +1. Land the **component** upgrades that must *precede* the new k8s minor (Cilium minor, Gateway API + CRDs, Longhorn minor, and any component whose *current* version won't run on the new k8s). Let + Flux reconcile; verify healthy on the **still-current** k8s. +2. Run the **k3s minor hop** (§5) node-by-node. +3. Verify the components again on the **new** k8s; then land the "opportunistic/loose" bumps. + +> Rationale for "Cilium before the apiserver hop": if the apiserver moves to a minor the running +> Cilium doesn't support, CNI/agent may fail to start on that node and the node never goes Ready. +> Cilium's own guidance is to run its **pre-flight check** DaemonSet before each Cilium minor bump +> and to upgrade Cilium **before or alongside** the k8s minor within its supported window. + +--- + +## 5. Upgrade method — evaluation & recommendation (D1) + +### 5.1 Options + +**(a) Rancher system-upgrade-controller (SUC) with Plans.** Canonical k3s upgrade tool. A `Plan` +targets nodes by label, concurrency 1, HA-aware, drains → replaces the k3s binary → uncordons. +- **Pros:** official, declarative, handles server-vs-agent ordering; could live in GitOps. +- **Cons for THIS cluster:** SUC's default **drain evicts pods**, which on this slow-HDD Longhorn + triggers replica **rebuild** storms that starve etcd WAL fsync and flap the VIP — the exact + failure `ssd-etcd-isolation.md` documents. You'd have to disable drain (`drain: {enabled:false}`) + and still manually apply the Longhorn freeze, at which point SUC's main value (drain automation) is + gone. Worse, SUC **automates away the human gate between nodes** — the user's hard requirement is + an explicit backup + PASS/STOP verification *between* every node, which SUC does not pause for. + +**(b) Scripted in-place per-node binary upgrade (Ansible-driven).** Re-run the k3s install script +with the new `INSTALL_K3S_VERSION` on one node at a time; k3s replaces its binary and restarts, +etcd re-joins at the new version. This is exactly the `ssd-etcd-execution-runbook.md` bounce shape. +- **Pros:** reuses the **proven** freeze + leader-last + per-node-verify pattern; no drain (no + rebuild storm — replicas *reattach*); the human gate sits naturally between nodes; minimal moving + parts. **Note the `creates:` guard:** the existing install task self-skips on a running node, so + the upgrade path must invoke the install script *explicitly* (a dedicated play/tag, or an ad-hoc + `curl … | INSTALL_K3S_VERSION= sh -s - server`), not rely on bumping the var alone. +- **Cons:** does not touch the OS (handled separately, §6/D2); slightly more manual than SUC. + +**(c) Immutable re-provision via tofu.** Bump `image_source` + `k3s_version`, `tofu replace` one VM +at a time, node rejoins etcd fresh and Longhorn rebuilds that node's replicas. +- **Pros:** **also solves the OS upgrade** in the same motion; clean, reproducible cattle nodes. +- **Cons:** destroys the node's etcd member + **all** its Longhorn replicas → full **rebuild** (not + reattach) on the slow HDD per node (throttled but heavy, ~hours for 18 volumes' replicas across + three rebuilds); the etcd SSD `virtio2` is wiped on replace (fine for a 3-node HA etcd — the new + member syncs from the other two). Heavier and slower than (b) for a *pure k8s* hop. + +### 5.2 Recommendation + +- **k3s minor hops (W1-W5): choose (b), scripted in-place per-node.** It matches the cluster's + proven safe-bounce, avoids rebuild storms, and preserves the mandatory per-node human gate. Pin + the new tag in `roles/k3s_server/defaults/main.yml` (`k3s_version`) **and** drive the actual + upgrade with an explicit install-script invocation (bypassing the `creates:` guard), `serial: 1`, + leader-last. +- **Node OS (Debian 12→13): choose (c), immutable re-provision**, as a **separate** campaign (§8). + A clean trixie image beats an in-place `apt` dist-upgrade for reproducibility, and the nodes are + already cattle (cloud-init + `node_baseline` rebuild them). Accept the per-node Longhorn rebuild + there (that campaign *wants* rebuild, throttled at limit=1, since replicas are genuinely gone). +- SUC remains a reasonable future option **once the cluster is on fast disks** (the prod/mini-PC + Phase-7 cluster) where drain-driven rebuild is cheap; note it for later. + +--- + +## 6. Data-validation gates at EVERY step (hard requirement) + +Every **node bounce** (there are 3 per k3s window; 3 per OS window) is wrapped by a pre-step backup +gate and a post-step verification gate. Every **window** additionally takes a full cluster backup +before the first node. + +### 6.1 PRE-STEP backup gate (before the first node of a window, and re-confirm before each node) + +```bash +# 1) etcd on-demand snapshot (k3s native) — the primary rollback artifact. Run on the leader. +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible -b -m shell \ + -a "k3s etcd-snapshot save --name pre-1.3X-$(date +%Y%m%d%H%M)"' +# verify it exists + is copied off-node (snapshots land in /var/lib/rancher/k3s/server/db/snapshots; +# also archived to S3 if configured). Confirm size > 0 and note the name. + +# 2) Velero backup (cluster resources + PVC data via kopia FSB → MinIO 192.168.1.9:9000) +./infra/devcontainer/dev.sh run bash -lc 'velero backup create pre-1.3X-$(date +%Y%m%d%H%M) --wait' +./infra/devcontainer/dev.sh run bash -lc 'velero backup describe --details' # Phase=Completed, 0 errors + +# 3) CNPG logical backup + verify, for each of keycloak-db / nextcloud-db / semaphore-db +# (they already run continuous WAL + daily base backups to MinIO via barmanObjectStore). +# Trigger an on-demand base backup and confirm it completes: +./infra/devcontainer/dev.sh run kubectl -n create -f - <<'EOF' +apiVersion: postgresql.cnpg.io/v1 +kind: Backup +metadata: { name: pre-upg-, namespace: } +spec: { method: barmanObjectStore, cluster: { name: } } +EOF +./infra/devcontainer/dev.sh run kubectl -n get backup pre-upg- -o jsonpath='{.status.phase}' # completed +# capture an integrity baseline to compare after: +./infra/devcontainer/dev.sh run kubectl -n exec -1 -- psql -tAc \ + "select (select count(*) from pg_database), (select sum(('x'||substr(md5(t.oid::text),1,8))::bit(32)::bigint) from pg_class t)" +``` + +**PASS to proceed:** etcd snapshot present (>0 bytes, name recorded) **AND** Velero backup +`Completed` with 0 errors **AND** all three CNPG on-demand backups `completed` with baseline +captured. +**STOP (do not start the hop):** any backup fails, errors, or cannot be verified restorable. + +### 6.2 The per-node bounce itself (reuse `ssd-etcd-execution-runbook.md`) + +Per node `NODE=k3s-dev-N`, `VMID`, **leader last** (re-verify leader before each node): + +```bash +# leader check +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell \ + -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_is_leader"' + +# 1) FREEZE Longhorn rebuilds (reattach, not rebuild) — the proven safe-bounce +./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io \ + concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"0"}' + +# 2) Upgrade k3s ON THIS NODE ONLY (explicit install, bypassing the creates: guard): +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible $NODE -b -m shell \ + -a "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.3X.z+k3s1 sh -s - server"' +# (k3s replaces its binary and restarts; embedded etcd re-joins at the new version) + +# 3) VERIFY this node fully before touching the next (see §6.3), THEN +# 4) THAW rebuilds only after 18/18 healthy: +./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io \ + concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"1"}' +``` + +### 6.3 POST-STEP verification gate (after each node; full set after the last node of a window) + +```bash +./infra/devcontainer/dev.sh run kubectl get nodes -o wide # NODE Ready, VERSION = new minor +./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd # ok +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell \ + -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader"' # all = 1 +./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io \ + -o custom-columns=R:.status.robustness --no-headers | grep -c healthy # 18 +./infra/devcontainer/dev.sh run kubectl -n kube-system rollout status ds/cilium --timeout=300s +./infra/devcontainer/dev.sh run bash -lc 'flux get kustomizations -A && flux get helmreleases -A' # all Ready +# etcd snapshot restorability spot-check (list; a scratch restore is the W-level drill, §7) +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible -b -m shell \ + -a "k3s etcd-snapshot ls"' +# CNPG integrity vs the §6.1 baseline +./infra/devcontainer/dev.sh run kubectl -n exec -1 -- pg_isready # accepting connections +# re-run the row-count/checksum query and diff against baseline (must match) +``` + +**App smoke tests (the QA-sweep URL set)** — after each *window* completes, expect HTTP 200/302 +(or the app's healthy code) from: +`auth` · `crafty` · `gitea` · `grafana` · `headlamp` · `jackett` · `keycloak` +(`/realms/homelab/.well-known/openid-configuration`) · `nextcloud` · `pihole` · `plex` · +`qbittorrent` · `radarr` · `seerr` · `semaphore` · `sonarr` · `unifi` — all under +`*.home.keithjackson.dev`. Plus one KubeVirt VM check (a VMI Running / console reachable). + +**PASS to advance to the next node/window:** all nodes Ready at the expected version · etcd +`/readyz/etcd`=ok and 3/3 have a leader · **18/18** Longhorn healthy (reattached, no rebuild storm) +· Cilium DaemonSet rolled out · all Flux Kustomizations + HelmReleases Ready · CNPG `pg_isready` + +row-count/checksum unchanged vs baseline · QA-sweep all green. +**STOP (halt, leave freeze=0, do not advance):** etcd `/readyz` not ok · a member without a leader · +VIP `192.168.2.50:6443` flapping · a node not rejoining/not Ready · any Longhorn volume Faulting · +k3s crash-looping · a CNPG DB failing `pg_isready` or checksum drift · a Flux HR stuck not-Ready · +any QA-sweep URL down. Recovery per §7. + +--- + +## 7. Rollback per step & point-of-no-return + +### 7.1 Per-node (before etcd majority is on the new minor) + +The safest rollback for a single bad node is to **re-pin the old k3s version on that node** and +re-run the install script for the old tag, then verify it rejoins at the old minor; or reset the +member and rejoin from the other two (2/3 quorum holds), per `ssd-etcd-isolation.md` §4.5. Because +you go **leader-last and one node at a time**, at any moment ≤1 node is on the new minor until the +2nd node upgrades. + +### 7.2 Point-of-no-return per hop + +**When the *second* server reaches the new minor, etcd majority (2/3) is on the new datastore +version** (etcd 3.5→3.6 across the path is one-way). Past that point, "roll back the node" is no +longer enough — a true rollback is an **etcd snapshot restore** of the whole cluster to the pre-hop +snapshot: + +```bash +# cluster-wide restore from the pre-hop snapshot (DISRUPTIVE — full control-plane stop): +# on ONE server, with k3s stopped on all three: +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -b -m shell -a "systemctl stop k3s"' +./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible -b -m shell \ + -a "k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/"' +# then wipe db on the other two and rejoin them (standard k3s cluster-reset-restore flow), +# and PIN k3s_version back to the old tag before they re-join. +``` + +After an etcd restore, **Longhorn volume data is recovered by Velero (kopia FSB) / CNPG PITR** for +anything written between snapshot and restore — this is why the §6.1 triad (etcd + Velero + CNPG) +is mandatory, not just etcd. + +### 7.3 Window-level DR drill (do once, in W1, to prove the path) + +Before the first real hop, prove restorability end-to-end: `velero restore create --from-backup + --namespace-mappings :-scratch` into a scratch namespace and confirm data lands +(this is the existing Phase-4 R-STATE-03 gate). If the drill fails, **do not begin the upgrade.** + +### 7.4 OS-campaign rollback + +Re-provision is inherently reversible per node: if a re-provisioned trixie node misbehaves, +`tofu` re-apply with `image_source` reverted to bookworm re-creates it on Debian 12; the node +rejoins etcd and rebuilds replicas as with any replacement. Point-of-no-return is the same etcd +majority rule if OS re-provision is (against advice) combined with a k3s bump — which is why D3 says +keep them separate. + +--- + +## 8. Node OS upgrade — Debian 12 → 13 (trixie) + +- **Debian 13 "trixie" released 2025-08-09, ships Linux 6.12 LTS** (up from bookworm's 6.1). + Support to 2028-08 (LTS 2030-08). Sources: · + . +- **Kernel implications (positive):** 6.12 is newer than Cilium's/Longhorn's minimums and improves + eBPF/BPF feature coverage for Cilium and has current `iscsi_tcp`/`nfs`/`dm_crypt`/`overlay`/ + `br_netfilter` modules Longhorn needs (all already pinned in + `roles/node_baseline/defaults/main.yml` → `node_baseline_kernel_modules`). No known regression for + this stack; `open-iscsi`/`nfs-common` packages exist on trixie. +- **Method (D2): immutable re-provision.** Bump `infra/tofu/variables.tf`: + - `image_source` → `https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2` + - `image_file_name` → `debian-13-generic-amd64.qcow2` + Then `tofu apply -replace=proxmox_virtual_environment_vm.node["k3s-dev-N"]` **one node at a time**, + let `node_baseline` + `k3s_server` (secondary/join) rebuild it, and let it rejoin etcd. Because a + replaced node loses its Longhorn replicas, run this campaign with rebuild **throttled at 1** (not + frozen at 0 — you *want* the rebuild here), wait for 18/18 healthy before the next node, and keep + the same per-node backup + verify gates (§6). +- **Coordinate with k3s (D3):** run the OS campaign **after** the cluster is stable on a chosen k3s + minor (either finish the whole 1.36 ladder first, or interleave one full campaign between k3s + windows) — **never** re-provision to trixie *and* bump k3s in the same node bounce; that entangles + two failure domains and defeats per-step rollback. +- **In-place `apt` dist-upgrade** is the fallback if re-provision is undesirable, but it is discarded + here: it mutates cattle nodes outside the tofu/cloud-init source of truth, risks half-migrated + states across three nodes, and violates the repo's "changes go through IaC" principle. + +--- + +## 9. Sequencing & HA safety + +- **Quorum:** 3 servers, embedded etcd, quorum 2/3. Exactly **one** node down at a time keeps + quorum. Never bounce two simultaneously. +- **Leader last:** re-verify the etcd leader before each node (`etcd_server_is_leader` on :2381) and + upgrade followers first, leader last, so leadership only moves once and late. +- **VIP stability:** kube-vip floats 192.168.1.49 to the current leader; with one node down the VIP + fails over to a surviving leader and the API stays reachable through the pfSense NAT + (192.168.2.50:6443). Watch for VIP flapping as a STOP signal. +- **Rebuild-freeze (reattach, not rebuild):** for k3s in-place hops set + `concurrent-replica-rebuild-per-node-limit=0` around each bounce so a briefly-down node's replicas + **reattach** instead of triggering a rebuild storm that starves etcd WAL fsync (the whole reason + etcd is on a dedicated SSD). Thaw to `1` only after 18/18 healthy. (For the OS re-provision + campaign use `1`, not `0`, since replicas are genuinely gone and must rebuild.) +- **CNPG note:** all three DBs are **single-instance** (`instances: 1`) — each incurs a brief + outage when its pod reschedules during a node bounce. Order/announce accordingly; the app + smoke-tests (§6.3) will blip while a DB pod moves. (Consider `instances: 3` for the prod cluster.) +- **KubeVirt/CDI:** already 1.36-ready; VMs without `evictionStrategy: LiveMigrate` simply + Evict+restart on a node bounce (we do **not** drain in method (b), so VMs are only affected on + the node actually bouncing). + +**Estimated maintenance windows:** per k3s minor ≈ **60-120 min** (3 nodes × ~15-30 min bounce + +verify + freeze/thaw + component reconcile), so **5 windows ≈ 5-10 h** total spread across days, +plus the W-level backups. OS re-provision campaign ≈ **2-4 h per node** (dominated by Longhorn +rebuild of that node's share of 18 volumes on HDD) → ~1-1.5 days, one node per session. + +--- + +## 10. Dev-first, then prod (Phase-7 carry-over) + +This dev run is the rehearsal for the eventual **prod / mini-PC** cluster. What carries over: +- The **method** (scripted in-place per-node for k3s; immutable re-provision for OS), the + **leader-last + freeze + per-node-gate** discipline, and the **backup triad** gate are + cluster-agnostic and become the prod runbook verbatim. +- The **component lockstep table** (§4.2) is the reusable dependency map; re-confirm exact patch + numbers at prod time (they will have moved). +- **Differences to expect on prod:** fast disks (NVMe/SSD) make Longhorn rebuild cheap → **SUC + becomes viable** there (drain no longer dangerous), and multi-instance CNPG (`instances: 3`) + removes the single-DB outage blip. Prod may also run dedicated workers, changing SUC Plan + server-vs-agent ordering. +- The **Flux v1beta2→v1 migration** (§3.2) is a one-time repo change shared by both clusters — do it + once, both benefit. + +--- + +## Appendix A — pre-flight checklist per window (copy/paste) + +``` +[ ] Read k3s release notes for the target minor + k8s CHANGELOG "Urgent Upgrade Notes" +[ ] pluto/kubent scan (live + manifests) against target minor → clean (or Flux v1beta2 handled) +[ ] Component pre-reqs landed & Ready on CURRENT k8s (Cilium minor, Gateway CRDs, Longhorn minor, …) +[ ] Cilium pre-flight check DaemonSet passed for the Cilium minor bump +[ ] PRE-STEP backup gate PASS: etcd snapshot + Velero Completed + 3× CNPG completed + baselines captured +[ ] (W1 only) Velero restore DR drill into scratch ns PASS +[ ] Leader identified; node order = followers → leader +[ ] For each node: freeze(0) → upgrade → §6.3 verify PASS → thaw(1) → 18/18 healthy → next +[ ] After last node: full §6.3 + QA-sweep all green + KubeVirt VM check +[ ] Land opportunistic/loose component bumps; Flux all Ready +[ ] Pin the new k3s_version in roles/k3s_server/defaults/main.yml; PR → main (never commit to main) +[ ] Record outcome in DECISIONS-LOG.md +``` + +## Appendix B — sources + +- k3s releases & notes: · · · channels +- k8s version-skew: · deprecation guide · v1.36 sneak peek +- Cilium compat: · · Gateway API +- Longhorn: · · · +- KubeVirt/CDI: · +- external-snapshotter: +- cert-manager: +- CNPG: +- Flux: · v2.7 upgrade · image API +- kube-prometheus-stack: +- Velero: · +- Kyverno: +- kube-vip: · +- Debian 13: · +- Tools: pluto · kubent +``` diff --git a/infra/ansible/roles/cilium/defaults/main.yml b/infra/ansible/roles/cilium/defaults/main.yml index cc930d5..a7da89b 100644 --- a/infra/ansible/roles/cilium/defaults/main.yml +++ b/infra/ansible/roles/cilium/defaults/main.yml @@ -5,7 +5,7 @@ # (kubeProxyReplacement, k8sServiceHost/Port) are templated from k3s_api_host. # Pin inside the platform/cilium HelmRelease window (>=1.16.0 <1.17.0). -cilium_version: "1.16.5" +cilium_version: "1.17.18" cilium_chart_repo_name: "cilium" cilium_chart_repo_url: "https://helm.cilium.io/" cilium_namespace: "kube-system" diff --git a/platform/cilium/helmrelease.yaml b/platform/cilium/helmrelease.yaml index d18b2ba..05bf97f 100644 --- a/platform/cilium/helmrelease.yaml +++ b/platform/cilium/helmrelease.yaml @@ -27,7 +27,7 @@ spec: chart: cilium # [parity] Pinned to the exact version the Ansible bootstrap installs # (cilium_version in roles/cilium/defaults/main.yml). Renovate bumps both together. - version: "1.16.5" + version: "1.17.18" sourceRef: kind: HelmRepository name: cilium diff --git a/platform/longhorn/helmrelease.yaml b/platform/longhorn/helmrelease.yaml index c27091c..c60cd8b 100644 --- a/platform/longhorn/helmrelease.yaml +++ b/platform/longhorn/helmrelease.yaml @@ -14,7 +14,7 @@ spec: chart: spec: chart: longhorn - version: ">=1.7.0 <1.8.0" # TODO(WS-STORAGE): pin at bring-up; Renovate maintains + version: ">=1.8.0 <1.9.0" # W1 upgrade 1.7->1.8 (resolves 1.8.2; avoid buggy 1.8.0). Renovate maintains sourceRef: kind: HelmRepository name: longhorn