jakman85andClaude Opus 4.8
cbc2301b30
Text-only cleanup after the upgrade campaign. Updates stale version strings, prose, and comments to the verified current facts; no Kubernetes behavior or version pins changed except the kubectl toolchain bump (below). Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie, Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB, dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18, Grafana 13.2.0. - Version strings across infra/ and platform/ READMEs + comments - kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew) - Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh) - COMPLETED/historical banners on the three executed plan docs - Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges - Appended dated DECISIONS-LOG entry (2026-08-30) superseding the 2026-08-26 LIVE STATE line Left as revalidation items (not blind-edited): external-snapshotter v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1, conformance ImageVolume feature-gate assertions under 1.36. Co-Authored-By: Claude Opus 4.8 <[email protected]>
platform/longhorn
Longhorn — distributed replicated block storage across the 3 nodes.
- Governing ADR: ADR-0003 — Distributed storage: Longhorn (+ bulk tier + Velero)
- Spec: docs/04 — Storage & State
- Serves: R-STATE-01, R-STATE-02, R-STATE-04 (R-STATE-03 is Velero's, Phase 4)
- Owner: WS-STORAGE
What this is
A real, dev-appropriate Flux HelmRelease for Longhorn plus two StorageClasses:
longhorn— the cluster's only default StorageClass, 3 replicas with hard node anti-affinity (one replica per node). Durable home for app/config state, databases, and KubeVirt VM disks (via CDI). Replica count is the${LONGHORN_DEFAULT_REPLICA_COUNT}overlay knob (cluster-vars, =3 in dev).bulk— thin indirection for the external (outside-cluster) media tier: Plex library, download staging, arr handoff. STUB placeholder modelled as an external NFS mount; its backing is OQ-2 (unresolved) and will not provision until answered. Plex's metadata DB stays on the 3-replicalonghornclass regardless (docs/04, ADR-0003).
Requirement mapping
| Requirement | Summary | How this component serves it |
|---|---|---|
| R-STATE-01 | Stateful data survives destroy/rebuild of the workload | Volumes are Longhorn PVCs, independent of pod/VM lifecycle; reclaimPolicy: Delete on the class ties data to the PVC, not the workload. Destroy the pod/Deployment/VM → PVC + data persist → rebuild re-attaches. (Per-service rebuild test is the Phase-2 exit gate; T-1.5 proves node-level rebuild.) |
| R-STATE-02 | Persistent storage replicated across the 3 nodes | numberOfReplicas=3 on the default class + replicaSoftAntiAffinity: false (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the dedicated /var/lib/longhorn data disk. replicaAutoBalance: best-effort re-spreads on node return. |
| R-STATE-04 | Bulk media has a capacity strategy distinct from the replicated tier | Separate bulk StorageClass (NON-replicated, external NFS mount) keeps multi-TB media off the 3x tier; only Plex's small metadata DB uses longhorn. |
Dev-scoped choices (change for prod)
reclaimPolicy: Deleteonlonghorn— convenient in dev; prod may preferRetainfor accidental-delete protection (backups still cover DR — R-STATE-03).storageOverProvisioningPercentage: 100— no thin over-provisioning in dev (small disks); prod may raise it.storageReservedPercentageForDefaultDisk: 20/storageMinimalAvailablePercentage: 25— headroom guardrails tuned for small dev disks; revisit for prod disk sizes.- Replica count = 3 comes from
cluster-vars.longhorn_default_replica_count; both dev and prod are 3 today (3 nodes), so the value is identical — the knob exists so it is never forked per env.
Guardrails baked in (docs/04)
- Reserved disk headroom (never fill a node to 100%).
- Hard replica anti-affinity across nodes (no two replicas of a volume on one node).
- Single-zone anti-affinity relaxed (3 mini-PCs = one failure zone).
- Volume/disk health + degraded-replica alerting is WS-OBS's job (docs/10) — not configured here.
Node prerequisites (already handled, informational)
Each node presents a dedicated Longhorn data disk (/dev/vdb → ext4) mounted at
/var/lib/longhorn, and installs open-iscsi + nfs-common + kernel modules — all via
infra/cloud-init/user-data.yaml.tftpl. This component assumes that floor.
Contract (consumed by everyone with state)
Per roadmap/workstreams.md, WS-STORAGE exposes the longhorn (replicated) and bulk (media)
StorageClasses; WS-APPS/WS-VM/WS-OBS reference them from ServiceDefinitions. Do not place bulk
media on the replicated tier, or app DBs on the bulk tier.
Open decisions / TODO (later phases)
- OQ-2 (human): decide the
bulk-tier backing (NAS/NFS host + share, vs dedicated large disk). Until answered, thebulkclass is a non-functional placeholder (server/share= REPLACE-ME). Blocksbulkfinalize; does not block the core platform (docs/04). - Version pin (WS-STORAGE):
>=1.12.0 <1.13.0is a range; pin exactly at bring-up, Renovate maintains. - Phase 4 (WS-STORAGE): off-cluster backup target via
defaultSettings.backupTarget+ SOPS-encrypted credential secret, wired alongsideplatform/velero/(R-STATE-03).
Build check
kubectl kustomize platform/longhorn renders the namespace, HelmRepository, HelmRelease, and both
StorageClasses. (Structural only — not applied to a cluster here; the PM validates + deploys.)