jakman85andClaude Opus 4.8
7a3da96d9f
The UniFi Block import got past the device-permission fix but then failed ReplicaSchedulingFailure / "insufficient storage": CDI sizes scratch ~= the disk virtual size and, on the default 3-replica `longhorn` class, the ~60Gi scratch x3 (189G) on top of the disk's x3 (180G) overflowed the Longhorn tier (~125G/node). - platform/longhorn/storageclasses.yaml: NEW `longhorn-scratch` StorageClass (numberOfReplicas: 1) — CDI scratch is ephemeral/throwaway, replicating it 3x is waste; if a node dies mid-import CDI just retries. - platform/kubevirt/cdi-cr.yaml: spec.config.scratchSpaceStorageClass -> longhorn-scratch. - apps/vms/unifi/20-datavolume.yaml: boot disk 60Gi -> 40Gi (UOS min is 25Gi; 40Gi is comfortable for a home site and lighter on the replicated tier). Co-Authored-By: Claude Opus 4.8 <[email protected]>
platform/longhorn
Longhorn — distributed replicated block storage across the 3 nodes.
- Governing ADR: ADR-0003 — Distributed storage: Longhorn (+ bulk tier + Velero)
- Spec: docs/04 — Storage & State
- Serves: R-STATE-01, R-STATE-02, R-STATE-04 (R-STATE-03 is Velero's, Phase 4)
- Owner: WS-STORAGE
What this is
A real, dev-appropriate Flux HelmRelease for Longhorn plus two StorageClasses:
longhorn— the cluster's only default StorageClass, 3 replicas with hard node anti-affinity (one replica per node). Durable home for app/config state, databases, and KubeVirt VM disks (via CDI). Replica count is the${LONGHORN_DEFAULT_REPLICA_COUNT}overlay knob (cluster-vars, =3 in dev).bulk— thin indirection for the external (outside-cluster) media tier: Plex library, download staging, arr handoff. STUB placeholder modelled as an external NFS mount; its backing is OQ-2 (unresolved) and will not provision until answered. Plex's metadata DB stays on the 3-replicalonghornclass regardless (docs/04, ADR-0003).
Requirement mapping
| Requirement | Summary | How this component serves it |
|---|---|---|
| R-STATE-01 | Stateful data survives destroy/rebuild of the workload | Volumes are Longhorn PVCs, independent of pod/VM lifecycle; reclaimPolicy: Delete on the class ties data to the PVC, not the workload. Destroy the pod/Deployment/VM → PVC + data persist → rebuild re-attaches. (Per-service rebuild test is the Phase-2 exit gate; T-1.5 proves node-level rebuild.) |
| R-STATE-02 | Persistent storage replicated across the 3 nodes | numberOfReplicas=3 on the default class + replicaSoftAntiAffinity: false (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the dedicated /var/lib/longhorn data disk. replicaAutoBalance: best-effort re-spreads on node return. |
| R-STATE-04 | Bulk media has a capacity strategy distinct from the replicated tier | Separate bulk StorageClass (NON-replicated, external NFS mount) keeps multi-TB media off the 3x tier; only Plex's small metadata DB uses longhorn. |
Dev-scoped choices (change for prod)
reclaimPolicy: Deleteonlonghorn— convenient in dev; prod may preferRetainfor accidental-delete protection (backups still cover DR — R-STATE-03).storageOverProvisioningPercentage: 100— no thin over-provisioning in dev (small disks); prod may raise it.storageReservedPercentageForDefaultDisk: 20/storageMinimalAvailablePercentage: 25— headroom guardrails tuned for small dev disks; revisit for prod disk sizes.- Replica count = 3 comes from
cluster-vars.longhorn_default_replica_count; both dev and prod are 3 today (3 nodes), so the value is identical — the knob exists so it is never forked per env.
Guardrails baked in (docs/04)
- Reserved disk headroom (never fill a node to 100%).
- Hard replica anti-affinity across nodes (no two replicas of a volume on one node).
- Single-zone anti-affinity relaxed (3 mini-PCs = one failure zone).
- Volume/disk health + degraded-replica alerting is WS-OBS's job (docs/10) — not configured here.
Node prerequisites (already handled, informational)
Each node presents a dedicated Longhorn data disk (/dev/vdb → ext4) mounted at
/var/lib/longhorn, and installs open-iscsi + nfs-common + kernel modules — all via
infra/cloud-init/user-data.yaml.tftpl. This component assumes that floor.
Contract (consumed by everyone with state)
Per roadmap/workstreams.md, WS-STORAGE exposes the longhorn (replicated) and bulk (media)
StorageClasses; WS-APPS/WS-VM/WS-OBS reference them from ServiceDefinitions. Do not place bulk
media on the replicated tier, or app DBs on the bulk tier.
Open decisions / TODO (later phases)
- OQ-2 (human): decide the
bulk-tier backing (NAS/NFS host + share, vs dedicated large disk). Until answered, thebulkclass is a non-functional placeholder (server/share= REPLACE-ME). Blocksbulkfinalize; does not block the core platform (docs/04). - Version pin (WS-STORAGE):
>=1.7.0 <1.8.0is a range; pin exactly at bring-up, Renovate maintains. - Phase 4 (WS-STORAGE): off-cluster backup target via
defaultSettings.backupTarget+ SOPS-encrypted credential secret, wired alongsideplatform/velero/(R-STATE-03).
Build check
kubectl kustomize platform/longhorn renders the namespace, HelmRepository, HelmRelease, and both
StorageClasses. (Structural only — not applied to a cluster here; the PM validates + deploys.)