jakman85andClaude Opus 4.8
e8c555cfeb
Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision & requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap (phases + workstreams), and AI-agent coordination (roles, protocol, definition-of-done, task board). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
5.3 KiB
04 — Storage & State
Serves: R-STATE-01..04, R-PLAT-04. Related: ADR-0003 (Longhorn).
The core promise: any service can be destroyed and rebuilt on a whim without losing data. This requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a tested backup/restore path.
Storage tiers
| Tier | Backing | Use | Replication |
|---|---|---|---|
| App/config state | Longhorn block volumes | PVCs for service config, databases, small app data | 3-way replica across nodes |
| Bulk media | Dedicated large store (see below) | Plex library, download staging | Not 3× replicated (capacity strategy) |
| VM disks | Longhorn (via KubeVirt/CDI) | KubeVirt application-VM root/data disks | 3-way replica |
| Backups | Off-cluster S3/NFS target | Velero + Longhorn backups for DR | Off-cluster, separate failure domain |
Longhorn (primary persistent storage — R-STATE-02)
- Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC cluster with local storage and no SAN.
- Default StorageClass: 3 replicas so any single node loss keeps data available.
- Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC and its data intact; rebuilding re-attaches it (R-STATE-01).
- Snapshots for point-in-time; backups (to off-cluster target) for DR.
- Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health and degraded replicas (alerts in 10).
Separation of state from workload (the rebuild guarantee)
Every service's catalog entry declares its persistent volumes explicitly. The rule:
If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — never in the container's ephemeral filesystem or on a single node's hostPath.
The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the service comes back with its data. This is verified per service, not assumed.
Bulk media strategy (R-STATE-04 — the Plex problem)
Plex is the largest service and its library can be multiple terabytes. 3× replicating multi-TB media across three mini-PCs is impractical (capacity and cost). Therefore bulk media is a distinct tier from replicated app-config storage. Options to be pinned by ADR-0003 / OQ-2:
- NAS / external NFS exporting the media library, mounted into Plex (and the arr stack) — keeps bulk data off the replicated tier; simplest capacity story; introduces an external dependency.
- Dedicated large disk on one node with single-replica Longhorn or a local PV — cheaper, but node-bound; requires its own backup plan.
- Object storage for some content — generally poor fit for Plex direct play.
Regardless of choice: Plex's config/metadata database is small and MUST live on 3-replica Longhorn (so Plex rebuilds cleanly); only the large media files use the bulk tier. Downloads (qbittorrent) stage to the bulk tier and hand off to the arr stack.
Backup & disaster recovery (R-STATE-03)
- Velero backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and orchestrates volume backups.
- Longhorn backups push volume data to an off-cluster S3-compatible or NFS target — a separate failure domain from the cluster itself.
- Schedule: regular automated backups (cadence per data criticality) with retention policy.
- Restore is a tested procedure, not a theoretical one: a DR drill (restore a namespace + its volumes to a clean state) is part of the platform acceptance suite.
- Bulk media has its own backup posture (e.g. it may be re-acquirable rather than backed up 1:1 — an explicit operator decision recorded per OQ-2), but all irreplaceable state (configs, databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster.
KubeVirt VM disks
Application-VM disks are Longhorn volumes provisioned via CDI (Containerized Data Importer). VMs
inherit the same durability and backup story as container PVCs — a KubeVirt VM can be destroyed and
rebuilt from its declarative definition + persistent disk. See 06-vm-workloads-kubevirt.md.
GitOps + state = full rebuildability
- Desired state lives in git (manifests, service definitions).
- Data lives on Longhorn/bulk store + off-cluster backups.
- Therefore a total cluster loss is recoverable: re-provision nodes (02), re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once in dev/test before prod is declared ready.
Acceptance gates
- Delete and recreate each stateful service; data survives (per-service rebuild test).
- Kill a node (dev/test); replicated volumes stay available; cluster re-converges.
- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.
Open questions
- OQ-2 — final bulk-media home (NAS/NFS vs dedicated disk vs mixed) and its backup posture.
- Off-cluster backup target: which S3-compatible endpoint or NFS host, and its own resilience.