home-network-k3s/docs/04-storage-and-state.md

04 — Storage & State

Serves: R-STATE-01..04, R-PLAT-04. Related: ADR-0003 (Longhorn).

The core promise: any service can be destroyed and rebuilt on a whim without losing data. This requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a tested backup/restore path.

Storage tiers

Tier	Backing	Use	Replication
App/config state	Longhorn block volumes	PVCs for service config, databases, small app data	3-way replica across nodes
Bulk media	Dedicated large store (see below)	Plex library, download staging	Not 3× replicated (capacity strategy)
VM disks	Longhorn (via KubeVirt/CDI)	KubeVirt application-VM root/data disks	3-way replica
Backups	Off-cluster S3/NFS target	Velero + Longhorn backups for DR	Off-cluster, separate failure domain

Longhorn (primary persistent storage — R-STATE-02)

• Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC cluster with local storage and no SAN.
• Default StorageClass: 3 replicas so any single node loss keeps data available.
• Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC and its data intact; rebuilding re-attaches it (R-STATE-01).
• Snapshots for point-in-time; backups (to off-cluster target) for DR.
• Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health and degraded replicas (alerts in 10).

Separation of state from workload (the rebuild guarantee)

Every service's catalog entry declares its persistent volumes explicitly. The rule:

If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — never in the container's ephemeral filesystem or on a single node's hostPath.

The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the service comes back with its data. This is verified per service, not assumed.

Bulk media strategy (R-STATE-04 — the Plex problem)

Plex is the largest service and its library can be multiple terabytes. 3× replicating multi-TB media across three mini-PCs is impractical (capacity and cost). Per OQ-2 (resolved), bulk media lives OUTSIDE the cluster; the specific backing is to be determined later and does not block the core platform.

• The bulk store is consumed by Plex and the arr stack as an external mount (NFS/SMB share to be specified when chosen). The bulk StorageClass / mount is a thin indirection so the concrete backing can be filled in later without touching service definitions.
• Downloads (qbittorrent) stage to the bulk store and hand off to the arr stack.

Regardless of the bulk backing: Plex's config/metadata database is small and MUST live on 3-replica Longhorn (so Plex rebuilds cleanly); only the large media files use the external bulk store. Bulk media is treated as reacquirable for backup purposes unless the operator specifies otherwise when the backing is chosen.

Backup & disaster recovery (R-STATE-03)

• Velero backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and orchestrates volume backups.
• Longhorn backups push volume data to an off-cluster S3-compatible or NFS target — a separate failure domain from the cluster itself.
• Schedule: regular automated backups (cadence per data criticality) with retention policy.
• Restore is a tested procedure, not a theoretical one: a DR drill (restore a namespace + its volumes to a clean state) is part of the platform acceptance suite.
• Bulk media has its own backup posture (e.g. it may be re-acquirable rather than backed up 1:1 — an explicit operator decision recorded per OQ-2), but all irreplaceable state (configs, databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster.

KubeVirt VM disks

Application-VM disks are Longhorn volumes provisioned via CDI (Containerized Data Importer). VMs inherit the same durability and backup story as container PVCs — a KubeVirt VM can be destroyed and rebuilt from its declarative definition + persistent disk. See 06-vm-workloads-kubevirt.md.

GitOps + state = full rebuildability

• Desired state lives in git (manifests, service definitions).
• Data lives on Longhorn/bulk store + off-cluster backups.
• Therefore a total cluster loss is recoverable: re-provision nodes (02), re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once in dev/test before prod is declared ready.

Acceptance gates

• Delete and recreate each stateful service; data survives (per-service rebuild test).
• Kill a node (dev/test); replicated volumes stay available; cluster re-converges.
• Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
• Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.

Resolved

• OQ-2 — bulk media lives outside the cluster; concrete backing TBD later (consumed as an external NFS/SMB mount via the bulk indirection). Does not block the core platform.

Open questions

• Off-cluster DR backup target (separate from bulk media): which S3-compatible endpoint or NFS host holds Velero/Longhorn backups of the irreplaceable tier (app config, Nextcloud, Gitea, DBs), and its own resilience. Needed by Phase 4, not Phase 2.
• Concrete bulk-media backing + whether any of it is non-reacquirable (operator decides when chosen).