home-network-k3s/docs/04-storage-and-state.md

04 — Storage & State

Serves: R-STATE-01..04, R-PLAT-04. Related: ADR-0003 (Longhorn), ADR-0018 (bulk tier + backup), ADR-0019 (PostgreSQL via CloudNativePG).

The core promise: any service can be destroyed and rebuilt on a whim without losing data. This requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a tested backup/restore path.

Storage tiers

Tier	Backing	Use	Replication
App/config state	Longhorn block volumes	PVCs for service config, embedded databases, small app data	3-way replica across nodes
Databases	CloudNativePG clusters on Longhorn	PostgreSQL for Keycloak, Nextcloud, Semaphore	3 instances, each on a 3-way replicated volume
Bulk media	Off-cluster NAS, consumed over NFS	Plex library, download staging, read-only legacy Nextcloud archive	Not 3× replicated (capacity strategy)
VM disks	Longhorn (via KubeVirt/CDI)	KubeVirt application-VM root/data disks	3-way replica
Backups	Cloudflare R2 (S3-compatible, off-site)	Velero + kopia volume data; CloudNativePG WAL and base backups	Off-cluster, separate failure domain

Longhorn (primary persistent storage — R-STATE-02)

• Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC cluster with local storage and no SAN.
• Default StorageClass: 3 replicas so any single node loss keeps data available.
• Two further classes serve KubeVirt: a live-migratable block class (3 replicas, the default for VM disks) and a single-replica class for CDI's throwaway import scratch, the one non-replicated use of Longhorn.
• Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC and its data intact; rebuilding re-attaches it (R-STATE-01).
• Snapshots are available on demand and are how VM disks clone; backups for DR go off-cluster through Velero (below), not through a Longhorn backup target.
• Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health and degraded replicas (alerts in 10), and cap Longhorn's write concurrency so replica I/O cannot starve etcd's WAL.
• Accepted risk: each node has one data device, so etcd's WAL and every Longhorn replica share it. The concurrency caps reduce that contention and do not remove it. The exit condition is a dedicated etcd device per node (see known gaps).

Database state (PostgreSQL)

Services that need PostgreSQL get a CloudNativePG cluster per database (ADR-0019): three instances on three different nodes, each on replicated Longhorn. Longhorn replication keeps the data through a node loss; a streaming standby keeps the service, because promotion takes seconds. Standbys replicate corruption faithfully, so neither is a backup: WAL is archived continuously and base backups run daily to the off-cluster target, giving point-in-time recovery.

Services whose datastore is embedded (Gitea's and Crafty's SQLite, for example) keep it on their Longhorn PVC and are covered by the ordinary volume backup.

Separation of state from workload (the rebuild guarantee)

Every service's catalog entry declares its persistent volumes explicitly. The rule:

If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — never in the container's ephemeral filesystem or on a single node's hostPath.

A volume counts only when it is mounted at the path the application actually writes. A PVC mounted anywhere else binds, reports healthy and backs up successfully while the real state sits on the ephemeral layer, so a service's mount path is declared deliberately rather than left to a default.

The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the service comes back with its data. This is verified per service, not assumed.

Bulk media strategy (R-STATE-04 — the Plex problem)

Plex is the largest service and its library is multiple terabytes. 3× replicating multi-TB media across three mini-PCs is impractical (capacity and cost). Per OQ-2 (resolved), bulk media lives OUTSIDE the cluster on a standalone TrueNAS appliance: storage only, ZFS-backed, and on the same LAN as the nodes (ADR-0018).

• The bulk store is consumed by Plex, the arr stack and qbittorrent as an external NFS mount. Nextcloud mounts a separate legacy archive from it read-only.
• A catalog entry's class: bulk plus a named export is a thin indirection: it attaches an existing dataset as a statically bound volume, and the NFS server is a per-environment value, so service definitions never name the storage host. Nothing is dynamically provisioned, because every bulk volume attaches data that already exists.
• Downloads (qbittorrent) stage to the bulk store and hand off to the arr stack.
• Layout, export settings and the uid model are in docs/BULK-NFS-STORAGE.md in the implementation repo.

Regardless of the bulk backing: Plex's config/metadata database is small and MUST live on 3-replica Longhorn (so Plex rebuilds cleanly); only the large media files use the external bulk store.

The bulk tier is outside platform backup. Velero skips NFS volumes by volume type, so the exclusion also covers any future export, while the same namespaces' Longhorn PVCs are still backed up. A service's backup.policy therefore speaks for its Longhorn volumes only, and the bulk data's durability is the NAS's own (see known gaps).

Backup & disaster recovery (R-STATE-03)

• The off-cluster target is Cloudflare R2: S3-compatible, off-site and a separate failure domain from the cluster and from the NAS. Backup tools share one bucket under disjoint prefixes, so one tool's retention can never reap another's objects.
• Velero backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and, through kopia file-system backup, the data of every volume. Backup is opt-out, so a new volume is covered without anyone remembering to enrol it.
• Longhorn has no backup target (ADR-0018). Velero and kopia are the single restore path for volume data, an accepted trade-off.
• CloudNativePG archives WAL and base backups to the same target, independently of Velero.
• Schedule: regular automated backups with retention policy. Databases archive continuously; everything else runs nightly.
• Exclusions are limited to reconstructable data, whose Kubernetes resources are still captured: monitoring's metrics and log volumes, and the UniFi VM's disk, for which the design relies on UniFi's own configuration export (06).
• Restore is a tested procedure, not a theoretical one: a DR drill (restore a namespace + its volumes to a clean state) is part of the platform acceptance suite.
• Bulk media is outside this posture (above), but all irreplaceable state (configs, databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster.

KubeVirt VM disks

Application-VM disks are Longhorn volumes provisioned via CDI (Containerized Data Importer). VMs share the durability of container PVCs and are backed up by default, with per-VM exceptions recorded in 06 — a KubeVirt VM can be destroyed and rebuilt from its declarative definition + persistent disk. See 06-vm-workloads-kubevirt.md.

GitOps + state = full rebuildability

• Desired state lives in git (manifests, service definitions).
• Data lives on Longhorn/bulk store + off-cluster backups.
• Therefore a total cluster loss is recoverable: re-provision nodes (02), re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once, in a planned maintenance window.

Acceptance gates

• Delete and recreate each stateful service; data survives (per-service rebuild test).
• Kill a node in a planned maintenance window; replicated volumes stay available; cluster re-converges.
• Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
• Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.

Resolved

• OQ-2 — bulk media lives outside the cluster, on a TrueNAS appliance consumed over NFS through the catalog's class: bulk indirection (ADR-0018).
• OQ-BKP — the off-cluster DR target for the irreplaceable tier is Cloudflare R2 (ADR-0018).

Open questions

• Whether any bulk data, notably the photo library, is non-reacquirable and needs its own backup (operator decides).