home-network-k3s/docs/04-storage-and-state.md

# 04 — Storage & State

Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md).

The core promise: **any service can be destroyed and rebuilt on a whim without losing data.** This
requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a
tested backup/restore path.

## Storage tiers

| Tier | Backing | Use | Replication |
|---|---|---|---|
| **App/config state** | Longhorn block volumes | PVCs for service config, databases, small app data | 3-way replica across nodes |
| **Bulk media** | Dedicated large store (see below) | Plex library, download staging | Not 3× replicated (capacity strategy) |
| **VM disks** | Longhorn (via KubeVirt/CDI) | KubeVirt application-VM root/data disks | 3-way replica |
| **Backups** | Off-cluster S3/NFS target | Velero + Longhorn backups for DR | Off-cluster, separate failure domain |

## Longhorn (primary persistent storage — R-STATE-02)

- Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC
  cluster with local storage and no SAN.
- Default StorageClass: **3 replicas** so any single node loss keeps data available.
- Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC
  and its data intact; rebuilding re-attaches it (R-STATE-01).
- Snapshots for point-in-time; **backups** (to off-cluster target) for DR.
- Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health
  and degraded replicas (alerts in [10](10-observability-and-alerting.md)).

## Separation of state from workload (the rebuild guarantee)

Every service's catalog entry declares its **persistent volumes** explicitly. The rule:

> If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — **never** in
> the container's ephemeral filesystem or on a single node's hostPath.

The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the
service comes back with its data. This is verified per service, not assumed.

## Bulk media strategy (R-STATE-04 — the Plex problem)

Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB
media across three mini-PCs is impractical** (capacity and cost). Therefore bulk media is a
**distinct tier** from replicated app-config storage. Options to be pinned by ADR-0003 / OQ-2:

- **NAS / external NFS** exporting the media library, mounted into Plex (and the arr stack) — keeps
  bulk data off the replicated tier; simplest capacity story; introduces an external dependency.
- **Dedicated large disk** on one node with **single-replica Longhorn** or a local PV — cheaper, but
  node-bound; requires its own backup plan.
- **Object storage** for some content — generally poor fit for Plex direct play.

Regardless of choice: **Plex's config/metadata database is small and MUST live on 3-replica
Longhorn** (so Plex rebuilds cleanly); only the large media files use the bulk tier. Downloads
(qbittorrent) stage to the bulk tier and hand off to the arr stack.

## Backup & disaster recovery (R-STATE-03)

- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and
  orchestrates volume backups.
- **Longhorn backups** push volume data to an **off-cluster** S3-compatible or NFS target — a
  separate failure domain from the cluster itself.
- **Schedule**: regular automated backups (cadence per data criticality) with retention policy.
- **Restore is a tested procedure**, not a theoretical one: a DR drill (restore a namespace + its
  volumes to a clean state) is part of the platform acceptance suite.
- Bulk media has its own backup posture (e.g. it may be re-acquirable rather than backed up 1:1 —
  an explicit operator decision recorded per OQ-2), but **all irreplaceable state** (configs,
  databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster.

## KubeVirt VM disks

Application-VM disks are Longhorn volumes provisioned via **CDI** (Containerized Data Importer). VMs
inherit the same durability and backup story as container PVCs — a KubeVirt VM can be destroyed and
rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).

## GitOps + state = full rebuildability

- **Desired state** lives in git (manifests, service definitions).
- **Data** lives on Longhorn/bulk store + off-cluster backups.
- Therefore a total cluster loss is recoverable: re-provision nodes ([02](02-environments-and-provisioning.md)),
  re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end
  recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once in
  dev/test before prod is declared ready.

## Acceptance gates
- Delete and recreate each stateful service; data survives (per-service rebuild test).
- Kill a node (dev/test); replicated volumes stay available; cluster re-converges.
- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.

## Open questions
- **OQ-2** — final bulk-media home (NAS/NFS vs dedicated disk vs mixed) and its backup posture.
- Off-cluster backup **target**: which S3-compatible endpoint or NFS host, and its own resilience.