# 04 — Storage & State

Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md).

The core promise: **any service can be destroyed and rebuilt on a whim without losing data.** This
requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a
tested backup/restore path.

## Storage tiers

| Tier | Backing | Use | Replication |
|---|---|---|---|
| **App/config state** | Longhorn block volumes | PVCs for service config, databases, small app data | 3-way replica across nodes |
| **Bulk media** | Dedicated large store (see below) | Plex library, download staging | Not 3× replicated (capacity strategy) |
| **VM disks** | Longhorn (via KubeVirt/CDI) | KubeVirt application-VM root/data disks | 3-way replica |
| **Backups** | Off-cluster S3/NFS target | Velero + Longhorn backups for DR | Off-cluster, separate failure domain |

## Longhorn (primary persistent storage — R-STATE-02)

- Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC
  cluster with local storage and no SAN.
- Default StorageClass: **3 replicas** so any single node loss keeps data available.
- Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC
  and its data intact; rebuilding re-attaches it (R-STATE-01).
- Snapshots for point-in-time; **backups** (to off-cluster target) for DR.
- Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health
  and degraded replicas (alerts in [10](10-observability-and-alerting.md)).

## Separation of state from workload (the rebuild guarantee)

Every service's catalog entry declares its **persistent volumes** explicitly. The rule:

> If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — **never** in
> the container's ephemeral filesystem or on a single node's hostPath.

The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the
service comes back with its data. This is verified per service, not assumed.

## Bulk media strategy (R-STATE-04 — the Plex problem)

Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB
media across three mini-PCs is impractical** (capacity and cost). Per **OQ-2 (resolved)**, **bulk
media lives OUTSIDE the cluster**; the specific backing is **to be determined later** and does not
block the core platform.

- The bulk store is consumed by Plex and the arr stack as an **external mount** (NFS/SMB share to be
  specified when chosen). The `bulk` StorageClass / mount is a thin indirection so the concrete
  backing can be filled in later without touching service definitions.
- Downloads (qbittorrent) stage to the bulk store and hand off to the arr stack.

**Regardless of the bulk backing: Plex's config/metadata database is small and MUST live on 3-replica
Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store. Bulk
media is treated as **`reacquirable`** for backup purposes unless the operator specifies otherwise
when the backing is chosen.

## Backup & disaster recovery (R-STATE-03)

- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and
  orchestrates volume backups.
- **Longhorn backups** push volume data to an **off-cluster** S3-compatible or NFS target — a
  separate failure domain from the cluster itself.
- **Schedule**: regular automated backups (cadence per data criticality) with retention policy.
- **Restore is a tested procedure**, not a theoretical one: a DR drill (restore a namespace + its
  volumes to a clean state) is part of the platform acceptance suite.
- Bulk media has its own backup posture (e.g. it may be re-acquirable rather than backed up 1:1 —
  an explicit operator decision recorded per OQ-2), but **all irreplaceable state** (configs,
  databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster.

## KubeVirt VM disks

Application-VM disks are Longhorn volumes provisioned via **CDI** (Containerized Data Importer). VMs
inherit the same durability and backup story as container PVCs — a KubeVirt VM can be destroyed and
rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).

## GitOps + state = full rebuildability

- **Desired state** lives in git (manifests, service definitions).
- **Data** lives on Longhorn/bulk store + off-cluster backups.
- Therefore a total cluster loss is recoverable: re-provision nodes ([02](02-environments-and-provisioning.md)),
  re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end
  recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once in
  dev/test before prod is declared ready.

## Acceptance gates
- Delete and recreate each stateful service; data survives (per-service rebuild test).
- Kill a node (dev/test); replicated volumes stay available; cluster re-converges.
- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.

## Resolved
- **OQ-2** — bulk media lives **outside the cluster**; concrete backing **TBD later** (consumed as an
  external NFS/SMB mount via the `bulk` indirection). Does not block the core platform.

## Open questions
- **Off-cluster DR backup target** (separate from bulk media): which S3-compatible endpoint or NFS
  host holds Velero/Longhorn backups of the *irreplaceable* tier (app config, Nextcloud, Gitea, DBs),
  and its own resilience. Needed by **Phase 4**, not Phase 2.
- Concrete bulk-media backing + whether any of it is non-`reacquirable` (operator decides when chosen).