home-network-k3s/docs/04-storage-and-state.md

# 04 — Storage & State

Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md),
[ADR-0018 (bulk tier + backup)](../decisions/adr-0018-storage-backup-truenas-r2.md),
[ADR-0019 (PostgreSQL via CloudNativePG)](../decisions/adr-0019-postgresql-cloudnativepg.md).

The core promise: **any service can be destroyed and rebuilt on a whim without losing data.** This
requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a
tested backup/restore path.

## Storage tiers

| Tier | Backing | Use | Replication |
|---|---|---|---|
| **App/config state** | Longhorn block volumes | PVCs for service config, embedded databases, small app data | 3-way replica across nodes |
| **Databases** | CloudNativePG clusters on Longhorn | PostgreSQL for Keycloak, Nextcloud, Semaphore | 3 instances, each on a 3-way replicated volume |
| **Bulk media** | Off-cluster NAS, consumed over NFS | Plex library, download staging, read-only legacy Nextcloud archive | Not 3× replicated (capacity strategy) |
| **VM disks** | Longhorn (via KubeVirt/CDI) | KubeVirt application-VM root/data disks | 3-way replica |
| **Backups** | Cloudflare R2 (S3-compatible, off-site) | Velero + kopia volume data; CloudNativePG WAL and base backups | Off-cluster, separate failure domain |

## Longhorn (primary persistent storage — R-STATE-02)

- Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC
  cluster with local storage and no SAN.
- Default StorageClass: **3 replicas** so any single node loss keeps data available.
- Two further classes serve KubeVirt: a live-migratable block class (3 replicas, the default for VM
  disks) and a single-replica class for CDI's throwaway import scratch, the one non-replicated use of
  Longhorn.
- Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC
  and its data intact; rebuilding re-attaches it (R-STATE-01).
- Snapshots are available on demand and are how VM disks clone; **backups** for DR go off-cluster
  through Velero (below), not through a Longhorn backup target.
- Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health
  and degraded replicas (alerts in [10](10-observability-and-alerting.md)), and cap Longhorn's write
  concurrency so replica I/O cannot starve etcd's WAL.
- **Accepted risk:** each node has one data device, so etcd's WAL and every Longhorn replica share it.
  The concurrency caps reduce that contention and do not remove it. The exit condition is a dedicated
  etcd device per node (see [known gaps](16-known-gaps.md)).

## Database state (PostgreSQL)

Services that need PostgreSQL get a **CloudNativePG** cluster per database ([ADR-0019](../decisions/adr-0019-postgresql-cloudnativepg.md)):
three instances on three different nodes, each on replicated Longhorn. Longhorn replication keeps the
*data* through a node loss; a streaming standby keeps the *service*, because promotion takes seconds.
Standbys replicate corruption faithfully, so neither is a backup: WAL is archived continuously and
base backups run daily to the off-cluster target, giving point-in-time recovery.

Services whose datastore is embedded (Gitea's and Crafty's SQLite, for example) keep it on their
Longhorn PVC and are covered by the ordinary volume backup.

## Separation of state from workload (the rebuild guarantee)

Every service's catalog entry declares its **persistent volumes** explicitly. The rule:

> If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — **never** in
> the container's ephemeral filesystem or on a single node's hostPath.

A volume counts only when it is mounted at the path the application actually writes. A PVC mounted
anywhere else binds, reports healthy and backs up successfully while the real state sits on the
ephemeral layer, so a service's mount path is declared deliberately rather than left to a default.

The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the
service comes back with its data. This is verified per service, not assumed.

## Bulk media strategy (R-STATE-04 — the Plex problem)

Plex is the largest service and its library is multiple terabytes. **3× replicating multi-TB
media across three mini-PCs is impractical** (capacity and cost). Per **OQ-2 (resolved)**, **bulk
media lives OUTSIDE the cluster** on a standalone TrueNAS appliance: storage only, ZFS-backed, and on
the same LAN as the nodes ([ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md)).

- The bulk store is consumed by Plex, the arr stack and qbittorrent as an **external NFS mount**.
  Nextcloud mounts a separate legacy archive from it read-only.
- A catalog entry's `class: bulk` plus a named export is a thin indirection: it attaches an existing
  dataset as a statically bound volume, and the NFS server is a per-environment value, so service
  definitions never name the storage host. Nothing is dynamically provisioned, because every bulk
  volume attaches data that already exists.
- Downloads (qbittorrent) stage to the bulk store and hand off to the arr stack.
- Layout, export settings and the uid model are in `docs/BULK-NFS-STORAGE.md` in the
  [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).

**Regardless of the bulk backing: Plex's config/metadata database is small and MUST live on 3-replica
Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store.

**The bulk tier is outside platform backup.** Velero skips NFS volumes by volume type, so the exclusion
also covers any future export, while the same namespaces' Longhorn PVCs are still backed up. A
service's `backup.policy` therefore speaks for its Longhorn volumes only, and the bulk data's
durability is the NAS's own (see [known gaps](16-known-gaps.md)).

## Backup & disaster recovery (R-STATE-03)

- The **off-cluster target is Cloudflare R2**: S3-compatible, off-site and a separate failure domain
  from the cluster and from the NAS. Backup tools share one bucket under disjoint prefixes, so one
  tool's retention can never reap another's objects.
- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and, through
  **kopia file-system backup**, the data of every volume. Backup is opt-out, so a new volume is
  covered without anyone remembering to enrol it.
- **Longhorn has no backup target** ([ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md)).
  Velero and kopia are the single restore path for volume data, an accepted trade-off.
- **CloudNativePG** archives WAL and base backups to the same target, independently of Velero.
- **Schedule**: regular automated backups with retention policy. Databases archive continuously;
  everything else runs nightly.
- **Exclusions are limited to reconstructable data**, whose Kubernetes resources are still captured:
  monitoring's metrics and log volumes, and the UniFi VM's disk, for which the design relies on
  UniFi's own configuration export ([06](06-vm-workloads-kubevirt.md)).
- **Restore is a tested procedure**, not a theoretical one: a DR drill (restore a namespace + its
  volumes to a clean state) is part of the platform acceptance suite.
- Bulk media is outside this posture (above), but **all irreplaceable state** (configs,
  databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster.

## KubeVirt VM disks

Application-VM disks are Longhorn volumes provisioned via **CDI** (Containerized Data Importer). VMs
share the durability of container PVCs and are backed up by default, with per-VM exceptions recorded
in [06](06-vm-workloads-kubevirt.md) — a KubeVirt VM can be destroyed and rebuilt from its declarative
definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).

## GitOps + state = full rebuildability

- **Desired state** lives in git (manifests, service definitions).
- **Data** lives on Longhorn/bulk store + off-cluster backups.
- Therefore a total cluster loss is recoverable: re-provision nodes ([02](02-environments-and-provisioning.md)),
  re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end
  recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once, in a
  planned maintenance window.

## Acceptance gates
- Delete and recreate each stateful service; data survives (per-service rebuild test).
- Kill a node in a planned maintenance window; replicated volumes stay available; cluster re-converges.
- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.

## Resolved
- **OQ-2** — bulk media lives **outside the cluster**, on a TrueNAS appliance consumed over NFS through
  the catalog's `class: bulk` indirection (ADR-0018).
- **OQ-BKP** — the off-cluster DR target for the *irreplaceable* tier is **Cloudflare R2** (ADR-0018).

## Open questions
- Whether any bulk data, notably the photo library, is non-`reacquirable` and needs its own backup
  (operator decides).