Initial baseline: home platform (k3s + Longhorn) specification repository

Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision &
requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap
(phases + workstreams), and AI-agent coordination (roles, protocol,
definition-of-done, task board).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

docs/04-storage-and-state.md

@@ -0,0 +1,91 @@
+# 04 — Storage & State
+
+Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md).
+
+The core promise: **any service can be destroyed and rebuilt on a whim without losing data.** This
+requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a
+tested backup/restore path.
+
+## Storage tiers
+
+| Tier | Backing | Use | Replication |
+|---|---|---|---|
+| **App/config state** | Longhorn block volumes | PVCs for service config, databases, small app data | 3-way replica across nodes |
+| **Bulk media** | Dedicated large store (see below) | Plex library, download staging | Not 3× replicated (capacity strategy) |
+| **VM disks** | Longhorn (via KubeVirt/CDI) | KubeVirt application-VM root/data disks | 3-way replica |
+| **Backups** | Off-cluster S3/NFS target | Velero + Longhorn backups for DR | Off-cluster, separate failure domain |
+
+## Longhorn (primary persistent storage — R-STATE-02)
+
+- Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC
+  cluster with local storage and no SAN.
+- Default StorageClass: **3 replicas** so any single node loss keeps data available.
+- Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC
+  and its data intact; rebuilding re-attaches it (R-STATE-01).
+- Snapshots for point-in-time; **backups** (to off-cluster target) for DR.
+- Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health
+  and degraded replicas (alerts in [10](10-observability-and-alerting.md)).
+
+## Separation of state from workload (the rebuild guarantee)
+
+Every service's catalog entry declares its **persistent volumes** explicitly. The rule:
+
+> If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — **never** in
+> the container's ephemeral filesystem or on a single node's hostPath.
+
+The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the
+service comes back with its data. This is verified per service, not assumed.
+
+## Bulk media strategy (R-STATE-04 — the Plex problem)
+
+Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB
+media across three mini-PCs is impractical** (capacity and cost). Therefore bulk media is a
+**distinct tier** from replicated app-config storage. Options to be pinned by ADR-0003 / OQ-2:
+
+- **NAS / external NFS** exporting the media library, mounted into Plex (and the arr stack) — keeps
+  bulk data off the replicated tier; simplest capacity story; introduces an external dependency.
+- **Dedicated large disk** on one node with **single-replica Longhorn** or a local PV — cheaper, but
+  node-bound; requires its own backup plan.
+- **Object storage** for some content — generally poor fit for Plex direct play.
+
+Regardless of choice: **Plex's config/metadata database is small and MUST live on 3-replica
+Longhorn** (so Plex rebuilds cleanly); only the large media files use the bulk tier. Downloads
+(qbittorrent) stage to the bulk tier and hand off to the arr stack.
+
+## Backup & disaster recovery (R-STATE-03)
+
+- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and
+  orchestrates volume backups.
+- **Longhorn backups** push volume data to an **off-cluster** S3-compatible or NFS target — a
+  separate failure domain from the cluster itself.
+- **Schedule**: regular automated backups (cadence per data criticality) with retention policy.
+- **Restore is a tested procedure**, not a theoretical one: a DR drill (restore a namespace + its
+  volumes to a clean state) is part of the platform acceptance suite.
+- Bulk media has its own backup posture (e.g. it may be re-acquirable rather than backed up 1:1 —
+  an explicit operator decision recorded per OQ-2), but **all irreplaceable state** (configs,
+  databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster.
+
+## KubeVirt VM disks
+
+Application-VM disks are Longhorn volumes provisioned via **CDI** (Containerized Data Importer). VMs
+inherit the same durability and backup story as container PVCs — a KubeVirt VM can be destroyed and
+rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
+
+## GitOps + state = full rebuildability
+
+- **Desired state** lives in git (manifests, service definitions).
+- **Data** lives on Longhorn/bulk store + off-cluster backups.
+- Therefore a total cluster loss is recoverable: re-provision nodes ([02](02-environments-and-provisioning.md)),
+  re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end
+  recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once in
+  dev/test before prod is declared ready.
+
+## Acceptance gates
+- Delete and recreate each stateful service; data survives (per-service rebuild test).
+- Kill a node (dev/test); replicated volumes stay available; cluster re-converges.
+- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
+- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.
+
+## Open questions
+- **OQ-2** — final bulk-media home (NAS/NFS vs dedicated disk vs mixed) and its backup posture.
+- Off-cluster backup **target**: which S3-compatible endpoint or NFS host, and its own resilience.