jakman85andClaude Opus 4.8
a2cbdf656d
Operator answers folded into the specs: - OQ-3 VPN: ExpressVPN, cluster-controlled SINGLE shared in-cluster tunnel (OpenVPN UDP / non-TCP), rules enforced outside the service. New ADR-0012 supersedes ADR-0008 (Gluetun-sidecar default dropped). Central kill-switch. - OQ-1 WAN/edge: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). Multi-WAN out of scope; segmentation + VPN in-cluster; edge = NAT + manual port-forward. Gateway VIP resolved to Cilium L2 announcement. T-3.3 marked won't-do. - OQ-2 storage: bulk media lives outside the cluster (backing TBD, non-blocking) via a 'bulk' external-mount indirection; Plex metadata DB stays on Longhorn. Split out a separate OQ-BKP for the off-cluster DR backup target (Phase 4). - OQ-4 refined: wildcard record covers reachability but NOT ACME DNS-01 (needs TXT-write access) and does not match *.home.* depth. Still OPEN: Cloudflare API token (Option A) vs acme-dns delegation (Option B). T-2.4/T-2.5 blocked. Updated: docs 00,01,03,04,08,09; ADR-0008/0012 + index; task board. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
99 lines
5.7 KiB
Markdown
99 lines
5.7 KiB
Markdown
# 04 — Storage & State
|
||
|
||
Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md).
|
||
|
||
The core promise: **any service can be destroyed and rebuilt on a whim without losing data.** This
|
||
requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a
|
||
tested backup/restore path.
|
||
|
||
## Storage tiers
|
||
|
||
| Tier | Backing | Use | Replication |
|
||
|---|---|---|---|
|
||
| **App/config state** | Longhorn block volumes | PVCs for service config, databases, small app data | 3-way replica across nodes |
|
||
| **Bulk media** | Dedicated large store (see below) | Plex library, download staging | Not 3× replicated (capacity strategy) |
|
||
| **VM disks** | Longhorn (via KubeVirt/CDI) | KubeVirt application-VM root/data disks | 3-way replica |
|
||
| **Backups** | Off-cluster S3/NFS target | Velero + Longhorn backups for DR | Off-cluster, separate failure domain |
|
||
|
||
## Longhorn (primary persistent storage — R-STATE-02)
|
||
|
||
- Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC
|
||
cluster with local storage and no SAN.
|
||
- Default StorageClass: **3 replicas** so any single node loss keeps data available.
|
||
- Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC
|
||
and its data intact; rebuilding re-attaches it (R-STATE-01).
|
||
- Snapshots for point-in-time; **backups** (to off-cluster target) for DR.
|
||
- Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health
|
||
and degraded replicas (alerts in [10](10-observability-and-alerting.md)).
|
||
|
||
## Separation of state from workload (the rebuild guarantee)
|
||
|
||
Every service's catalog entry declares its **persistent volumes** explicitly. The rule:
|
||
|
||
> If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — **never** in
|
||
> the container's ephemeral filesystem or on a single node's hostPath.
|
||
|
||
The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the
|
||
service comes back with its data. This is verified per service, not assumed.
|
||
|
||
## Bulk media strategy (R-STATE-04 — the Plex problem)
|
||
|
||
Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB
|
||
media across three mini-PCs is impractical** (capacity and cost). Per **OQ-2 (resolved)**, **bulk
|
||
media lives OUTSIDE the cluster**; the specific backing is **to be determined later** and does not
|
||
block the core platform.
|
||
|
||
- The bulk store is consumed by Plex and the arr stack as an **external mount** (NFS/SMB share to be
|
||
specified when chosen). The `bulk` StorageClass / mount is a thin indirection so the concrete
|
||
backing can be filled in later without touching service definitions.
|
||
- Downloads (qbittorrent) stage to the bulk store and hand off to the arr stack.
|
||
|
||
**Regardless of the bulk backing: Plex's config/metadata database is small and MUST live on 3-replica
|
||
Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store. Bulk
|
||
media is treated as **`reacquirable`** for backup purposes unless the operator specifies otherwise
|
||
when the backing is chosen.
|
||
|
||
## Backup & disaster recovery (R-STATE-03)
|
||
|
||
- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and
|
||
orchestrates volume backups.
|
||
- **Longhorn backups** push volume data to an **off-cluster** S3-compatible or NFS target — a
|
||
separate failure domain from the cluster itself.
|
||
- **Schedule**: regular automated backups (cadence per data criticality) with retention policy.
|
||
- **Restore is a tested procedure**, not a theoretical one: a DR drill (restore a namespace + its
|
||
volumes to a clean state) is part of the platform acceptance suite.
|
||
- Bulk media has its own backup posture (e.g. it may be re-acquirable rather than backed up 1:1 —
|
||
an explicit operator decision recorded per OQ-2), but **all irreplaceable state** (configs,
|
||
databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster.
|
||
|
||
## KubeVirt VM disks
|
||
|
||
Application-VM disks are Longhorn volumes provisioned via **CDI** (Containerized Data Importer). VMs
|
||
inherit the same durability and backup story as container PVCs — a KubeVirt VM can be destroyed and
|
||
rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
|
||
|
||
## GitOps + state = full rebuildability
|
||
|
||
- **Desired state** lives in git (manifests, service definitions).
|
||
- **Data** lives on Longhorn/bulk store + off-cluster backups.
|
||
- Therefore a total cluster loss is recoverable: re-provision nodes ([02](02-environments-and-provisioning.md)),
|
||
re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end
|
||
recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once in
|
||
dev/test before prod is declared ready.
|
||
|
||
## Acceptance gates
|
||
- Delete and recreate each stateful service; data survives (per-service rebuild test).
|
||
- Kill a node (dev/test); replicated volumes stay available; cluster re-converges.
|
||
- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
|
||
- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.
|
||
|
||
## Resolved
|
||
- **OQ-2** — bulk media lives **outside the cluster**; concrete backing **TBD later** (consumed as an
|
||
external NFS/SMB mount via the `bulk` indirection). Does not block the core platform.
|
||
|
||
## Open questions
|
||
- **Off-cluster DR backup target** (separate from bulk media): which S3-compatible endpoint or NFS
|
||
host holds Velero/Longhorn backups of the *irreplaceable* tier (app config, Nextcloud, Gitea, DBs),
|
||
and its own resilience. Needed by **Phase 4**, not Phase 2.
|
||
- Concrete bulk-media backing + whether any of it is non-`reacquirable` (operator decides when chosen).
|