docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
1ddc14b918
commit
250fb54154
43 files changed
+1672
-575
No files matched your search
@@ -1,6 +1,8 @@
|
||||
# 04 — Storage & State
|
||||
|
||||
Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md).
|
||||
Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md),
|
||||
[ADR-0018 (bulk tier + backup)](../decisions/adr-0018-storage-backup-truenas-r2.md),
|
||||
[ADR-0019 (PostgreSQL via CloudNativePG)](../decisions/adr-0019-postgresql-cloudnativepg.md).
|
||||
|
||||
The core promise: **any service can be destroyed and rebuilt on a whim without losing data.** This
|
||||
requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a
|
||||
@@ -10,21 +12,41 @@ tested backup/restore path.
|
||||
|
||||
| Tier | Backing | Use | Replication |
|
||||
|---|---|---|---|
|
||||
| **App/config state** | Longhorn block volumes | PVCs for service config, databases, small app data | 3-way replica across nodes |
|
||||
| **Bulk media** | Dedicated large store (see below) | Plex library, download staging | Not 3× replicated (capacity strategy) |
|
||||
| **App/config state** | Longhorn block volumes | PVCs for service config, embedded databases, small app data | 3-way replica across nodes |
|
||||
| **Databases** | CloudNativePG clusters on Longhorn | PostgreSQL for Keycloak, Nextcloud, Semaphore | 3 instances, each on a 3-way replicated volume |
|
||||
| **Bulk media** | Off-cluster NAS, consumed over NFS | Plex library, download staging, read-only legacy Nextcloud archive | Not 3× replicated (capacity strategy) |
|
||||
| **VM disks** | Longhorn (via KubeVirt/CDI) | KubeVirt application-VM root/data disks | 3-way replica |
|
||||
| **Backups** | Off-cluster S3/NFS target | Velero + Longhorn backups for DR | Off-cluster, separate failure domain |
|
||||
| **Backups** | Cloudflare R2 (S3-compatible, off-site) | Velero + kopia volume data; CloudNativePG WAL and base backups | Off-cluster, separate failure domain |
|
||||
|
||||
## Longhorn (primary persistent storage — R-STATE-02)
|
||||
|
||||
- Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC
|
||||
cluster with local storage and no SAN.
|
||||
- Default StorageClass: **3 replicas** so any single node loss keeps data available.
|
||||
- Two further classes serve KubeVirt: a live-migratable block class (3 replicas, the default for VM
|
||||
disks) and a single-replica class for CDI's throwaway import scratch, the one non-replicated use of
|
||||
Longhorn.
|
||||
- Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC
|
||||
and its data intact; rebuilding re-attaches it (R-STATE-01).
|
||||
- Snapshots for point-in-time; **backups** (to off-cluster target) for DR.
|
||||
- Snapshots are available on demand and are how VM disks clone; **backups** for DR go off-cluster
|
||||
through Velero (below), not through a Longhorn backup target.
|
||||
- Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health
|
||||
and degraded replicas (alerts in [10](10-observability-and-alerting.md)).
|
||||
and degraded replicas (alerts in [10](10-observability-and-alerting.md)), and cap Longhorn's write
|
||||
concurrency so replica I/O cannot starve etcd's WAL.
|
||||
- **Accepted risk:** each node has one data device, so etcd's WAL and every Longhorn replica share it.
|
||||
The concurrency caps reduce that contention and do not remove it. The exit condition is a dedicated
|
||||
etcd device per node (see [known gaps](16-known-gaps.md)).
|
||||
|
||||
## Database state (PostgreSQL)
|
||||
|
||||
Services that need PostgreSQL get a **CloudNativePG** cluster per database ([ADR-0019](../decisions/adr-0019-postgresql-cloudnativepg.md)):
|
||||
three instances on three different nodes, each on replicated Longhorn. Longhorn replication keeps the
|
||||
*data* through a node loss; a streaming standby keeps the *service*, because promotion takes seconds.
|
||||
Standbys replicate corruption faithfully, so neither is a backup: WAL is archived continuously and
|
||||
base backups run daily to the off-cluster target, giving point-in-time recovery.
|
||||
|
||||
Services whose datastore is embedded (Gitea's and Crafty's SQLite, for example) keep it on their
|
||||
Longhorn PVC and are covered by the ordinary volume backup.
|
||||
|
||||
## Separation of state from workload (the rebuild guarantee)
|
||||
|
||||
@@ -33,44 +55,65 @@ Every service's catalog entry declares its **persistent volumes** explicitly. Th
|
||||
> If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — **never** in
|
||||
> the container's ephemeral filesystem or on a single node's hostPath.
|
||||
|
||||
A volume counts only when it is mounted at the path the application actually writes. A PVC mounted
|
||||
anywhere else binds, reports healthy and backs up successfully while the real state sits on the
|
||||
ephemeral layer, so a service's mount path is declared deliberately rather than left to a default.
|
||||
|
||||
The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the
|
||||
service comes back with its data. This is verified per service, not assumed.
|
||||
|
||||
## Bulk media strategy (R-STATE-04 — the Plex problem)
|
||||
|
||||
Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB
|
||||
Plex is the largest service and its library is multiple terabytes. **3× replicating multi-TB
|
||||
media across three mini-PCs is impractical** (capacity and cost). Per **OQ-2 (resolved)**, **bulk
|
||||
media lives OUTSIDE the cluster**; the specific backing is **to be determined later** and does not
|
||||
block the core platform.
|
||||
media lives OUTSIDE the cluster** on a standalone TrueNAS appliance: storage only, ZFS-backed, and on
|
||||
the same LAN as the nodes ([ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md)).
|
||||
|
||||
- The bulk store is consumed by Plex and the arr stack as an **external mount** (NFS/SMB share to be
|
||||
specified when chosen). The `bulk` StorageClass / mount is a thin indirection so the concrete
|
||||
backing can be filled in later without touching service definitions.
|
||||
- The bulk store is consumed by Plex, the arr stack and qbittorrent as an **external NFS mount**.
|
||||
Nextcloud mounts a separate legacy archive from it read-only.
|
||||
- A catalog entry's `class: bulk` plus a named export is a thin indirection: it attaches an existing
|
||||
dataset as a statically bound volume, and the NFS server is a per-environment value, so service
|
||||
definitions never name the storage host. Nothing is dynamically provisioned, because every bulk
|
||||
volume attaches data that already exists.
|
||||
- Downloads (qbittorrent) stage to the bulk store and hand off to the arr stack.
|
||||
- Layout, export settings and the uid model are in `docs/BULK-NFS-STORAGE.md` in the
|
||||
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).
|
||||
|
||||
**Regardless of the bulk backing: Plex's config/metadata database is small and MUST live on 3-replica
|
||||
Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store. Bulk
|
||||
media is treated as **`reacquirable`** for backup purposes unless the operator specifies otherwise
|
||||
when the backing is chosen.
|
||||
Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store.
|
||||
|
||||
**The bulk tier is outside platform backup.** Velero skips NFS volumes by volume type, so the exclusion
|
||||
also covers any future export, while the same namespaces' Longhorn PVCs are still backed up. A
|
||||
service's `backup.policy` therefore speaks for its Longhorn volumes only, and the bulk data's
|
||||
durability is the NAS's own (see [known gaps](16-known-gaps.md)).
|
||||
|
||||
## Backup & disaster recovery (R-STATE-03)
|
||||
|
||||
- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and
|
||||
orchestrates volume backups.
|
||||
- **Longhorn backups** push volume data to an **off-cluster** S3-compatible or NFS target — a
|
||||
separate failure domain from the cluster itself.
|
||||
- **Schedule**: regular automated backups (cadence per data criticality) with retention policy.
|
||||
- The **off-cluster target is Cloudflare R2**: S3-compatible, off-site and a separate failure domain
|
||||
from the cluster and from the NAS. Backup tools share one bucket under disjoint prefixes, so one
|
||||
tool's retention can never reap another's objects.
|
||||
- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and, through
|
||||
**kopia file-system backup**, the data of every volume. Backup is opt-out, so a new volume is
|
||||
covered without anyone remembering to enrol it.
|
||||
- **Longhorn has no backup target** ([ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md)).
|
||||
Velero and kopia are the single restore path for volume data, an accepted trade-off.
|
||||
- **CloudNativePG** archives WAL and base backups to the same target, independently of Velero.
|
||||
- **Schedule**: regular automated backups with retention policy. Databases archive continuously;
|
||||
everything else runs nightly.
|
||||
- **Exclusions are limited to reconstructable data**, whose Kubernetes resources are still captured:
|
||||
monitoring's metrics and log volumes, and the UniFi VM's disk, for which the design relies on
|
||||
UniFi's own configuration export ([06](06-vm-workloads-kubevirt.md)).
|
||||
- **Restore is a tested procedure**, not a theoretical one: a DR drill (restore a namespace + its
|
||||
volumes to a clean state) is part of the platform acceptance suite.
|
||||
- Bulk media has its own backup posture (e.g. it may be re-acquirable rather than backed up 1:1 —
|
||||
an explicit operator decision recorded per OQ-2), but **all irreplaceable state** (configs,
|
||||
- Bulk media is outside this posture (above), but **all irreplaceable state** (configs,
|
||||
databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster.
|
||||
|
||||
## KubeVirt VM disks
|
||||
|
||||
Application-VM disks are Longhorn volumes provisioned via **CDI** (Containerized Data Importer). VMs
|
||||
inherit the same durability and backup story as container PVCs — a KubeVirt VM can be destroyed and
|
||||
rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
|
||||
share the durability of container PVCs and are backed up by default, with per-VM exceptions recorded
|
||||
in [06](06-vm-workloads-kubevirt.md) — a KubeVirt VM can be destroyed and rebuilt from its declarative
|
||||
definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
|
||||
|
||||
## GitOps + state = full rebuildability
|
||||
|
||||
@@ -78,21 +121,20 @@ rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads
|
||||
- **Data** lives on Longhorn/bulk store + off-cluster backups.
|
||||
- Therefore a total cluster loss is recoverable: re-provision nodes ([02](02-environments-and-provisioning.md)),
|
||||
re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end
|
||||
recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once in
|
||||
dev/test before prod is declared ready.
|
||||
recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once, in a
|
||||
planned maintenance window.
|
||||
|
||||
## Acceptance gates
|
||||
- Delete and recreate each stateful service; data survives (per-service rebuild test).
|
||||
- Kill a node (dev/test); replicated volumes stay available; cluster re-converges.
|
||||
- Kill a node in a planned maintenance window; replicated volumes stay available; cluster re-converges.
|
||||
- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
|
||||
- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.
|
||||
|
||||
## Resolved
|
||||
- **OQ-2** — bulk media lives **outside the cluster**; concrete backing **TBD later** (consumed as an
|
||||
external NFS/SMB mount via the `bulk` indirection). Does not block the core platform.
|
||||
- **OQ-2** — bulk media lives **outside the cluster**, on a TrueNAS appliance consumed over NFS through
|
||||
the catalog's `class: bulk` indirection (ADR-0018).
|
||||
- **OQ-BKP** — the off-cluster DR target for the *irreplaceable* tier is **Cloudflare R2** (ADR-0018).
|
||||
|
||||
## Open questions
|
||||
- **Off-cluster DR backup target** (separate from bulk media): which S3-compatible endpoint or NFS
|
||||
host holds Velero/Longhorn backups of the *irreplaceable* tier (app config, Nextcloud, Gitea, DBs),
|
||||
and its own resilience. Needed by **Phase 4**, not Phase 2.
|
||||
- Concrete bulk-media backing + whether any of it is non-`reacquirable` (operator decides when chosen).
|
||||
- Whether any bulk data, notably the photo library, is non-`reacquirable` and needs its own backup
|
||||
(operator decides).
|
||||
Reference in new issue
Block a user