docs: bring the specification in line with the accepted implementation

The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-11-08 22:46:00 -06:00
1 parent 1ddc14b918
commit 250fb54154
43 files changed
+1672 -575

No files matched your search

+75 -33
View File
@@ -1,6 +1,8 @@
# 04 — Storage & State
Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md).
Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md),
[ADR-0018 (bulk tier + backup)](../decisions/adr-0018-storage-backup-truenas-r2.md),
[ADR-0019 (PostgreSQL via CloudNativePG)](../decisions/adr-0019-postgresql-cloudnativepg.md).
The core promise: **any service can be destroyed and rebuilt on a whim without losing data.** This
requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a
@@ -10,21 +12,41 @@ tested backup/restore path.
| Tier | Backing | Use | Replication |
|---|---|---|---|
| **App/config state** | Longhorn block volumes | PVCs for service config, databases, small app data | 3-way replica across nodes |
| **Bulk media** | Dedicated large store (see below) | Plex library, download staging | Not 3× replicated (capacity strategy) |
| **App/config state** | Longhorn block volumes | PVCs for service config, embedded databases, small app data | 3-way replica across nodes |
| **Databases** | CloudNativePG clusters on Longhorn | PostgreSQL for Keycloak, Nextcloud, Semaphore | 3 instances, each on a 3-way replicated volume |
| **Bulk media** | Off-cluster NAS, consumed over NFS | Plex library, download staging, read-only legacy Nextcloud archive | Not 3× replicated (capacity strategy) |
| **VM disks** | Longhorn (via KubeVirt/CDI) | KubeVirt application-VM root/data disks | 3-way replica |
| **Backups** | Off-cluster S3/NFS target | Velero + Longhorn backups for DR | Off-cluster, separate failure domain |
| **Backups** | Cloudflare R2 (S3-compatible, off-site) | Velero + kopia volume data; CloudNativePG WAL and base backups | Off-cluster, separate failure domain |
## Longhorn (primary persistent storage — R-STATE-02)
- Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC
cluster with local storage and no SAN.
- Default StorageClass: **3 replicas** so any single node loss keeps data available.
- Two further classes serve KubeVirt: a live-migratable block class (3 replicas, the default for VM
disks) and a single-replica class for CDI's throwaway import scratch, the one non-replicated use of
Longhorn.
- Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC
and its data intact; rebuilding re-attaches it (R-STATE-01).
- Snapshots for point-in-time; **backups** (to off-cluster target) for DR.
- Snapshots are available on demand and are how VM disks clone; **backups** for DR go off-cluster
through Velero (below), not through a Longhorn backup target.
- Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health
and degraded replicas (alerts in [10](10-observability-and-alerting.md)).
and degraded replicas (alerts in [10](10-observability-and-alerting.md)), and cap Longhorn's write
concurrency so replica I/O cannot starve etcd's WAL.
- **Accepted risk:** each node has one data device, so etcd's WAL and every Longhorn replica share it.
The concurrency caps reduce that contention and do not remove it. The exit condition is a dedicated
etcd device per node (see [known gaps](16-known-gaps.md)).
## Database state (PostgreSQL)
Services that need PostgreSQL get a **CloudNativePG** cluster per database ([ADR-0019](../decisions/adr-0019-postgresql-cloudnativepg.md)):
three instances on three different nodes, each on replicated Longhorn. Longhorn replication keeps the
*data* through a node loss; a streaming standby keeps the *service*, because promotion takes seconds.
Standbys replicate corruption faithfully, so neither is a backup: WAL is archived continuously and
base backups run daily to the off-cluster target, giving point-in-time recovery.
Services whose datastore is embedded (Gitea's and Crafty's SQLite, for example) keep it on their
Longhorn PVC and are covered by the ordinary volume backup.
## Separation of state from workload (the rebuild guarantee)
@@ -33,44 +55,65 @@ Every service's catalog entry declares its **persistent volumes** explicitly. Th
> If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — **never** in
> the container's ephemeral filesystem or on a single node's hostPath.
A volume counts only when it is mounted at the path the application actually writes. A PVC mounted
anywhere else binds, reports healthy and backs up successfully while the real state sits on the
ephemeral layer, so a service's mount path is declared deliberately rather than left to a default.
The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the
service comes back with its data. This is verified per service, not assumed.
## Bulk media strategy (R-STATE-04 — the Plex problem)
Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB
Plex is the largest service and its library is multiple terabytes. **3× replicating multi-TB
media across three mini-PCs is impractical** (capacity and cost). Per **OQ-2 (resolved)**, **bulk
media lives OUTSIDE the cluster**; the specific backing is **to be determined later** and does not
block the core platform.
media lives OUTSIDE the cluster** on a standalone TrueNAS appliance: storage only, ZFS-backed, and on
the same LAN as the nodes ([ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md)).
- The bulk store is consumed by Plex and the arr stack as an **external mount** (NFS/SMB share to be
specified when chosen). The `bulk` StorageClass / mount is a thin indirection so the concrete
backing can be filled in later without touching service definitions.
- The bulk store is consumed by Plex, the arr stack and qbittorrent as an **external NFS mount**.
Nextcloud mounts a separate legacy archive from it read-only.
- A catalog entry's `class: bulk` plus a named export is a thin indirection: it attaches an existing
dataset as a statically bound volume, and the NFS server is a per-environment value, so service
definitions never name the storage host. Nothing is dynamically provisioned, because every bulk
volume attaches data that already exists.
- Downloads (qbittorrent) stage to the bulk store and hand off to the arr stack.
- Layout, export settings and the uid model are in `docs/BULK-NFS-STORAGE.md` in the
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).
**Regardless of the bulk backing: Plex's config/metadata database is small and MUST live on 3-replica
Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store. Bulk
media is treated as **`reacquirable`** for backup purposes unless the operator specifies otherwise
when the backing is chosen.
Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store.
**The bulk tier is outside platform backup.** Velero skips NFS volumes by volume type, so the exclusion
also covers any future export, while the same namespaces' Longhorn PVCs are still backed up. A
service's `backup.policy` therefore speaks for its Longhorn volumes only, and the bulk data's
durability is the NAS's own (see [known gaps](16-known-gaps.md)).
## Backup & disaster recovery (R-STATE-03)
- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and
orchestrates volume backups.
- **Longhorn backups** push volume data to an **off-cluster** S3-compatible or NFS target — a
separate failure domain from the cluster itself.
- **Schedule**: regular automated backups (cadence per data criticality) with retention policy.
- The **off-cluster target is Cloudflare R2**: S3-compatible, off-site and a separate failure domain
from the cluster and from the NAS. Backup tools share one bucket under disjoint prefixes, so one
tool's retention can never reap another's objects.
- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and, through
**kopia file-system backup**, the data of every volume. Backup is opt-out, so a new volume is
covered without anyone remembering to enrol it.
- **Longhorn has no backup target** ([ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md)).
Velero and kopia are the single restore path for volume data, an accepted trade-off.
- **CloudNativePG** archives WAL and base backups to the same target, independently of Velero.
- **Schedule**: regular automated backups with retention policy. Databases archive continuously;
everything else runs nightly.
- **Exclusions are limited to reconstructable data**, whose Kubernetes resources are still captured:
monitoring's metrics and log volumes, and the UniFi VM's disk, for which the design relies on
UniFi's own configuration export ([06](06-vm-workloads-kubevirt.md)).
- **Restore is a tested procedure**, not a theoretical one: a DR drill (restore a namespace + its
volumes to a clean state) is part of the platform acceptance suite.
- Bulk media has its own backup posture (e.g. it may be re-acquirable rather than backed up 1:1 —
an explicit operator decision recorded per OQ-2), but **all irreplaceable state** (configs,
- Bulk media is outside this posture (above), but **all irreplaceable state** (configs,
databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster.
## KubeVirt VM disks
Application-VM disks are Longhorn volumes provisioned via **CDI** (Containerized Data Importer). VMs
inherit the same durability and backup story as container PVCs — a KubeVirt VM can be destroyed and
rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
share the durability of container PVCs and are backed up by default, with per-VM exceptions recorded
in [06](06-vm-workloads-kubevirt.md) — a KubeVirt VM can be destroyed and rebuilt from its declarative
definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
## GitOps + state = full rebuildability
@@ -78,21 +121,20 @@ rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads
- **Data** lives on Longhorn/bulk store + off-cluster backups.
- Therefore a total cluster loss is recoverable: re-provision nodes ([02](02-environments-and-provisioning.md)),
re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end
recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once in
dev/test before prod is declared ready.
recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once, in a
planned maintenance window.
## Acceptance gates
- Delete and recreate each stateful service; data survives (per-service rebuild test).
- Kill a node (dev/test); replicated volumes stay available; cluster re-converges.
- Kill a node in a planned maintenance window; replicated volumes stay available; cluster re-converges.
- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.
## Resolved
- **OQ-2** — bulk media lives **outside the cluster**; concrete backing **TBD later** (consumed as an
external NFS/SMB mount via the `bulk` indirection). Does not block the core platform.
- **OQ-2** — bulk media lives **outside the cluster**, on a TrueNAS appliance consumed over NFS through
the catalog's `class: bulk` indirection (ADR-0018).
- **OQ-BKP** — the off-cluster DR target for the *irreplaceable* tier is **Cloudflare R2** (ADR-0018).
## Open questions
- **Off-cluster DR backup target** (separate from bulk media): which S3-compatible endpoint or NFS
host holds Velero/Longhorn backups of the *irreplaceable* tier (app config, Nextcloud, Gitea, DBs),
and its own resilience. Needed by **Phase 4**, not Phase 2.
- Concrete bulk-media backing + whether any of it is non-`reacquirable` (operator decides when chosen).
- Whether any bulk data, notably the photo library, is non-`reacquirable` and needs its own backup
(operator decides).