The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
73 lines
4.6 KiB
Markdown
73 lines
4.6 KiB
Markdown
# ADR-0018 — Storage and backup: TrueNAS NFS bulk tier; Velero/kopia + Barman to Cloudflare R2; no Longhorn backup target
|
|
- Status: Accepted
|
|
- Date: 2026-11-08
|
|
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
|
|
- Serves: R-STATE-03, R-STATE-04
|
|
- Supersedes: [ADR-0003](adr-0003-storage-longhorn.md) — the backup and bulk-backing clauses only.
|
|
**Longhorn as the replicated block store, 3 replicas by default, and Plex's metadata on Longhorn
|
|
all stand.**
|
|
|
|
## Context
|
|
ADR-0003 left two things open or conditional. The bulk tier's backing was "decided under OQ-2", and
|
|
the DR path was "Velero (with Longhorn backups)" to an unnamed off-cluster target (OQ-BKP).
|
|
|
|
Operating the platform settled both:
|
|
- The S3 endpoint on the old Proxmox host could not serve prod. It was unreachable across the network
|
|
boundary, measured from a pod inside prod, and that host shared a failure domain with the VMs it
|
|
ran. With WAL archiving on and no working target, Postgres refused to recycle WAL and database
|
|
volumes filled, so the missing target was an availability incident as well as a DR gap.
|
|
- A Longhorn backup target was escalated as a blocker for the DR drill. The owner ruled on 2026-09-13
|
|
that Velero + kopia to R2 is the volume backup and that no Longhorn target will be configured.
|
|
- The Proxmox host was decommissioned and its ZFS pools physically moved to a storage-only TrueNAS
|
|
appliance on the nodes' own LAN. The multi-terabyte library was already there, so the tier attaches
|
|
existing data and creates none.
|
|
|
|
Uniform file-system backup also collides with NFS. Opt-out backup would read the whole media library
|
|
over NFS every night and upload it, once per consuming namespace.
|
|
|
|
## Decision
|
|
**Bulk tier.** Bulk media lives on a standalone **TrueNAS** appliance, storage only, consumed over
|
|
**NFS**. A catalog `class: bulk` entry with a named export renders a statically bound volume, and the
|
|
server address is a per-environment value. There is no dynamic provisioning and no NFS CSI driver.
|
|
|
|
**Off-cluster target.** **Cloudflare R2**, one bucket, with Velero and CloudNativePG/Barman under
|
|
disjoint prefixes so that neither's retention can reach the other's objects.
|
|
|
|
**Volume and resource backup.** **Velero with kopia file-system backup**, opt-out by default so no
|
|
volume is forgotten. No CSI snapshot data-mover and no Volume Snapshot Location.
|
|
|
|
**No Longhorn backup target.** Longhorn snapshots remain an in-cluster mechanism only.
|
|
|
|
**Database backup.** Continuous WAL and base backups through CloudNativePG/Barman ([ADR-0019](adr-0019-postgresql-cloudnativepg.md)).
|
|
|
|
**Exclusions by property.** A volume policy skips NFS and emptyDir volumes by source type, so the rule
|
|
covers volumes that do not exist yet. The bulk tier is therefore **outside platform backup**.
|
|
Reconstructable data (monitoring's metrics and logs, the UniFi VM disk) is captured as Kubernetes
|
|
resources only.
|
|
|
|
## Alternatives considered
|
|
- **MinIO/versitygw on the Proxmox host** — unreachable from prod and in the same failure domain as
|
|
what it protected; the host no longer exists.
|
|
- **MinIO on the TrueNAS** — planned during the host migration; not adopted. It would keep backups on
|
|
site beside the cluster, where R2 is off-site with free egress, so restore rehearsals carry no cost.
|
|
- **Longhorn backup target as well as, or instead of, Velero** — a second restore path, declined by
|
|
the owner because kopia to R2 already provides a genuine off-cluster copy.
|
|
- **Excluding media namespaces from Velero** — rejected: those namespaces also hold irreplaceable
|
|
Longhorn PVCs.
|
|
- **Per-pod exclusion annotations on NFS volumes** — rejected: fails open. The next NFS volume is
|
|
uploaded in full until someone remembers to annotate it.
|
|
- **Dynamic NFS provisioning (the `bulk` StorageClass ADR-0003 implied)** — unnecessary: every bulk
|
|
volume attaches a dataset that already holds data.
|
|
|
|
## Consequences
|
|
- The restore path is single. A failure in Velero's path to R2 has no Longhorn-side fallback; revisit
|
|
if that path proves unreliable, not on principle.
|
|
- R2 is not a full S3 implementation and rejects some behaviours other stores tolerate. Compatibility
|
|
settings live in the implementation repo.
|
|
- A bound NFS volume's source is immutable, so moving the bulk server means recreating the volumes
|
|
rather than reconciling them.
|
|
- The bulk tier's durability is the NAS's own. Whether any of it is irreplaceable, notably photos, is
|
|
an open operator decision.
|
|
- Detail: `docs/BULK-NFS-STORAGE.md` and `platform/velero/` in the
|
|
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).
|