Files
home-network-k3s/decisions/adr-0018-storage-backup-truenas-r2.md
jakman85andClaude Opus 5.5 250fb54154 docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 22:46:00 -06:00

73 lines
4.6 KiB
Markdown

# ADR-0018 — Storage and backup: TrueNAS NFS bulk tier; Velero/kopia + Barman to Cloudflare R2; no Longhorn backup target
- Status: Accepted
- Date: 2026-11-08
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
- Serves: R-STATE-03, R-STATE-04
- Supersedes: [ADR-0003](adr-0003-storage-longhorn.md) — the backup and bulk-backing clauses only.
**Longhorn as the replicated block store, 3 replicas by default, and Plex's metadata on Longhorn
all stand.**
## Context
ADR-0003 left two things open or conditional. The bulk tier's backing was "decided under OQ-2", and
the DR path was "Velero (with Longhorn backups)" to an unnamed off-cluster target (OQ-BKP).
Operating the platform settled both:
- The S3 endpoint on the old Proxmox host could not serve prod. It was unreachable across the network
boundary, measured from a pod inside prod, and that host shared a failure domain with the VMs it
ran. With WAL archiving on and no working target, Postgres refused to recycle WAL and database
volumes filled, so the missing target was an availability incident as well as a DR gap.
- A Longhorn backup target was escalated as a blocker for the DR drill. The owner ruled on 2026-09-13
that Velero + kopia to R2 is the volume backup and that no Longhorn target will be configured.
- The Proxmox host was decommissioned and its ZFS pools physically moved to a storage-only TrueNAS
appliance on the nodes' own LAN. The multi-terabyte library was already there, so the tier attaches
existing data and creates none.
Uniform file-system backup also collides with NFS. Opt-out backup would read the whole media library
over NFS every night and upload it, once per consuming namespace.
## Decision
**Bulk tier.** Bulk media lives on a standalone **TrueNAS** appliance, storage only, consumed over
**NFS**. A catalog `class: bulk` entry with a named export renders a statically bound volume, and the
server address is a per-environment value. There is no dynamic provisioning and no NFS CSI driver.
**Off-cluster target.** **Cloudflare R2**, one bucket, with Velero and CloudNativePG/Barman under
disjoint prefixes so that neither's retention can reach the other's objects.
**Volume and resource backup.** **Velero with kopia file-system backup**, opt-out by default so no
volume is forgotten. No CSI snapshot data-mover and no Volume Snapshot Location.
**No Longhorn backup target.** Longhorn snapshots remain an in-cluster mechanism only.
**Database backup.** Continuous WAL and base backups through CloudNativePG/Barman ([ADR-0019](adr-0019-postgresql-cloudnativepg.md)).
**Exclusions by property.** A volume policy skips NFS and emptyDir volumes by source type, so the rule
covers volumes that do not exist yet. The bulk tier is therefore **outside platform backup**.
Reconstructable data (monitoring's metrics and logs, the UniFi VM disk) is captured as Kubernetes
resources only.
## Alternatives considered
- **MinIO/versitygw on the Proxmox host** — unreachable from prod and in the same failure domain as
what it protected; the host no longer exists.
- **MinIO on the TrueNAS** — planned during the host migration; not adopted. It would keep backups on
site beside the cluster, where R2 is off-site with free egress, so restore rehearsals carry no cost.
- **Longhorn backup target as well as, or instead of, Velero** — a second restore path, declined by
the owner because kopia to R2 already provides a genuine off-cluster copy.
- **Excluding media namespaces from Velero** — rejected: those namespaces also hold irreplaceable
Longhorn PVCs.
- **Per-pod exclusion annotations on NFS volumes** — rejected: fails open. The next NFS volume is
uploaded in full until someone remembers to annotate it.
- **Dynamic NFS provisioning (the `bulk` StorageClass ADR-0003 implied)** — unnecessary: every bulk
volume attaches a dataset that already holds data.
## Consequences
- The restore path is single. A failure in Velero's path to R2 has no Longhorn-side fallback; revisit
if that path proves unreliable, not on principle.
- R2 is not a full S3 implementation and rejects some behaviours other stores tolerate. Compatibility
settings live in the implementation repo.
- A bound NFS volume's source is immutable, so moving the bulk server means recreating the volumes
rather than reconciling them.
- The bulk tier's durability is the NAS's own. Whether any of it is irreplaceable, notably photos, is
an open operator decision.
- Detail: `docs/BULK-NFS-STORAGE.md` and `platform/velero/` in the
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).