0 Fork 0

Files

T

jakman85andClaude Opus 5.5 250fb54154 docs: bring the specification in line with the accepted implementation

The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

2026-11-08 22:46:00 -06:00

4.6 KiB

Raw Permalink Blame History

ADR-0018 — Storage and backup: TrueNAS NFS bulk tier; Velero/kopia + Barman to Cloudflare R2; no Longhorn backup target

Status: Accepted
Date: 2026-11-08
Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
Serves: R-STATE-03, R-STATE-04
Supersedes: ADR-0003 — the backup and bulk-backing clauses only. Longhorn as the replicated block store, 3 replicas by default, and Plex's metadata on Longhorn all stand.

Context

ADR-0003 left two things open or conditional. The bulk tier's backing was "decided under OQ-2", and the DR path was "Velero (with Longhorn backups)" to an unnamed off-cluster target (OQ-BKP).

Operating the platform settled both:

The S3 endpoint on the old Proxmox host could not serve prod. It was unreachable across the network boundary, measured from a pod inside prod, and that host shared a failure domain with the VMs it ran. With WAL archiving on and no working target, Postgres refused to recycle WAL and database volumes filled, so the missing target was an availability incident as well as a DR gap.
A Longhorn backup target was escalated as a blocker for the DR drill. The owner ruled on 2026-09-13 that Velero + kopia to R2 is the volume backup and that no Longhorn target will be configured.
The Proxmox host was decommissioned and its ZFS pools physically moved to a storage-only TrueNAS appliance on the nodes' own LAN. The multi-terabyte library was already there, so the tier attaches existing data and creates none.

Uniform file-system backup also collides with NFS. Opt-out backup would read the whole media library over NFS every night and upload it, once per consuming namespace.

Decision

Bulk tier. Bulk media lives on a standalone TrueNAS appliance, storage only, consumed over NFS. A catalog class: bulk entry with a named export renders a statically bound volume, and the server address is a per-environment value. There is no dynamic provisioning and no NFS CSI driver.

Off-cluster target. Cloudflare R2, one bucket, with Velero and CloudNativePG/Barman under disjoint prefixes so that neither's retention can reach the other's objects.

Volume and resource backup. Velero with kopia file-system backup, opt-out by default so no volume is forgotten. No CSI snapshot data-mover and no Volume Snapshot Location.

No Longhorn backup target. Longhorn snapshots remain an in-cluster mechanism only.

Database backup. Continuous WAL and base backups through CloudNativePG/Barman (ADR-0019).

Exclusions by property. A volume policy skips NFS and emptyDir volumes by source type, so the rule covers volumes that do not exist yet. The bulk tier is therefore outside platform backup. Reconstructable data (monitoring's metrics and logs, the UniFi VM disk) is captured as Kubernetes resources only.

Alternatives considered

MinIO/versitygw on the Proxmox host — unreachable from prod and in the same failure domain as what it protected; the host no longer exists.
MinIO on the TrueNAS — planned during the host migration; not adopted. It would keep backups on site beside the cluster, where R2 is off-site with free egress, so restore rehearsals carry no cost.
Longhorn backup target as well as, or instead of, Velero — a second restore path, declined by the owner because kopia to R2 already provides a genuine off-cluster copy.
Excluding media namespaces from Velero — rejected: those namespaces also hold irreplaceable Longhorn PVCs.
Per-pod exclusion annotations on NFS volumes — rejected: fails open. The next NFS volume is uploaded in full until someone remembers to annotate it.
Dynamic NFS provisioning (the bulk StorageClass ADR-0003 implied) — unnecessary: every bulk volume attaches a dataset that already holds data.

Consequences

The restore path is single. A failure in Velero's path to R2 has no Longhorn-side fallback; revisit if that path proves unreliable, not on principle.
R2 is not a full S3 implementation and rejects some behaviours other stores tolerate. Compatibility settings live in the implementation repo.
A bound NFS volume's source is immutable, so moving the bulk server means recreating the volumes rather than reconciling them.
The bulk tier's durability is the NAS's own. Whether any of it is irreplaceable, notably photos, is an open operator decision.
Detail: docs/BULK-NFS-STORAGE.md and platform/velero/ in the implementation repo.

4.6 KiB Raw Permalink Blame History