Prod has been archiving WAL to `TBD-PROD-S3-BACKUP-ENDPOINT` since bootstrap. Verified live 2026-09-07 on all three CNPG Clusters: ContinuousArchiving=False, "unexpected failure invoking barman-cloud-wal-archive: exit status 4". Because archive_mode is ON, that is not a degraded backup — it is a DISK-FULL CLOCK. Postgres will not recycle a segment it has not archived, so keycloak-db's pgdata PVC is at 579 MiB used / 4360 MiB free and climbing monotonically. When it fills, the primary PANICs and takes SSO down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Wiring the endpoint is what lets Postgres drain the whole accumulated backlog in one pass. THE KNOB LAYOUT (three new, one repurposed) and why it is shaped this way. The operator created exactly ONE R2 bucket, `k3s-prod-backup`, but the two backup tools want differently-SHAPED values: the Velero chart takes `bucket` and `prefix` as separate fields, while CNPG's barmanObjectStore.destinationPath is one full s3:// URL whose first path segment IS the bucket. So: S3_BACKUP_ENDPOINT dev http://192.168.1.9:9000 prod https://<acct>.r2... S3_BACKUP_REGION dev us-east-1 prod auto VELERO_BACKUP_BUCKET dev velero prod k3s-prod-backup VELERO_BACKUP_PREFIX dev "" prod velero CNPG_BACKUP_PATH dev s3://cnpg prod s3://k3s-prod-backup/barman There is deliberately NO separate `S3_BACKUP_BUCKET`: CNPG's bucket already lives inside CNPG_BACKUP_PATH, and a second knob holding the same bucket name twice is a rotation waiting to drift. Three knobs, each shaped for its own consumer, and no two of them can contradict each other. TENANCY. Inside k3s-prod-backup, velero and barman get disjoint TOP-LEVEL prefixes — s3://k3s-prod-backup/velero/ and s3://k3s-prod-backup/barman/<cluster>/ — with the bucket ROOT left empty so a misrendered prefix lands somewhere obviously wrong instead of silently colliding. This is not tidiness: Velero ADOPTS every backup under its location and TTL-DELETES it on retention, so an overlapping prefix would let it reap WAL segments; and interleaving two Postgres servers' WAL into one prefix corrupts the chain outright. Dev↔prod isolation is doubly independent — different endpoint AND different bucket names — so neither cluster can reach the other's objects even with a mis-substituted prefix. DEV IS UNCHANGED, BYTE-FOR-BYTE, and that is asserted rather than assumed. Base defaults are dev's LIVE values (verified against the running cluster: BSL bucket=velero with NO prefix, region us-east-1; all three CNPG Clusters ContinuousArchiving=True at s3://cnpg on the MinIO), and the dev overlay now restates all five explicitly for the same reason LONGHORN_DATA_PATH is restated — a silent drift in base would ORPHAN the backups dev's PROVEN restore depends on. The empty-prefix case was the one real risk, and it is settled empirically: with prefix "" the chart omits objectStorage.prefix from the rendered BackupStorageLocation ENTIRELY (helm template, chart 12.1.0), so dev's live BSL is unchanged, not merely equivalent. R2's TWO GOTCHAS, both wired up front: - region must be exactly `auto`. head-bucket answers {"BucketRegion":"WNAM"}, which is a PHYSICAL LOCATION HINT, not a signing region. This was hardcoded us-east-1 inside the Velero HelmRelease — correct-by-accident for MinIO, wrong for anything else — so it is a knob now. - checksumAlgorithm: "". velero-plugin-for-aws v1.14.2 bundles an AWS SDK with flexible checksums on by default and sends x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT; R2 rejects those with HTTP 400 InvalidArgument, failing every backup and every kopia blob upload. It must be PRESENT-but-empty (the plugin branches on the key existing), and the chart does emit an empty value — verified by rendering it. Harmless against MinIO, so it is unconditional rather than a fourth knob. WHAT THIS DOES NOT DO. Velero stays SUSPENDED on prod. `spec.suspend: true` is owned by field manager `flux` (verified: kustomize-controller's managedFields do not include f:suspend), so server-side apply will not strip it, and nothing here declares suspension — which would also trip R-ENV-03, since a prod overlay may patch only cluster-vars and a Flux Kustomization's `decryption`. Unsuspending is the operator's, after `standup.sh --s3-endpoint <r2> s3check` proves a round trip. NO DNS KNOB IS TOUCHED. BASE_DOMAIN stays prod.home.keithjackson.dev and every EXTERNALDNS_* value is untouched. The apex inversion on work/prod-standup-campaign was deliberately NOT carried: this branch ports that branch's R2 reasoning knob by knob, and merges nothing. VERIFIED: ci/l0.sh green (flux-postbuild-substitution 272 assertions, 0 hazards, 33 knobs defined / 28 referenced, every new token resolved in BOTH overlays); render-golden, lint, schema, spec-immutability green; touched requirement R-STATE-03 covered. Refs R-STATE-03, R-IDP-05, R-ENV-02, R-ENV-03. Co-Authored-By: Claude Opus 5 <[email protected]>
118 lines
5.5 KiB
YAML
118 lines
5.5 KiB
YAML
---
|
|
# Semaphore's PostgreSQL — a CloudNativePG Cluster (CR of the CNPG CRDs from platform/cnpg), modeled
|
|
# on apps/nextcloud-db.yaml (the first CNPG-DB) / platform/keycloak/postgres-cluster.yaml. This is the
|
|
# SECOND CNPG-DB in apps/, reusing the pattern established by nextcloud.
|
|
#
|
|
# WHY A DEDICATED `semaphore-db` NAMESPACE (not co-located in `semaphore`):
|
|
# The app->db edge in catalog/semaphore.yaml (`dependencies: to: semaphore-db`) compiles to a
|
|
# CiliumNetworkPolicy whose egress selector pins BOTH the destination pod label
|
|
# (app.kubernetes.io/name=semaphore-db) AND its namespace (k8s:io.kubernetes.pod.namespace=
|
|
# semaphore-db) — cross-namespace Cilium selectors must name the namespace (catalog/render.py
|
|
# r_dependency_policies). So the DB MUST live in a namespace literally named `semaphore-db` and its
|
|
# pods MUST carry app.kubernetes.io/name=semaphore-db, injected below via inheritedMetadata.
|
|
#
|
|
# DEFAULT-DENY POSTURE — mirrors keycloak/nextcloud-db (platform/keycloak/namespace.yaml): this
|
|
# namespace is INTENTIONALLY not homelab.keithjackson.dev/app-namespace=true, so Kyverno does NOT
|
|
# generate the app default-deny baseline here and CNPG runs under Cilium's cluster-wide default-allow —
|
|
# the SAME posture the verified keycloak-db / nextcloud-db use. The semaphore SIDE keeps its
|
|
# default-deny; its rendered egress-allow opens ONLY semaphore -> semaphore-db:5432, and the DB side's
|
|
# default-allow accepts that reciprocal ingress. No app-namespace policy (auth-mode/host-namespace/
|
|
# secret-provenance) is weakened.
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: semaphore-db
|
|
labels:
|
|
# Match the Cilium egress selector's namespace pin; NOT an app-namespace (see header).
|
|
app.kubernetes.io/name: semaphore-db
|
|
---
|
|
# Owner credentials for the CNPG cluster. SOPS-encrypted (age); type basic-auth so CNPG's initdb
|
|
# consumes it as the `semaphore` role's password instead of generating a random one — this lets the
|
|
# SAME password be mirrored into ns semaphore (semaphore-bootstrap/db-password) for the app's env,
|
|
# since a Pod cannot read a Secret in another namespace. Provided as apps/semaphore-db-app.sops.yaml.
|
|
#
|
|
# CNPG Cluster (CR of postgresql.cnpg.io installed by platform/cnpg).
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: Cluster
|
|
metadata:
|
|
name: semaphore-db
|
|
namespace: semaphore-db
|
|
spec:
|
|
# Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
|
|
# keycloak-db / nextcloud-db). The storage is already 3-way replicated by Longhorn; raise to 3 for
|
|
# Postgres-level HA as a future hardening.
|
|
instances: 1
|
|
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
|
|
# Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
|
|
# semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
|
|
inheritedMetadata:
|
|
labels:
|
|
app.kubernetes.io/name: semaphore-db
|
|
storage:
|
|
storageClass: longhorn
|
|
size: 5Gi
|
|
bootstrap:
|
|
initdb:
|
|
database: semaphore
|
|
owner: semaphore
|
|
# Use the pre-provisioned owner secret (basic-auth) so its password is known + reusable by the
|
|
# app in the other namespace, instead of the random CNPG-generated one.
|
|
secret:
|
|
name: semaphore-db-app
|
|
# Emit Postgres metrics so Prometheus scrapes DB health (R-OBS-01). The CNPG operator creates the
|
|
# PodMonitor at runtime (CRD present from Phase-4 monitoring).
|
|
monitoring:
|
|
enablePodMonitor: true
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 256Mi
|
|
limits:
|
|
memory: 512Mi
|
|
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
|
|
# Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
|
|
# backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
|
|
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/semaphore-db/ — dev
|
|
# s3://cnpg/semaphore-db/ on the on-Proxmox MinIO, prod
|
|
# s3://k3s-prod-backup/barman/semaphore-db/ on Cloudflare R2. Both endpointURL and
|
|
# destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
|
|
# Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials
|
|
# (same-ns only).
|
|
#
|
|
# archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
|
|
# Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
|
|
# the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
|
|
# endpoint. Treat ContinuousArchiving=False here as an availability incident.
|
|
backup:
|
|
retentionPolicy: "30d"
|
|
barmanObjectStore:
|
|
destinationPath: ${CNPG_BACKUP_PATH}
|
|
endpointURL: ${S3_BACKUP_ENDPOINT}
|
|
s3Credentials:
|
|
accessKeyId:
|
|
name: semaphore-db-backup-credentials
|
|
key: ACCESS_KEY_ID
|
|
secretAccessKey:
|
|
name: semaphore-db-backup-credentials
|
|
key: ACCESS_SECRET_KEY
|
|
wal:
|
|
compression: gzip
|
|
maxParallel: 2
|
|
data:
|
|
compression: gzip
|
|
---
|
|
# Daily base backup of semaphore-db (WAL archived continuously by the barmanObjectStore above). CNPG
|
|
# cron is 6-field (sec min hour dom mon dow); immediate:true seeds the first base backup. 04:00 slot.
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: ScheduledBackup
|
|
metadata:
|
|
name: semaphore-db-daily
|
|
namespace: semaphore-db
|
|
spec:
|
|
immediate: true
|
|
schedule: "0 0 4 * * *"
|
|
backupOwnerReference: self
|
|
cluster:
|
|
name: semaphore-db
|
|
method: barmanObjectStore
|