Files
home-network-k3s-impl/apps/wave-0/semaphore-db.yaml
T
jakman85andClaude Opus 5 b80e8eef82 feat(backup): prod shares dev's MinIO — plus the path tenancy that makes it survivable
Operator ruling: "the S3 backup location is the same as Dev." Prod's
S3_BACKUP_ENDPOINT is now http://192.168.1.9:9000 and the TBD-PROD-* placeholder
is retired. Set EXPLICITLY rather than by omitting the op — an omitted op would
inherit the same value silently, and a shared backup target must be visible in
the overlay that owns it.

SHARING THE ENDPOINT WAS NOT SAFE AS AUTHORED. Nothing about the backup PATHS
was per-environment, and enabling prod against dev's MinIO would have been the
2026-09-04 external-dns incident again, in a worse place:

  VELERO — bucket `velero`, NO prefix, and both clusters run schedules named
    `daily` / `daily-resources-only`. Velero SYNCS backups it finds in its BSL
    into itself as Backup CRs and GARBAGE-COLLECTS them at TTL, so prod would
    have adopted dev's backups and then deleted them on schedule.
  CNPG — the worse one. destinationPath `s3://cnpg` with CNPG appending
    serverName, which defaults to the CLUSTER NAME. Both environments have
    keycloak-db, nextcloud-db and semaphore-db, so prod's Postgres would archive
    WAL into the EXACT prefix dev's does. Two servers interleaving WAL into one
    Barman stream CORRUPTS THE CHAIN and destroys PITR — and dev's restore is
    PROVEN (R-STATE-03). Sharing an endpoint is fine; sharing a path is not.

Two knobs, authored once, environments differing only by scoped overrides
(R-ENV-02):
  VELERO_BACKUP_PREFIX  dev ""          | prod "prod"
  CNPG_BACKUP_PATH      dev "s3://cnpg" | prod "s3://cnpg/prod"

THE BASE VALUES ARE DEV'S CURRENT LAYOUT ON PURPOSE. Dev's existing backups live
at the bucket root and at s3://cnpg; giving dev a prefix would ORPHAN the very
backups its proven restore depends on — the same reasoning that keeps dev's
EXTERNALDNS_TXT_OWNER_ID at `default`. Prod has no history to preserve.

*** PROD STILL HAS NO BACKUP COVERAGE — the endpoint is not reachable. ***
Measured from a throwaway pod INSIDE the prod cluster (the workstation proves
nothing about pod egress, and a workstation-side reading has already misled this
project once): curl http://192.168.1.9:9000/minio/health/live → connection timed
out after 15s. pfSense does not route prod's pod network to the dev LAN's MinIO.
OPERATOR ACTION REQUIRED. velero stays suspended; note apps-wave-0 applies GREEN
regardless — the token IS defined so strict substitution succeeds — while every
write fails at RUNTIME. A green apps-wave-0 does not mean backups work.

New `standup.sh s3check` step probes the endpoint from a throwaway in-cluster
pod and fails closed, with `--s3-endpoint <url>` to test a CANDIDATE before
authoring it into cluster-vars. That is how the unreachability above was
established rather than assumed.

VERIFIED: full CI gate PASS. flux-postbuild-substitution 254 -> 260 assertions,
and neither new knob appears in the dead-knob advisory list, so both are really
consumed. Rendered both overlays: all three CNPG destinationPaths and Velero's
prefix carry the tokens; dev resolves to its existing layout, prod to the nested
paths.

DELIVERY: prod reconciles the bootstrap Gitea's main at c8c1d29. This branch also
carries the DNS inversion, which must NOT reach prod. Delivering needs a
c8c1d29-based branch with only these commits — no urgency while the routing is
still closed.

Refs R-STATE-03, R-ENV-02, R-IDP-05.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 10:59:50 -05:00

116 lines
5.6 KiB
YAML

---
# Semaphore's PostgreSQL — a CloudNativePG Cluster (CR of the CNPG CRDs from platform/cnpg), modeled
# on apps/nextcloud-db.yaml (the first CNPG-DB) / platform/keycloak/postgres-cluster.yaml. This is the
# SECOND CNPG-DB in apps/, reusing the pattern established by nextcloud.
#
# WHY A DEDICATED `semaphore-db` NAMESPACE (not co-located in `semaphore`):
# The app->db edge in catalog/semaphore.yaml (`dependencies: to: semaphore-db`) compiles to a
# CiliumNetworkPolicy whose egress selector pins BOTH the destination pod label
# (app.kubernetes.io/name=semaphore-db) AND its namespace (k8s:io.kubernetes.pod.namespace=
# semaphore-db) — cross-namespace Cilium selectors must name the namespace (catalog/render.py
# r_dependency_policies). So the DB MUST live in a namespace literally named `semaphore-db` and its
# pods MUST carry app.kubernetes.io/name=semaphore-db, injected below via inheritedMetadata.
#
# DEFAULT-DENY POSTURE — mirrors keycloak/nextcloud-db (platform/keycloak/namespace.yaml): this
# namespace is INTENTIONALLY not homelab.keithjackson.dev/app-namespace=true, so Kyverno does NOT
# generate the app default-deny baseline here and CNPG runs under Cilium's cluster-wide default-allow —
# the SAME posture the verified keycloak-db / nextcloud-db use. The semaphore SIDE keeps its
# default-deny; its rendered egress-allow opens ONLY semaphore -> semaphore-db:5432, and the DB side's
# default-allow accepts that reciprocal ingress. No app-namespace policy (auth-mode/host-namespace/
# secret-provenance) is weakened.
apiVersion: v1
kind: Namespace
metadata:
name: semaphore-db
labels:
# Match the Cilium egress selector's namespace pin; NOT an app-namespace (see header).
app.kubernetes.io/name: semaphore-db
---
# Owner credentials for the CNPG cluster. SOPS-encrypted (age); type basic-auth so CNPG's initdb
# consumes it as the `semaphore` role's password instead of generating a random one — this lets the
# SAME password be mirrored into ns semaphore (semaphore-bootstrap/db-password) for the app's env,
# since a Pod cannot read a Secret in another namespace. Provided as apps/semaphore-db-app.sops.yaml.
#
# CNPG Cluster (CR of postgresql.cnpg.io installed by platform/cnpg).
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: semaphore-db
namespace: semaphore-db
spec:
# Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
# keycloak-db / nextcloud-db). The storage is already 3-way replicated by Longhorn; raise to 3 for
# Postgres-level HA as a future hardening.
instances: 1
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
# Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
# semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
inheritedMetadata:
labels:
app.kubernetes.io/name: semaphore-db
storage:
storageClass: longhorn
size: 5Gi
bootstrap:
initdb:
database: semaphore
owner: semaphore
# Use the pre-provisioned owner secret (basic-auth) so its password is known + reusable by the
# app in the other namespace, instead of the random CNPG-generated one.
secret:
name: semaphore-db-app
# Emit Postgres metrics so Prometheus scrapes DB health (R-OBS-01). The CNPG operator creates the
# PodMonitor at runtime (CRD present from Phase-4 monitoring).
monitoring:
enablePodMonitor: true
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 512Mi
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
# Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
# backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
# (defaults to the Cluster name), so objects land under s3://cnpg/semaphore-db/. Path-style HTTP.
# Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials
# (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
# endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
# substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
backup:
retentionPolicy: "30d"
barmanObjectStore:
# PER-ENVIRONMENT PATH (cluster-vars). CNPG appends serverName, which defaults to the CLUSTER
# NAME — and both environments have a `semaphore-db`. Sharing one MinIO with a shared path
# would interleave two Postgres servers' WAL into ONE Barman stream, corrupting the chain and
# destroying PITR for both. Dev keeps `s3://cnpg` (its existing, PROVEN backups); prod nests.
destinationPath: ${CNPG_BACKUP_PATH}
endpointURL: ${S3_BACKUP_ENDPOINT}
s3Credentials:
accessKeyId:
name: semaphore-db-backup-credentials
key: ACCESS_KEY_ID
secretAccessKey:
name: semaphore-db-backup-credentials
key: ACCESS_SECRET_KEY
wal:
compression: gzip
maxParallel: 2
data:
compression: gzip
---
# Daily base backup of semaphore-db (WAL archived continuously by the barmanObjectStore above). CNPG
# cron is 6-field (sec min hour dom mon dow); immediate:true seeds the first base backup. 04:00 slot.
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: semaphore-db-daily
namespace: semaphore-db
spec:
immediate: true
schedule: "0 0 4 * * *"
backupOwnerReference: self
cluster:
name: semaphore-db
method: barmanObjectStore