jakman85
f338e91770
All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication saves the DATA when a node dies; it does not save the SERVICE — a single instance has to be rescheduled, re-attach its volume and replay WAL before Postgres accepts a connection again. With nothing in the spec keeping them apart, all three primaries had drifted onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Three instances give each Cluster a streaming standby CNPG can promote in seconds, repointing the -rw Service with no volume re-attach and no restore. UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate corruption and deletions faithfully; only the Barman archive is a backup, and adding standbys while archive_mode=on with a failing archive_command would have multiplied the disk-full clock, not hedged it. All three Clusters on both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real firstRecoverabilityPoint (verified read-only, 2026-09-07). Scaling out is non-disruptive on its own: CNPG creates -2 then -3 sequentially, each seeded by pg_basebackup from the RUNNING primary. The primary is not restarted and clients are not disconnected. The anti-affinity change that DOES cause a switchover is deliberately the next commit. DEV IMPACT: identical — dev is on the same shape (three single-instance DBs, all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes and Longhorn headroom for it (see the next commit's message for the numbers). Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning. Verified: yamllint clean; `kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0` both render `instances: 3`. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02)
132 lines
6.8 KiB
YAML
132 lines
6.8 KiB
YAML
---
|
|
# Semaphore's PostgreSQL — a CloudNativePG Cluster (CR of the CNPG CRDs from platform/cnpg), modeled
|
|
# on apps/nextcloud-db.yaml (the first CNPG-DB) / platform/keycloak/postgres-cluster.yaml. This is the
|
|
# SECOND CNPG-DB in apps/, reusing the pattern established by nextcloud.
|
|
#
|
|
# WHY A DEDICATED `semaphore-db` NAMESPACE (not co-located in `semaphore`):
|
|
# The app->db edge in catalog/semaphore.yaml (`dependencies: to: semaphore-db`) compiles to a
|
|
# CiliumNetworkPolicy whose egress selector pins BOTH the destination pod label
|
|
# (app.kubernetes.io/name=semaphore-db) AND its namespace (k8s:io.kubernetes.pod.namespace=
|
|
# semaphore-db) — cross-namespace Cilium selectors must name the namespace (catalog/render.py
|
|
# r_dependency_policies). So the DB MUST live in a namespace literally named `semaphore-db` and its
|
|
# pods MUST carry app.kubernetes.io/name=semaphore-db, injected below via inheritedMetadata.
|
|
#
|
|
# DEFAULT-DENY POSTURE — mirrors keycloak/nextcloud-db (platform/keycloak/namespace.yaml): this
|
|
# namespace is INTENTIONALLY not homelab.keithjackson.dev/app-namespace=true, so Kyverno does NOT
|
|
# generate the app default-deny baseline here and CNPG runs under Cilium's cluster-wide default-allow —
|
|
# the SAME posture the verified keycloak-db / nextcloud-db use. The semaphore SIDE keeps its
|
|
# default-deny; its rendered egress-allow opens ONLY semaphore -> semaphore-db:5432, and the DB side's
|
|
# default-allow accepts that reciprocal ingress. No app-namespace policy (auth-mode/host-namespace/
|
|
# secret-provenance) is weakened.
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: semaphore-db
|
|
labels:
|
|
# Match the Cilium egress selector's namespace pin; NOT an app-namespace (see header).
|
|
app.kubernetes.io/name: semaphore-db
|
|
---
|
|
# Owner credentials for the CNPG cluster. SOPS-encrypted (age); type basic-auth so CNPG's initdb
|
|
# consumes it as the `semaphore` role's password instead of generating a random one — this lets the
|
|
# SAME password be mirrored into ns semaphore (semaphore-bootstrap/db-password) for the app's env,
|
|
# since a Pod cannot read a Secret in another namespace. Provided as apps/semaphore-db-app.sops.yaml.
|
|
#
|
|
# CNPG Cluster (CR of postgresql.cnpg.io installed by platform/cnpg).
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: Cluster
|
|
metadata:
|
|
name: semaphore-db
|
|
namespace: semaphore-db
|
|
spec:
|
|
# THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db /
|
|
# nextcloud-db; see platform/keycloak/postgres-cluster.yaml for the full rationale. Short version:
|
|
# Longhorn's 3-way replication saves the DATA on node loss, not the SERVICE, and with
|
|
# `instances: 1` all three CNPG primaries had drifted onto k3s-prod-2, so one node took Semaphore
|
|
# down alongside SSO and Nextcloud. A streaming standby lets CNPG promote in seconds and repoint
|
|
# the -rw Service. Gated on the Barman archive below actually working (ContinuousArchiving=True
|
|
# with a real firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
|
|
#
|
|
# Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
|
|
# pg_basebackup from the running primary, which is neither restarted nor disconnected.
|
|
instances: 3
|
|
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
|
|
# Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
|
|
# semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
|
|
inheritedMetadata:
|
|
labels:
|
|
app.kubernetes.io/name: semaphore-db
|
|
storage:
|
|
storageClass: longhorn
|
|
size: 5Gi
|
|
bootstrap:
|
|
initdb:
|
|
database: semaphore
|
|
owner: semaphore
|
|
# Use the pre-provisioned owner secret (basic-auth) so its password is known + reusable by the
|
|
# app in the other namespace, instead of the random CNPG-generated one.
|
|
secret:
|
|
name: semaphore-db-app
|
|
# Emit Postgres metrics so Prometheus scrapes DB health (R-OBS-01). The CNPG operator creates the
|
|
# PodMonitor at runtime (CRD present from Phase-4 monitoring).
|
|
monitoring:
|
|
enablePodMonitor: true
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 256Mi
|
|
limits:
|
|
memory: 512Mi
|
|
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
|
|
# Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
|
|
# backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
|
|
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/semaphore-db/ — dev
|
|
# s3://cnpg/semaphore-db/ on the on-Proxmox MinIO, prod
|
|
# s3://k3s-prod-backup/barman/semaphore-db/ on Cloudflare R2. Both endpointURL and
|
|
# destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
|
|
# Credentials: PER-ENVIRONMENT (2026-09-07). dev reads semaphore-db-backup-credentials (the
|
|
# scoped MinIO key), prod reads semaphore-db-backup-credentials-prod (the Cloudflare R2
|
|
# token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
|
|
# generated golden — render.py places the -prod sidecar in this DB's wave); the
|
|
# cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
|
|
# one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
|
|
# ONE shared Secret until the split, so giving prod its R2 token would have left dev
|
|
# signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
|
|
# disk-full clock below rather than a merely missing backup.
|
|
#
|
|
# archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
|
|
# Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
|
|
# the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
|
|
# endpoint. Treat ContinuousArchiving=False here as an availability incident.
|
|
backup:
|
|
retentionPolicy: "30d"
|
|
barmanObjectStore:
|
|
destinationPath: ${CNPG_BACKUP_PATH}
|
|
endpointURL: ${S3_BACKUP_ENDPOINT}
|
|
s3Credentials:
|
|
accessKeyId:
|
|
name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
|
key: ACCESS_KEY_ID
|
|
secretAccessKey:
|
|
name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
|
key: ACCESS_SECRET_KEY
|
|
wal:
|
|
compression: gzip
|
|
maxParallel: 2
|
|
data:
|
|
compression: gzip
|
|
---
|
|
# Daily base backup of semaphore-db (WAL archived continuously by the barmanObjectStore above). CNPG
|
|
# cron is 6-field (sec min hour dom mon dow); immediate:true seeds the first base backup. 04:00 slot.
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: ScheduledBackup
|
|
metadata:
|
|
name: semaphore-db-daily
|
|
namespace: semaphore-db
|
|
spec:
|
|
immediate: true
|
|
schedule: "0 0 4 * * *"
|
|
backupOwnerReference: self
|
|
cluster:
|
|
name: semaphore-db
|
|
method: barmanObjectStore
|