home-network-k3s-impl/apps/wave-0/semaphore-db.yaml

---
# Semaphore's PostgreSQL — a CloudNativePG Cluster (CR of the CNPG CRDs from platform/cnpg), modeled
# on apps/nextcloud-db.yaml (the first CNPG-DB) / platform/keycloak/postgres-cluster.yaml. This is the
# SECOND CNPG-DB in apps/, reusing the pattern established by nextcloud.
#
# WHY A DEDICATED `semaphore-db` NAMESPACE (not co-located in `semaphore`):
#   The app->db edge in catalog/semaphore.yaml (`dependencies: to: semaphore-db`) compiles to a
#   CiliumNetworkPolicy whose egress selector pins BOTH the destination pod label
#   (app.kubernetes.io/name=semaphore-db) AND its namespace (k8s:io.kubernetes.pod.namespace=
#   semaphore-db) — cross-namespace Cilium selectors must name the namespace (catalog/render.py
#   r_dependency_policies). So the DB MUST live in a namespace literally named `semaphore-db` and its
#   pods MUST carry app.kubernetes.io/name=semaphore-db, injected below via inheritedMetadata.
#
# DEFAULT-DENY POSTURE — mirrors keycloak/nextcloud-db (platform/keycloak/namespace.yaml): this
# namespace is INTENTIONALLY not homelab.keithjackson.dev/app-namespace=true, so Kyverno does NOT
# generate the app default-deny baseline here and CNPG runs under Cilium's cluster-wide default-allow —
# the SAME posture the verified keycloak-db / nextcloud-db use. The semaphore SIDE keeps its
# default-deny; its rendered egress-allow opens ONLY semaphore -> semaphore-db:5432, and the DB side's
# default-allow accepts that reciprocal ingress. No app-namespace policy (auth-mode/host-namespace/
# secret-provenance) is weakened.
apiVersion: v1
kind: Namespace
metadata:
  name: semaphore-db
  labels:
    # Match the Cilium egress selector's namespace pin; NOT an app-namespace (see header).
    app.kubernetes.io/name: semaphore-db
---
# Owner credentials for the CNPG cluster. SOPS-encrypted (age); type basic-auth so CNPG's initdb
# consumes it as the `semaphore` role's password instead of generating a random one — this lets the
# SAME password be mirrored into ns semaphore (semaphore-bootstrap/db-password) for the app's env,
# since a Pod cannot read a Secret in another namespace. Provided as apps/semaphore-db-app.sops.yaml.
#
# CNPG Cluster (CR of postgresql.cnpg.io installed by platform/cnpg).
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: semaphore-db
  namespace: semaphore-db
spec:
  # THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db /
  # nextcloud-db; see platform/keycloak/postgres-cluster.yaml for the full rationale. Short version:
  # Longhorn's 3-way replication saves the DATA on node loss, not the SERVICE, and with
  # `instances: 1` all three CNPG primaries had drifted onto k3s-prod-2, so one node took Semaphore
  # down alongside SSO and Nextcloud. A streaming standby lets CNPG promote in seconds and repoint
  # the -rw Service. Gated on the Barman archive below actually working (ContinuousArchiving=True
  # with a real firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
  #
  # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
  # pg_basebackup from the running primary, which is neither restarted nor disconnected.
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
  # Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
  # semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
  inheritedMetadata:
    labels:
      app.kubernetes.io/name: semaphore-db
  # HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
  # primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
  # recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
  # version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
  # after a node loss it may stack two instances on one node — which is how `instances: 3` decays
  # back into the single-node failure domain this change removes. `required` makes it a constraint;
  # the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
  # serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
  #
  # APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
  # switchover. Expect a single brief connection reset for Semaphore.
  affinity:
    enablePodAntiAffinity: true
    podAntiAffinityType: required
  primaryUpdateMethod: switchover
  # SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
  # rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
  # last few transactions when a primary dies before its standbys catch up, and `method: any` +
  # `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
  # never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
  # are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
  # standby into a total write outage. `preferred` relaxes to async instead.
  #
  # NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
  # no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
  postgresql:
    synchronous:
      method: any
      number: 1
      dataDurability: preferred
  storage:
    storageClass: longhorn
    size: 5Gi
  bootstrap:
    initdb:
      database: semaphore
      owner: semaphore
      # Use the pre-provisioned owner secret (basic-auth) so its password is known + reusable by the
      # app in the other namespace, instead of the random CNPG-generated one.
      secret:
        name: semaphore-db-app
  # Emit Postgres metrics so Prometheus scrapes DB health (R-OBS-01). The CNPG operator creates the
  # PodMonitor at runtime (CRD present from Phase-4 monitoring).
  monitoring:
    enablePodMonitor: true
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      memory: 512Mi
  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
  # Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
  # backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
  # destinationPath, so objects land under ${CNPG_BACKUP_PATH}/semaphore-db/ — dev
  # s3://cnpg/semaphore-db/ on the on-Proxmox MinIO, prod
  # s3://k3s-prod-backup/barman/semaphore-db/ on Cloudflare R2. Both endpointURL and
  # destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
  # Credentials: PER-ENVIRONMENT (2026-09-07). dev reads semaphore-db-backup-credentials (the
  # scoped MinIO key), prod reads semaphore-db-backup-credentials-prod (the Cloudflare R2
  # token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
  # generated golden — render.py places the -prod sidecar in this DB's wave); the
  # cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
  # one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
  # ONE shared Secret until the split, so giving prod its R2 token would have left dev
  # signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
  # disk-full clock below rather than a merely missing backup.
  #
  # archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
  # Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
  # the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
  # endpoint. Treat ContinuousArchiving=False here as an availability incident.
  backup:
    retentionPolicy: "30d"
    barmanObjectStore:
      destinationPath: ${CNPG_BACKUP_PATH}
      endpointURL: ${S3_BACKUP_ENDPOINT}
      s3Credentials:
        accessKeyId:
          name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
          key: ACCESS_SECRET_KEY
      wal:
        compression: gzip
        maxParallel: 2
      data:
        compression: gzip
---
# Daily base backup of semaphore-db (WAL archived continuously by the barmanObjectStore above). CNPG
# cron is 6-field (sec min hour dom mon dow); immediate:true seeds the first base backup. 04:00 slot.
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: semaphore-db-daily
  namespace: semaphore-db
spec:
  immediate: true
  schedule: "0 0 4 * * *"
  backupOwnerReference: self
  cluster:
    name: semaphore-db
  method: barmanObjectStore