Files
home-network-k3s-impl/apps/wave-0/nextcloud-db.yaml
T
jakman85 e8d8601ca6
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
feat(cnpg): required pod anti-affinity + switchover primary rolls
`instances: 3` alone does not keep the three instances apart. All three
Clusters used CNPG's default `podAntiAffinityType: preferred`, which is a
scheduler HINT — two instances may share a node whenever that scores better,
and they WILL on the path that matters: when CNPG recreates an instance after
a node loss, the surviving nodes are the only candidates and one of them
already runs an instance. `preferred` is how a 3-instance cluster decays back
into the single-node failure domain the previous commit exists to remove.

`required` makes it a scheduling constraint. On a 3-node cluster with no
spare, the cost is that an instance whose node is gone stays Pending until a
node returns. That is correct: a replacement squeezed onto an already-occupied
node dies with that node while making the cluster LOOK three-way replicated.
2 of 3 instances still serve and CNPG still promotes.

ALSO SETS primaryUpdateMethod: switchover. This is not cosmetic. Changing the
affinity changes the instance pod spec, so CNPG performs a rolling update, and
the CRD default for primaryUpdateMethod is `restart` (verified against the
live CNPG 1.30.0 CRD on both clusters) — a delete-and-recreate of the PRIMARY
pod, i.e. tens of seconds of no Postgres. On keycloak-db that is an SSO
outage, which also means kubectl OIDC, Grafana, Headlamp, Gitea and
forward-auth. With standbys now present, `switchover` promotes an
already-updated streaming replica instead, reducing the roll to a connection
reset. Known constraint documented in the manifest: under `switchover` the
operator rejects an update that changes imageName and a postgresql parameter
in the same reconcile.

Verified satisfiable before merge (read-only): k3s-prod and k3s-dev each have
3 Ready, schedulable, untainted nodes, and every pgdata PVC already has a
Longhorn replica on all three (replica-soft-anti-affinity=false), so any
instance can attach from any node.

DEV IMPACT: identical, and dev can absorb it. Longhorn dev nodes are 124.9 GiB
each with 152.0 GiB already scheduled under a 200% over-provisioning ceiling
(249.8 GiB); the six new 5Gi volumes add 30 GiB scheduled per node -> 182.0,
still inside the ceiling, and only ~600 MiB per volume is really consumed.
Prod is far looser: 239 -> 269 GiB scheduled against a 829 GiB ceiling.

THIS IS THE DISRUPTIVE COMMIT of the three. Land it on its own reconcile,
after the scale-out has produced two Ready standbys per Cluster, so the blast
radius is one clean switchover per database rather than a scale-out and a roll
at once. Expect a brief SSO interruption for keycloak-db.

Verified: yamllint clean; both tiers render the new fields; field names,
types and enum values checked against the live clusters.postgresql.cnpg.io
CRD (CNPG 1.30.0, identical on prod and dev).

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit e1d364238d73e6ee99c9a50108fe8b4e09481a7e)
2026-09-07 23:48:32 -05:00

146 lines
7.8 KiB
YAML

---
# Nextcloud's PostgreSQL — a CloudNativePG Cluster (CR of the CNPG CRDs from platform/cnpg), modeled
# on platform/keycloak/postgres-cluster.yaml. This is the first CNPG-DB in apps/ and establishes the
# reusable pattern for later DB-backed services (semaphore, unifi, ...).
#
# WHY A DEDICATED `nextcloud-db` NAMESPACE (not co-located in `nextcloud`):
# The app->db edge in catalog/nextcloud.yaml (`dependencies: to: nextcloud-db`) compiles to a
# CiliumNetworkPolicy whose egress selector pins BOTH the destination pod label
# (app.kubernetes.io/name=nextcloud-db) AND its namespace (k8s:io.kubernetes.pod.namespace=
# nextcloud-db) — cross-namespace Cilium selectors must name the namespace (catalog/render.py
# r_dependency_policies). So the DB MUST live in a namespace literally named `nextcloud-db` and its
# pods MUST carry app.kubernetes.io/name=nextcloud-db, injected below via inheritedMetadata.
#
# DEFAULT-DENY POSTURE — mirrors keycloak (platform/keycloak/namespace.yaml): this namespace is
# INTENTIONALLY not homelab.keithjackson.dev/app-namespace=true, so Kyverno does NOT generate the app
# default-deny baseline here and CNPG runs under Cilium's cluster-wide default-allow — the SAME posture
# the verified keycloak-db uses. The nextcloud SIDE keeps its default-deny; its rendered egress-allow
# opens ONLY nextcloud -> nextcloud-db:5432, and the DB side's default-allow accepts that reciprocal
# ingress. No app-namespace policy (auth-mode/host-namespace/secret-provenance) is weakened.
apiVersion: v1
kind: Namespace
metadata:
name: nextcloud-db
labels:
# Match the Cilium egress selector's namespace pin; NOT an app-namespace (see header).
app.kubernetes.io/name: nextcloud-db
---
# Owner credentials for the CNPG cluster. SOPS-encrypted (age); type basic-auth so CNPG's initdb
# consumes it as the `nextcloud` role's password instead of generating a random one — this lets the
# SAME password be mirrored into ns nextcloud (nextcloud-bootstrap/db-password) for the app's env,
# since a Pod cannot read a Secret in another namespace. Provided as apps/nextcloud-db-app.sops.yaml.
#
# CNPG Cluster (CR of postgresql.cnpg.io installed by platform/cnpg).
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: nextcloud-db
namespace: nextcloud-db
spec:
# THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db; see
# platform/keycloak/postgres-cluster.yaml for the full rationale. Short version: Longhorn's 3-way
# replication saves the DATA on node loss, not the SERVICE, and with `instances: 1` all three CNPG
# primaries had drifted onto k3s-prod-2, so one node took Nextcloud down alongside SSO and
# Semaphore. A streaming standby lets CNPG promote in seconds and repoint the -rw Service.
# Gated on the Barman archive below actually working (ContinuousArchiving=True with a real
# firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
#
# Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
# pg_basebackup from the running primary, which is neither restarted nor disconnected.
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
# Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
# nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
inheritedMetadata:
labels:
app.kubernetes.io/name: nextcloud-db
# HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
# primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
# recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
# version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
# after a node loss it may stack two instances on one node — which is how `instances: 3` decays
# back into the single-node failure domain this change removes. `required` makes it a constraint;
# the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
# serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
#
# APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
# switchover. Expect a single brief connection reset for Nextcloud.
affinity:
enablePodAntiAffinity: true
podAntiAffinityType: required
primaryUpdateMethod: switchover
storage:
storageClass: longhorn
size: 5Gi
bootstrap:
initdb:
database: nextcloud
owner: nextcloud
# Use the pre-provisioned owner secret (basic-auth) so its password is known + reusable by the
# app in the other namespace, instead of the random CNPG-generated one.
secret:
name: nextcloud-db-app
# Emit Postgres metrics so Prometheus scrapes DB health (R-OBS-01). The CNPG operator creates the
# PodMonitor at runtime (CRD present from Phase-4 monitoring).
monitoring:
enablePodMonitor: true
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 512Mi
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
# Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
# backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/nextcloud-db/ — dev
# s3://cnpg/nextcloud-db/ on the on-Proxmox MinIO, prod
# s3://k3s-prod-backup/barman/nextcloud-db/ on Cloudflare R2. Both endpointURL and
# destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
# Credentials: PER-ENVIRONMENT (2026-09-07). dev reads nextcloud-db-backup-credentials (the
# scoped MinIO key), prod reads nextcloud-db-backup-credentials-prod (the Cloudflare R2
# token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
# generated golden — render.py places the -prod sidecar in this DB's wave); the
# cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
# one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
# ONE shared Secret until the split, so giving prod its R2 token would have left dev
# signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
# disk-full clock below rather than a merely missing backup.
#
# archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
# Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
# the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
# endpoint. Treat ContinuousArchiving=False here as an availability incident.
backup:
retentionPolicy: "30d"
barmanObjectStore:
destinationPath: ${CNPG_BACKUP_PATH}
endpointURL: ${S3_BACKUP_ENDPOINT}
s3Credentials:
accessKeyId:
name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
key: ACCESS_KEY_ID
secretAccessKey:
name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
key: ACCESS_SECRET_KEY
wal:
compression: gzip
maxParallel: 2
data:
compression: gzip
---
# Daily base backup of nextcloud-db (WAL archived continuously by the barmanObjectStore above). CNPG
# cron is 6-field (sec min hour dom mon dow); immediate:true seeds the first base backup. 02:00 slot.
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: nextcloud-db-daily
namespace: nextcloud-db
spec:
immediate: true
schedule: "0 0 2 * * *"
backupOwnerReference: self
cluster:
name: nextcloud-db
method: barmanObjectStore