jakman85andClaude Opus 5
161430d915
The previous commit created the two halves. This one wires the choice, so dev renders MinIO
credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the
overlay patch is the entire difference between environments; no base manifest is forked).
cluster-vars BACKUP_CREDENTIALS_SUFFIX base "" | dev "" | prod "-prod"
velero HelmRelease credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
3x CNPG Cluster s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters
of the way, leaving one consumer reading the OTHER environment's key with nothing rendering
wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED
variable, never on a defined-but-stale one. With one suffix the four move together or not at
all. The cost is that the rendered Secret name is not greppable from the consumer, so both
kustomizations spell the two names out in full beside the resources.
THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects
that exist in every render, and that dev is running today. The alternative failure mode is
strictly worse than the shared-Secret state this replaces: velero's pod hangs in
ContainerCreating forever when `existingSecret` names a missing Secret (the volume never
mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which
under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable
default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the
value explicitly anyway, so neither inherits it silently.
Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is
untouched (it stays decryption-only, and velero stays suspended on prod).
VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and
clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform,
platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with
that environment's variables:
strict envsubst PASSES for all 8 (env x tier) combinations
dev velero existingSecret velero-cloud-credentials
dev keycloak/nextcloud/semaphore-db <db>-backup-credentials (both key refs each)
prod velero existingSecret velero-cloud-credentials-prod
prod keycloak/nextcloud/semaphore-db <db>-backup-credentials-prod (both key refs each)
both all EIGHT Secret objects present in the rendered output (dev and prod alike)
no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell
escapes in apps/wave-0, which are pre-existing and unrelated.
Read alongside the endpoints the same overlays already set, the two environments are now
isolated on credential AND target:
dev velero s3://velero (root) + cnpg s3://cnpg @ http://192.168.1.9:9000 region us-east-1
prod velero k3s-prod-backup/velero + barman prefix @ https://<acct>.r2.cloudflarestorage.com region auto
Co-Authored-By: Claude Opus 5 <[email protected]>
329 lines
24 KiB
YAML
329 lines
24 KiB
YAML
---
|
|
# cluster-vars — the overlay knob surface (R-ENV-02).
|
|
#
|
|
# This ConfigMap holds BASE DEFAULTS. Flux `postBuild.substituteFrom` injects these into every
|
|
# `${TOKEN}` in platform/ and apps/ manifests at reconcile time. dev/ and prod/ overlays STRATEGIC-
|
|
# MERGE PATCH this single ConfigMap — that patch is the *entire* difference between environments.
|
|
#
|
|
# Add a knob here (with a base default) before referencing `${KNOB}` in a component manifest, then
|
|
# give each overlay its env-scoped value. Never fork a base manifest to differ per environment.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: cluster-vars
|
|
namespace: flux-system
|
|
data:
|
|
# --- identity of the environment ---
|
|
CLUSTER_ENV: "base" # dev | prod (overlays set this)
|
|
|
|
# --- DNS / TLS ---
|
|
# THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
|
|
# the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
|
|
# ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
|
|
#
|
|
# THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
|
|
# hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
|
|
# authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
|
|
# (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
|
|
# See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
|
|
# subdomain, so the two clusters can no longer describe the same name.
|
|
#
|
|
# dev -> home.keithjackson.dev (UNCHANGED — see below)
|
|
# prod -> prod.home.keithjackson.dev
|
|
#
|
|
# DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
|
|
# `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
|
|
# relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
|
|
# the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
|
|
# (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
|
|
# GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
|
|
# it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
|
|
# tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
|
|
BASE_DOMAIN: "home.keithjackson.dev"
|
|
|
|
# ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
|
|
# EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
|
|
# so it cannot be created, updated or deleted regardless of TXT ownership.
|
|
#
|
|
# DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
|
|
# apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
|
|
# that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
|
|
# record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
|
|
# from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
|
|
#
|
|
# PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
|
|
# (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
|
|
# chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
|
|
# — it can never delete another environment's records — and the fallback is to set prod's value
|
|
# back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
|
|
EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
|
|
|
|
# ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
|
|
# neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
|
|
# record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
|
|
# `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
|
|
# one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
|
|
#
|
|
# DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
|
|
# under owner `default`; changing dev's value would make dev's own external-dns stop recognising
|
|
# them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
|
|
#
|
|
# THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
|
|
# forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
|
|
# this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
|
|
# visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
|
|
EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
|
|
|
|
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
|
|
GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
|
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
|
|
# ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
|
|
# takes the pool's first free IP, and coredns-custom answers
|
|
# *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
|
|
# hairpins through the Gateway (correct SNI -> wildcard cert).
|
|
# Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
|
|
# long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
|
|
# forgets this knob at least lands on a REAL network). A prod overlay
|
|
# MUST still set the prod LAN VIP alongside the prod L2 pool range.
|
|
|
|
# --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply -------------
|
|
# These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint:
|
|
# `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24)
|
|
# — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values
|
|
# (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the
|
|
# dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block
|
|
# before `flux bootstrap --path=clusters/prod`.
|
|
#
|
|
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
|
|
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
|
|
# (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
|
|
# platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
|
|
# tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
|
|
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH), velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
|
|
# VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
|
|
# UNSUBSTITUTED path would ship the literal string into the cluster.
|
|
#
|
|
# ADDING A KNOB IS SAFE UNDER StrictPostBuildSubstitutions ONLY BECAUSE OF THE ROOT EDGE. An
|
|
# UNDEFINED variable is a HARD Kustomization failure, so a manifest that references a knob the
|
|
# live ConfigMap does not yet carry breaks its whole tier. `clusters/base/platform.yaml` gained
|
|
# `dependsOn: [flux-system]` for exactly this: dependsOn is REVISION-gated, so no substituting
|
|
# tier can reconcile revision N until the ROOT Kustomization has APPLIED revision N — i.e. until
|
|
# the new cluster-vars key EXISTS. Every substituting consumer chains transitively through
|
|
# `platform`, so one edge covers the graph, and a new knob plus its first reference may ship in
|
|
# ONE commit. (A hand-run `kubectl apply -k clusters/<env>` bypasses that ordering entirely: apply
|
|
# the overlay BEFORE the tier that references the knob, or the tier fails until it retries.)
|
|
|
|
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
|
|
#
|
|
# *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
|
|
# 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
|
|
# control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
|
|
# live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
|
|
# own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
|
|
# Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
|
|
# outage you would notice. A virtual address that follows the control plane is the fix, and it is
|
|
# what kube-vip was stood up to provide.
|
|
#
|
|
# dev -> 192.168.1.49 (= kube_vip_address, group_vars/all.yml; base default, below)
|
|
# prod -> 192.168.2.9 (= kube_vip_address, group_vars/k3s_prod/vars.yml)
|
|
#
|
|
# THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
|
|
# unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
|
|
# move. Read it as "the API host Cilium dials", which is what it always meant.
|
|
#
|
|
# [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
|
|
# (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
|
|
# already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
|
|
# the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
|
|
# already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
|
|
# block at the top of group_vars/all.yml.
|
|
#
|
|
# Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
|
|
# in both envs, so it is NOT a knob.
|
|
#
|
|
# TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
|
|
# both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
|
|
# https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
|
|
K3S_API_HOST: "192.168.1.49"
|
|
|
|
# CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
|
|
# L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
|
|
# tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in
|
|
# practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's
|
|
# first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and
|
|
# the flagged block in clusters/prod/kustomization.yaml.)
|
|
CILIUM_LB_POOL_START: "192.168.1.53"
|
|
CILIUM_LB_POOL_STOP: "192.168.1.60"
|
|
|
|
# Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the
|
|
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
|
|
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
|
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
|
|
|
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
|
|
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
|
|
# nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
|
|
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR —
|
|
# and, measured 2026-09-05 from a pod INSIDE prod, it CANNOT reach it anyway (prod is on the SOHO
|
|
# network 192.168.2.0/24, which is pfSense's WAN, while this MinIO sits inside pfSense's LAN:
|
|
# `curl http://192.168.1.9:9000/minio/health/live` timed out after 15s). Prod uses Cloudflare R2.
|
|
S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
|
|
|
|
# S3 region. Every S3 client REQUIRES one; MinIO ignores whatever it is; Cloudflare R2 demands
|
|
# exactly `auto` (its `head-bucket` reports the physical hint `WNAM`, which is NOT a usable region
|
|
# string). This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-
|
|
# accident for MinIO and wrong for anything else, which is why it is a knob now. The base default
|
|
# is dev's LIVE value, deliberately: dev's Velero has been signing requests with `us-east-1`
|
|
# against that MinIO since bootstrap, and this repo's rule is that base == the known-good dev
|
|
# value. Prod overrides to `auto`.
|
|
S3_BACKUP_REGION: "us-east-1"
|
|
|
|
# --- storage (ADR-0003) ---
|
|
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
|
# Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
|
|
# replica is written into. It must be the path the NODE's own data disk is actually mounted at —
|
|
# roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
|
|
# fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
|
|
# Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
|
|
# working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
|
|
# on it — changing dev's value would orphan every replica, so the dev overlay restates it
|
|
# verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
|
|
# dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
|
|
# neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
|
|
LONGHORN_DATA_PATH: "/var/lib/longhorn"
|
|
|
|
# --- observability sizing (docs/10) ---
|
|
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
|
|
MONITORING_PROMETHEUS_PVC_SIZE: "10Gi"
|
|
# SIZE-based retention cap for Prometheus (Prometheus *iB units, NOT the K8s `Gi` of the PVC knob).
|
|
# MUST stay below MONITORING_PROMETHEUS_PVC_SIZE — leaves headroom for the WAL + head + compaction
|
|
# scratch (retentionSize counts only persisted blocks). Prevents a busy TSDB from filling /prometheus
|
|
# (a full disk wedges all WAL commits and makes every absent()/==0 alert fire falsely). Base 10Gi PVC.
|
|
MONITORING_PROMETHEUS_RETENTION_SIZE: "8GiB"
|
|
MONITORING_LOKI_PVC_SIZE: "10Gi"
|
|
|
|
# --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture;
|
|
# the dev overlay softens them for bootstrap (Audit / Ignore / 1) so a cold cluster can
|
|
# come up before policies are proven and admission is HA. ---
|
|
KYVERNO_VALIDATION_ACTION: "Enforce" # dev: Audit (observe before enforcing)
|
|
KYVERNO_FORCE_FAILURE_POLICY_IGNORE: "false" # dev: "true" (chart forceFailurePolicyIgnore = fail-open)
|
|
KYVERNO_ADMISSION_REPLICAS: "3" # dev: 1 (no HA needed to iterate)
|
|
|
|
# --- egress / VPN (Phase 3, ADR-0015) — node-level host OpenVPN tunnel steered by Cilium Egress
|
|
# Gateway (platform/vpn-egress/egress-steering.yaml). No in-cluster gateway pod. ---
|
|
PIA_REGION: "SE Stockholm" # PF-capable PIA region (Sweden). Under ADR-0015 the tunnel is
|
|
# host-side (Ansible vault / the node's .ovpn per docs/07), so this
|
|
# is now a reference default only — retained, not substituted into
|
|
# any manifest. ("SE Stockholm" is PF-capable; NOT "SE Streaming
|
|
# Optimized" / a US region, which lack PF.)
|
|
CLUSTER_POD_CIDR: "10.0.0.0/16" # Documents the Cilium cluster-pool pod CIDR that the egress-steering
|
|
# CiliumEgressGatewayPolicy carves out of the tunnel via excludedCIDRs
|
|
# (kept on the LOCAL datapath, never SNAT'd out tun0). The policy
|
|
# currently excludes the whole /8 RFC1918 range; this /16 records the
|
|
# actual pod allocations (Cilium assigns /24 per node from 10.0.0.0 up).
|
|
|
|
# --- identity / auth (ADR-0014) ---
|
|
KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
OAUTH2_PROXY_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
|
|
# --- backup / DR (ADR-0003, R-STATE-03) ---
|
|
VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ
|
|
VELERO_RETENTION: "168h" # dev short | prod long
|
|
|
|
# ==========================================================================================
|
|
# OBJECT-STORE TENANCY — the three knobs that keep two clusters and two backup tools from
|
|
# writing into each other's data. THESE ARE SAFETY KNOBS, not preferences.
|
|
# ==========================================================================================
|
|
# Nothing about the object layout was per-environment before this: Velero's bucket was the
|
|
# hardcoded literal `velero` with no prefix, and every CNPG Cluster used the hardcoded
|
|
# `s3://cnpg` with barman's `serverName` defaulting to the CLUSTER NAME — and BOTH environments
|
|
# have a keycloak-db, a nextcloud-db and a semaphore-db. Point two clusters at one bucket under
|
|
# that layout and:
|
|
# (a) each cluster's Velero ADOPTS the other's backups and TTL-DELETES them on retention; and
|
|
# (b) two different Postgres servers stream WAL into ONE Barman prefix, which does not merely
|
|
# mix files — it CORRUPTS THE CHAIN and destroys PITR for both. That is the worse half:
|
|
# dev's DR restore is PROVEN (R-STATE-03) and this would silently un-prove it.
|
|
#
|
|
# WHY THERE IS NO SINGLE `S3_BACKUP_BUCKET`. The two consumers take differently-SHAPED values:
|
|
# the Velero chart wants `bucket` and `prefix` as separate fields, while CNPG's
|
|
# barmanObjectStore.destinationPath is ONE full s3:// URL whose first path segment IS the bucket.
|
|
# CNPG's bucket therefore already lives inside CNPG_BACKUP_PATH; adding a second knob that had to
|
|
# hold the same bucket name a second time would be a rotation waiting to drift — the exact class
|
|
# of duplication this repo has been bitten by. Three knobs, each shaped for its own consumer, and
|
|
# no two of them can contradict each other.
|
|
#
|
|
# BASE = DEV'S LIVE LAYOUT, BYTE-FOR-BYTE, AND MUST STAY THAT WAY. Dev's existing backups are at
|
|
# `s3://velero/` (bucket root, no prefix) and `s3://cnpg/<clusterName>/` on that MinIO. Giving dev
|
|
# a prefix, or renaming its buckets, would ORPHAN the very backups its proven restore depends on —
|
|
# the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default` and its LONGHORN_DATA_PATH stays
|
|
# /var/lib/longhorn. Verified: with an EMPTY prefix the Velero chart omits `objectStorage.prefix`
|
|
# from the rendered BackupStorageLocation entirely (helm template, chart 12.1.0), so dev's live
|
|
# BSL is unchanged by this parameterization rather than merely equivalent to it.
|
|
#
|
|
# PROD, which has no history to preserve, gets ONE bucket with TWO disjoint top-level prefixes:
|
|
#
|
|
# s3://k3s-prod-backup/velero/… Velero (kopia repo + backup metadata)
|
|
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG WAL + base backups
|
|
#
|
|
# The operator created exactly one R2 bucket, so velero-vs-barman separation has to come from the
|
|
# prefix rather than the bucket. The BUCKET ROOT IS DELIBERATELY LEFT EMPTY: if a prefix ever
|
|
# renders wrong, its objects land somewhere obviously bogus instead of silently colliding with a
|
|
# real tenant's data. Dev↔prod isolation is doubly independent — different endpoint AND different
|
|
# bucket — so neither cluster can reach the other's objects even with a mis-substituted prefix.
|
|
VELERO_BACKUP_BUCKET: "velero" # dev: the `velero` MinIO bucket | prod: k3s-prod-backup
|
|
VELERO_BACKUP_PREFIX: "" # dev: bucket root (live layout) | prod: "velero"
|
|
CNPG_BACKUP_PATH: "s3://cnpg" # dev: the `cnpg` MinIO bucket root | prod: s3://k3s-prod-backup/barman
|
|
|
|
# ------------------------------------------------------------------------------------------------
|
|
# WHICH CREDENTIAL SECRET THE BACKUP CONSUMERS READ (added 2026-09-07 — the per-env credential split)
|
|
# ------------------------------------------------------------------------------------------------
|
|
# The three knobs above say WHERE the backups go. This one says WHICH KEY signs the requests, and
|
|
# it exists because the endpoint moved per-environment while the credential did not.
|
|
#
|
|
# THE BUG IT FIXES. The four backup-credential Secrets — velero-cloud.sops.yaml and the three
|
|
# <db>-backup-credentials.sops.yaml — are SHARED BASE manifests: platform/ and apps/ are rendered
|
|
# by the same Flux tiers on both clusters, and cluster-vars substitution is the ONLY thing that
|
|
# differs (R-ENV-02). So writing prod's R2 token into those four files did not GIVE prod a
|
|
# credential, it REPLACED DEV'S: dev's Velero and all three dev CNPG Clusters would have started
|
|
# signing MinIO requests with an R2 key id. For Velero that is a broken BSL; for CNPG it is worse,
|
|
# because `archive_mode=on` means Postgres REFUSES TO RECYCLE a WAL segment it could not archive,
|
|
# so a 403 loop is a monotonically filling pgdata PVC ending in a PANIC — the same failure prod
|
|
# spent its whole bring-up in with a placeholder endpoint. One shared Secret also means rotating
|
|
# either environment's key silently breaks the other; that is the structural reason for the split,
|
|
# independent of dev's eventual decommission.
|
|
#
|
|
# THE MECHANISM. Each credential is now TWO Secret objects, both applied to BOTH clusters (they are
|
|
# base resources; nothing here forks a manifest per environment). The consumer names one:
|
|
#
|
|
# velero: credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
|
# CNPG: s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
|
#
|
|
# ONE knob, not four, ON PURPOSE. Four independent secret-name knobs could be updated three-
|
|
# quarters of the way, leaving one consumer pointed at the OTHER environment's key with nothing
|
|
# rendering wrong — and StrictPostBuildSubstitutions cannot catch that, because it only fails on an
|
|
# UNDEFINED variable, never on a defined-but-stale one. With a single suffix the four move together
|
|
# or not at all. The cost is that the rendered Secret name is not greppable from the consumer; the
|
|
# names are spelled out in full in each kustomization.yaml beside the resources.
|
|
#
|
|
# THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names,
|
|
# which are the objects that already exist and that the live dev cluster is running. A default
|
|
# naming a Secret that does not exist would be worse than the shared-Secret state it replaces:
|
|
# Velero's pod would sit in ContainerCreating forever on a missing `existingSecret` volume, and a
|
|
# CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — i.e. the disk-full clock
|
|
# again — so an unresolvable default breaks the referencing cluster HARDER than a wrong key does.
|
|
# Empty is also the value dev must carry, so base and dev agree, exactly like VELERO_BACKUP_PREFIX.
|
|
#
|
|
# An EMPTY VALUE IS SAFE UNDER StrictPostBuildSubstitutions: strict mode fails on a variable that is
|
|
# not DEFINED, not on one defined as "". VELERO_BACKUP_PREFIX above has shipped empty since the
|
|
# gate was enabled and proves it. Both overlays state this knob explicitly — dev "" and prod
|
|
# "-prod" — so neither inherits it silently.
|
|
#
|
|
# ADDING A NEW BACKUP CREDENTIAL: create BOTH halves (`x-credentials.sops.yaml` and
|
|
# `x-credentials-prod.sops.yaml`), list both in the kustomization, and reference
|
|
# `x-credentials${BACKUP_CREDENTIALS_SUFFIX}`. Never a single shared Secret again.
|
|
BACKUP_CREDENTIALS_SUFFIX: "" # dev: "" (MinIO creds) | prod: "-prod" (R2 creds)
|
|
|
|
# --- placement ---
|
|
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
|
|
PLATFORM_NODE_SELECTOR_VALUE: "linux"
|