Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 6 Actions Packages Projects Releases Wiki Activity
Files
0a0527e7db5680bd3a627f0771ca913972741cf4
home-network-k3s-impl/clusters/base/cluster-vars.yaml
T
jakman85andClaude Opus 5 0a0527e7db feat(backup): prod's off-cluster DR lands on R2 — and stops the WAL clock 
Prod has been archiving WAL to `TBD-PROD-S3-BACKUP-ENDPOINT` since bootstrap.
Verified live 2026-09-07 on all three CNPG Clusters: ContinuousArchiving=False,
"unexpected failure invoking barman-cloud-wal-archive: exit status 4". Because
archive_mode is ON, that is not a degraded backup — it is a DISK-FULL CLOCK.
Postgres will not recycle a segment it has not archived, so keycloak-db's pgdata
PVC is at 579 MiB used / 4360 MiB free and climbing monotonically. When it fills,
the primary PANICs and takes SSO down, and with it kubectl OIDC, Grafana,
Headlamp, Gitea and forward-auth. Wiring the endpoint is what lets Postgres drain
the whole accumulated backlog in one pass.

THE KNOB LAYOUT (three new, one repurposed) and why it is shaped this way.

The operator created exactly ONE R2 bucket, `k3s-prod-backup`, but the two backup
tools want differently-SHAPED values: the Velero chart takes `bucket` and `prefix`
as separate fields, while CNPG's barmanObjectStore.destinationPath is one full
s3:// URL whose first path segment IS the bucket. So:

  S3_BACKUP_ENDPOINT     dev http://192.168.1.9:9000   prod https://<acct>.r2...
  S3_BACKUP_REGION       dev us-east-1                 prod auto
  VELERO_BACKUP_BUCKET   dev velero                    prod k3s-prod-backup
  VELERO_BACKUP_PREFIX   dev ""                        prod velero
  CNPG_BACKUP_PATH       dev s3://cnpg                 prod s3://k3s-prod-backup/barman

There is deliberately NO separate `S3_BACKUP_BUCKET`: CNPG's bucket already lives
inside CNPG_BACKUP_PATH, and a second knob holding the same bucket name twice is a
rotation waiting to drift. Three knobs, each shaped for its own consumer, and no
two of them can contradict each other.

TENANCY. Inside k3s-prod-backup, velero and barman get disjoint TOP-LEVEL
prefixes — s3://k3s-prod-backup/velero/ and s3://k3s-prod-backup/barman/<cluster>/
— with the bucket ROOT left empty so a misrendered prefix lands somewhere
obviously wrong instead of silently colliding. This is not tidiness: Velero ADOPTS
every backup under its location and TTL-DELETES it on retention, so an overlapping
prefix would let it reap WAL segments; and interleaving two Postgres servers' WAL
into one prefix corrupts the chain outright. Dev↔prod isolation is doubly
independent — different endpoint AND different bucket names — so neither cluster
can reach the other's objects even with a mis-substituted prefix.

DEV IS UNCHANGED, BYTE-FOR-BYTE, and that is asserted rather than assumed. Base
defaults are dev's LIVE values (verified against the running cluster: BSL
bucket=velero with NO prefix, region us-east-1; all three CNPG Clusters
ContinuousArchiving=True at s3://cnpg on the MinIO), and the dev overlay now
restates all five explicitly for the same reason LONGHORN_DATA_PATH is restated —
a silent drift in base would ORPHAN the backups dev's PROVEN restore depends on.
The empty-prefix case was the one real risk, and it is settled empirically: with
prefix "" the chart omits objectStorage.prefix from the rendered
BackupStorageLocation ENTIRELY (helm template, chart 12.1.0), so dev's live BSL is
unchanged, not merely equivalent.

R2's TWO GOTCHAS, both wired up front:
  - region must be exactly `auto`. head-bucket answers {"BucketRegion":"WNAM"},
    which is a PHYSICAL LOCATION HINT, not a signing region. This was hardcoded
    us-east-1 inside the Velero HelmRelease — correct-by-accident for MinIO, wrong
    for anything else — so it is a knob now.
  - checksumAlgorithm: "". velero-plugin-for-aws v1.14.2 bundles an AWS SDK with
    flexible checksums on by default and sends x-amz-sdk-checksum-algorithm /
    x-amz-checksum-crc32 on every PUT; R2 rejects those with HTTP 400
    InvalidArgument, failing every backup and every kopia blob upload. It must be
    PRESENT-but-empty (the plugin branches on the key existing), and the chart does
    emit an empty value — verified by rendering it. Harmless against MinIO, so it
    is unconditional rather than a fourth knob.

WHAT THIS DOES NOT DO. Velero stays SUSPENDED on prod. `spec.suspend: true` is
owned by field manager `flux` (verified: kustomize-controller's managedFields do
not include f:suspend), so server-side apply will not strip it, and nothing here
declares suspension — which would also trip R-ENV-03, since a prod overlay may
patch only cluster-vars and a Flux Kustomization's `decryption`. Unsuspending is
the operator's, after `standup.sh --s3-endpoint <r2> s3check` proves a round trip.

NO DNS KNOB IS TOUCHED. BASE_DOMAIN stays prod.home.keithjackson.dev and every
EXTERNALDNS_* value is untouched. The apex inversion on work/prod-standup-campaign
was deliberately NOT carried: this branch ports that branch's R2 reasoning knob by
knob, and merges nothing.

VERIFIED: ci/l0.sh green (flux-postbuild-substitution 272 assertions, 0 hazards,
33 knobs defined / 28 referenced, every new token resolved in BOTH overlays);
render-golden, lint, schema, spec-immutability green; touched requirement
R-STATE-03 covered.

Refs R-STATE-03, R-IDP-05, R-ENV-02, R-ENV-03.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 01:14:29 -05:00

280 lines
21 KiB
YAML
Raw Blame History

---
# cluster-vars — the overlay knob surface (R-ENV-02).
#
# This ConfigMap holds BASE DEFAULTS. Flux `postBuild.substituteFrom` injects these into every
# `${TOKEN}` in platform/ and apps/ manifests at reconcile time. dev/ and prod/ overlays STRATEGIC-
# MERGE PATCH this single ConfigMap — that patch is the *entire* difference between environments.
#
# Add a knob here (with a base default) before referencing `${KNOB}` in a component manifest, then
# give each overlay its env-scoped value. Never fork a base manifest to differ per environment.
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-vars
namespace: flux-system
data:
# --- identity of the environment ---
CLUSTER_ENV: "base" # dev | prod (overlays set this)
# --- DNS / TLS ---
# THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
# the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
# ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
#
# THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
# hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
# authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
# (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
# See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
# subdomain, so the two clusters can no longer describe the same name.
#
# dev -> home.keithjackson.dev (UNCHANGED — see below)
# prod -> prod.home.keithjackson.dev
#
# DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
# `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
# relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
# the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
# (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
# GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
# it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
# tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
BASE_DOMAIN: "home.keithjackson.dev"
# ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
# EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
# so it cannot be created, updated or deleted regardless of TXT ownership.
#
# DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
# apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
# that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
# record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
# from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
#
# PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
# (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
# chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
# — it can never delete another environment's records — and the fallback is to set prod's value
# back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
# ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
# neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
# record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
# `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
# one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
#
# DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
# under owner `default`; changing dev's value would make dev's own external-dns stop recognising
# them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
#
# THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
# forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
# this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
# visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
# ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
# takes the pool's first free IP, and coredns-custom answers
# *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
# hairpins through the Gateway (correct SNI -> wildcard cert).
# Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
# long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
# forgets this knob at least lands on a REAL network). A prod overlay
# MUST still set the prod LAN VIP alongside the prod L2 pool range.
# --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply -------------
# These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint:
# `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24)
# — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values
# (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the
# dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block
# before `flux bootstrap --path=clusters/prod`.
#
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
# (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
# platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
# tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH), velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
# VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
# UNSUBSTITUTED path would ship the literal string into the cluster.
#
# ADDING A KNOB IS SAFE UNDER StrictPostBuildSubstitutions ONLY BECAUSE OF THE ROOT EDGE. An
# UNDEFINED variable is a HARD Kustomization failure, so a manifest that references a knob the
# live ConfigMap does not yet carry breaks its whole tier. `clusters/base/platform.yaml` gained
# `dependsOn: [flux-system]` for exactly this: dependsOn is REVISION-gated, so no substituting
# tier can reconcile revision N until the ROOT Kustomization has APPLIED revision N — i.e. until
# the new cluster-vars key EXISTS. Every substituting consumer chains transitively through
# `platform`, so one edge covers the graph, and a new knob plus its first reference may ship in
# ONE commit. (A hand-run `kubectl apply -k clusters/<env>` bypasses that ordering entirely: apply
# the overlay BEFORE the tier that references the knob, or the tier fails until it retries.)
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
#
# *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
# 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
# control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
# live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
# own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
# Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
# outage you would notice. A virtual address that follows the control plane is the fix, and it is
# what kube-vip was stood up to provide.
#
# dev -> 192.168.1.49 (= kube_vip_address, group_vars/all.yml; base default, below)
# prod -> 192.168.2.9 (= kube_vip_address, group_vars/k3s_prod/vars.yml)
#
# THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
# unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
# move. Read it as "the API host Cilium dials", which is what it always meant.
#
# [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
# (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
# already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
# the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
# already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
# block at the top of group_vars/all.yml.
#
# Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
# in both envs, so it is NOT a knob.
#
# TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
# both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
# https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
K3S_API_HOST: "192.168.1.49"
# CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
# L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
# tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in
# practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's
# first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and
# the flagged block in clusters/prod/kustomization.yaml.)
CILIUM_LB_POOL_START: "192.168.1.53"
CILIUM_LB_POOL_STOP: "192.168.1.60"
# Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
# nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR —
# and, measured 2026-09-05 from a pod INSIDE prod, it CANNOT reach it anyway (prod is on the SOHO
# network 192.168.2.0/24, which is pfSense's WAN, while this MinIO sits inside pfSense's LAN:
# `curl http://192.168.1.9:9000/minio/health/live` timed out after 15s). Prod uses Cloudflare R2.
S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
# S3 region. Every S3 client REQUIRES one; MinIO ignores whatever it is; Cloudflare R2 demands
# exactly `auto` (its `head-bucket` reports the physical hint `WNAM`, which is NOT a usable region
# string). This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-
# accident for MinIO and wrong for anything else, which is why it is a knob now. The base default
# is dev's LIVE value, deliberately: dev's Velero has been signing requests with `us-east-1`
# against that MinIO since bootstrap, and this repo's rule is that base == the known-good dev
# value. Prod overrides to `auto`.
S3_BACKUP_REGION: "us-east-1"
# --- storage (ADR-0003) ---
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
# Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
# replica is written into. It must be the path the NODE's own data disk is actually mounted at —
# roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
# fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
# Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
# working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
# on it — changing dev's value would orphan every replica, so the dev overlay restates it
# verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
# dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
# neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
LONGHORN_DATA_PATH: "/var/lib/longhorn"
# --- observability sizing (docs/10) ---
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
MONITORING_PROMETHEUS_PVC_SIZE: "10Gi"
# SIZE-based retention cap for Prometheus (Prometheus *iB units, NOT the K8s `Gi` of the PVC knob).
# MUST stay below MONITORING_PROMETHEUS_PVC_SIZE — leaves headroom for the WAL + head + compaction
# scratch (retentionSize counts only persisted blocks). Prevents a busy TSDB from filling /prometheus
# (a full disk wedges all WAL commits and makes every absent()/==0 alert fire falsely). Base 10Gi PVC.
MONITORING_PROMETHEUS_RETENTION_SIZE: "8GiB"
MONITORING_LOKI_PVC_SIZE: "10Gi"
# --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture;
# the dev overlay softens them for bootstrap (Audit / Ignore / 1) so a cold cluster can
# come up before policies are proven and admission is HA. ---
KYVERNO_VALIDATION_ACTION: "Enforce" # dev: Audit (observe before enforcing)
KYVERNO_FORCE_FAILURE_POLICY_IGNORE: "false" # dev: "true" (chart forceFailurePolicyIgnore = fail-open)
KYVERNO_ADMISSION_REPLICAS: "3" # dev: 1 (no HA needed to iterate)
# --- egress / VPN (Phase 3, ADR-0015) — node-level host OpenVPN tunnel steered by Cilium Egress
# Gateway (platform/vpn-egress/egress-steering.yaml). No in-cluster gateway pod. ---
PIA_REGION: "SE Stockholm" # PF-capable PIA region (Sweden). Under ADR-0015 the tunnel is
# host-side (Ansible vault / the node's .ovpn per docs/07), so this
# is now a reference default only — retained, not substituted into
# any manifest. ("SE Stockholm" is PF-capable; NOT "SE Streaming
# Optimized" / a US region, which lack PF.)
CLUSTER_POD_CIDR: "10.0.0.0/16" # Documents the Cilium cluster-pool pod CIDR that the egress-steering
# CiliumEgressGatewayPolicy carves out of the tunnel via excludedCIDRs
# (kept on the LOCAL datapath, never SNAT'd out tun0). The policy
# currently excludes the whole /8 RFC1918 range; this /16 records the
# actual pod allocations (Cilium assigns /24 per node from 10.0.0.0 up).
# --- identity / auth (ADR-0014) ---
KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA)
OAUTH2_PROXY_REPLICAS: "1" # dev 1 | prod 2 (HA)
# --- backup / DR (ADR-0003, R-STATE-03) ---
VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ
VELERO_RETENTION: "168h" # dev short | prod long
# ==========================================================================================
# OBJECT-STORE TENANCY — the three knobs that keep two clusters and two backup tools from
# writing into each other's data. THESE ARE SAFETY KNOBS, not preferences.
# ==========================================================================================
# Nothing about the object layout was per-environment before this: Velero's bucket was the
# hardcoded literal `velero` with no prefix, and every CNPG Cluster used the hardcoded
# `s3://cnpg` with barman's `serverName` defaulting to the CLUSTER NAME — and BOTH environments
# have a keycloak-db, a nextcloud-db and a semaphore-db. Point two clusters at one bucket under
# that layout and:
# (a) each cluster's Velero ADOPTS the other's backups and TTL-DELETES them on retention; and
# (b) two different Postgres servers stream WAL into ONE Barman prefix, which does not merely
# mix files — it CORRUPTS THE CHAIN and destroys PITR for both. That is the worse half:
# dev's DR restore is PROVEN (R-STATE-03) and this would silently un-prove it.
#
# WHY THERE IS NO SINGLE `S3_BACKUP_BUCKET`. The two consumers take differently-SHAPED values:
# the Velero chart wants `bucket` and `prefix` as separate fields, while CNPG's
# barmanObjectStore.destinationPath is ONE full s3:// URL whose first path segment IS the bucket.
# CNPG's bucket therefore already lives inside CNPG_BACKUP_PATH; adding a second knob that had to
# hold the same bucket name a second time would be a rotation waiting to drift — the exact class
# of duplication this repo has been bitten by. Three knobs, each shaped for its own consumer, and
# no two of them can contradict each other.
#
# BASE = DEV'S LIVE LAYOUT, BYTE-FOR-BYTE, AND MUST STAY THAT WAY. Dev's existing backups are at
# `s3://velero/` (bucket root, no prefix) and `s3://cnpg/<clusterName>/` on that MinIO. Giving dev
# a prefix, or renaming its buckets, would ORPHAN the very backups its proven restore depends on —
# the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default` and its LONGHORN_DATA_PATH stays
# /var/lib/longhorn. Verified: with an EMPTY prefix the Velero chart omits `objectStorage.prefix`
# from the rendered BackupStorageLocation entirely (helm template, chart 12.1.0), so dev's live
# BSL is unchanged by this parameterization rather than merely equivalent to it.
#
# PROD, which has no history to preserve, gets ONE bucket with TWO disjoint top-level prefixes:
#
# s3://k3s-prod-backup/velero/… Velero (kopia repo + backup metadata)
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG WAL + base backups
#
# The operator created exactly one R2 bucket, so velero-vs-barman separation has to come from the
# prefix rather than the bucket. The BUCKET ROOT IS DELIBERATELY LEFT EMPTY: if a prefix ever
# renders wrong, its objects land somewhere obviously bogus instead of silently colliding with a
# real tenant's data. Dev↔prod isolation is doubly independent — different endpoint AND different
# bucket — so neither cluster can reach the other's objects even with a mis-substituted prefix.
VELERO_BACKUP_BUCKET: "velero" # dev: the `velero` MinIO bucket | prod: k3s-prod-backup
VELERO_BACKUP_PREFIX: "" # dev: bucket root (live layout) | prod: "velero"
CNPG_BACKUP_PATH: "s3://cnpg" # dev: the `cnpg` MinIO bucket root | prod: s3://k3s-prod-backup/barman
# --- placement ---
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
PLATFORM_NODE_SELECTOR_VALUE: "linux"
Reference in new issue View Git Blame Copy Permalink