jakman85andClaude Opus 5
0747849837
Both ZFS pools moved off the Proxmox host onto a standalone TrueNAS on 192.168.2.30 -- prod's OWN /24, so the pfSense crossing that made this knob load-bearing is gone, along with the DNAT at 192.168.2.102 it mounted through. Measured 2026-10-07: 192.168.2.102:2049 no longer answers; .30:2049 and :111 do. BULK_NFS_SERVER -> 192.168.2.30 in prod, base and dev. The dev value is marked inert (dev is decommissioned); it carries the real address only so nothing in the tree renders a host that no longer exists. THE OPERATIONAL POINT, and why this is not just a value change: a bound PersistentVolume's NFS source is immutable. Server dry-run, 2026-10-07: spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation so Flux's apply FAILS rather than updating, and the pods keep their old, dead mount. The five PVs must be deleted and recreated -- safe, because Retain never touches data and render.py emits a pre-set claimRef so each PV rebinds its own PVC. The sequence is docs/BULK-NFS-STORAGE.md section 10 and is an OPERATOR action; this commit does not perform it. Pairs with the hazard already on the page: spec.nfs.server is unvalidated at create (any string binds) and immutable at update. A wrong value is accepted silently and then cannot be corrected in place. docs/BULK-NFS-STORAGE.md: sections 1 and 5 rewritten for TrueNAS -- the measured exports, the service settings that decide whether uid 1000 survives the wire, the three stale ZFS sharenfs properties that had to be cleared, and the ACL hazard (acltype=nfsv4 + aclinherit=passthrough means one click in the ACL editor propagates to the whole future library). Section 7 rewritten now that the placeholder is gone, section 9 retargeted from dev to prod, section 10 new. Sections 2/3/4/6 are unchanged by the move. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
426 lines
32 KiB
YAML
426 lines
32 KiB
YAML
---
|
|
# cluster-vars — the overlay knob surface (R-ENV-02).
|
|
#
|
|
# This ConfigMap holds BASE DEFAULTS. Flux `postBuild.substituteFrom` injects these into every
|
|
# `${TOKEN}` in platform/ and apps/ manifests at reconcile time. dev/ and prod/ overlays STRATEGIC-
|
|
# MERGE PATCH this single ConfigMap — that patch is the *entire* difference between environments.
|
|
#
|
|
# Add a knob here (with a base default) before referencing `${KNOB}` in a component manifest, then
|
|
# give each overlay its env-scoped value. Never fork a base manifest to differ per environment.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: cluster-vars
|
|
namespace: flux-system
|
|
data:
|
|
# --- identity of the environment ---
|
|
CLUSTER_ENV: "base" # dev | prod (overlays set this)
|
|
|
|
# --- DNS / TLS ---
|
|
# THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
|
|
# the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
|
|
# ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
|
|
#
|
|
# THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
|
|
# hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
|
|
# authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
|
|
# (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
|
|
# See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
|
|
# subdomain, so the two clusters can no longer describe the same name.
|
|
#
|
|
# dev -> home.keithjackson.dev (UNCHANGED — see below)
|
|
# prod -> prod.home.keithjackson.dev
|
|
#
|
|
# DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
|
|
# `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
|
|
# relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
|
|
# the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
|
|
# (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
|
|
# GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
|
|
# it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
|
|
# tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
|
|
BASE_DOMAIN: "home.keithjackson.dev"
|
|
|
|
# ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
|
|
# EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
|
|
# so it cannot be created, updated or deleted regardless of TXT ownership.
|
|
#
|
|
# DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
|
|
# apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
|
|
# that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
|
|
# record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
|
|
# from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
|
|
#
|
|
# PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
|
|
# (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
|
|
# chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
|
|
# — it can never delete another environment's records — and the fallback is to set prod's value
|
|
# back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
|
|
EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
|
|
|
|
# ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
|
|
# neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
|
|
# record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
|
|
# `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
|
|
# one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
|
|
#
|
|
# DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
|
|
# under owner `default`; changing dev's value would make dev's own external-dns stop recognising
|
|
# them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
|
|
#
|
|
# THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
|
|
# forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
|
|
# this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
|
|
# visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
|
|
EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
|
|
|
|
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
|
|
GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
|
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
|
|
# ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
|
|
# takes the pool's first free IP, and coredns-custom answers
|
|
# *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
|
|
# hairpins through the Gateway (correct SNI -> wildcard cert).
|
|
# Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
|
|
# long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
|
|
# forgets this knob at least lands on a REAL network). A prod overlay
|
|
# MUST still set the prod LAN VIP alongside the prod L2 pool range.
|
|
|
|
# --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply -------------
|
|
# These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint:
|
|
# `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24)
|
|
# — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values
|
|
# (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the
|
|
# dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block
|
|
# before `flux bootstrap --path=clusters/prod`.
|
|
#
|
|
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
|
|
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
|
|
# (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
|
|
# platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
|
|
# tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
|
|
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
|
|
# GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
|
|
# plex-remote + pihole: PLEX_L4_VIP),
|
|
# vpn-ingress (WG_CLIENT_CIDR),
|
|
# velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
|
|
# VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
|
|
# UNSUBSTITUTED path would ship the literal string into the cluster.
|
|
#
|
|
# ADDING A KNOB IS SAFE UNDER StrictPostBuildSubstitutions ONLY BECAUSE OF THE ROOT EDGE. An
|
|
# UNDEFINED variable is a HARD Kustomization failure, so a manifest that references a knob the
|
|
# live ConfigMap does not yet carry breaks its whole tier. `clusters/base/platform.yaml` gained
|
|
# `dependsOn: [flux-system]` for exactly this: dependsOn is REVISION-gated, so no substituting
|
|
# tier can reconcile revision N until the ROOT Kustomization has APPLIED revision N — i.e. until
|
|
# the new cluster-vars key EXISTS. Every substituting consumer chains transitively through
|
|
# `platform`, so one edge covers the graph, and a new knob plus its first reference may ship in
|
|
# ONE commit. (A hand-run `kubectl apply -k clusters/<env>` bypasses that ordering entirely: apply
|
|
# the overlay BEFORE the tier that references the knob, or the tier fails until it retries.)
|
|
|
|
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
|
|
#
|
|
# *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
|
|
# 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
|
|
# control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
|
|
# live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
|
|
# own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
|
|
# Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
|
|
# outage you would notice. A virtual address that follows the control plane is the fix, and it is
|
|
# what kube-vip was stood up to provide.
|
|
#
|
|
# dev -> 192.168.1.49 (= kube_vip_address, group_vars/all.yml; base default, below)
|
|
# prod -> 192.168.2.9 (= kube_vip_address, group_vars/k3s_prod/vars.yml)
|
|
#
|
|
# THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
|
|
# unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
|
|
# move. Read it as "the API host Cilium dials", which is what it always meant.
|
|
#
|
|
# [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
|
|
# (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
|
|
# already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
|
|
# the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
|
|
# already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
|
|
# block at the top of group_vars/all.yml.
|
|
#
|
|
# Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
|
|
# in both envs, so it is NOT a knob.
|
|
#
|
|
# TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
|
|
# both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
|
|
# https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
|
|
K3S_API_HOST: "192.168.1.49"
|
|
|
|
# CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
|
|
# L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
|
|
# tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in
|
|
# practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's
|
|
# first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and
|
|
# the flagged block in clusters/prod/kustomization.yaml.)
|
|
CILIUM_LB_POOL_START: "192.168.1.53"
|
|
CILIUM_LB_POOL_STOP: "192.168.1.60"
|
|
|
|
# Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the
|
|
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
|
|
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
|
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
|
# PiHole's DNS listener: udp/53 + tcp/53 for LAN CLIENTS (apps/wave-0/pihole-dns.yaml). It is a
|
|
# THIRD pinned L4 VIP for the same reason as the two above — the `shared` Gateway is an L7 HTTPS
|
|
# terminator and cannot express DNS at all (udp/53 especially). It must NOT be ${GATEWAY_VIP}:
|
|
# LAN clients are pointed at this address as their resolver, and it has to stay put across
|
|
# reschedules, so it is pinned by lbipam.cilium.io/ips rather than taken from the pool head.
|
|
#
|
|
# WHY A SEPARATE ADDRESS AND NOT THE GATEWAY VIP: every `<svc>.${BASE_DOMAIN}` name resolves TO
|
|
# the Gateway VIP. Serving DNS from the same address would be legal but conflates "the thing that
|
|
# answers where services live" with "a service", and it would put LAN resolution behind the
|
|
# Gateway Service's own lifecycle. Keep them separate.
|
|
#
|
|
# dev -> 192.168.1.56 (base default; next free after crafty .55 in dev's .53-.60 pool)
|
|
# prod -> 192.168.2.23 (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
|
|
PIHOLE_DNS_VIP: "192.168.1.56" # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
|
|
# Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
|
|
# A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
|
|
# cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
|
|
# server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
|
|
# measured working) cannot put the server on a client's connection list at all.
|
|
#
|
|
# PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
|
|
# address by hand. A VIP taken from the pool head would silently move on a reschedule and break
|
|
# remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
|
|
#
|
|
# dev -> 192.168.1.57 (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
|
|
# prod -> 192.168.2.24 (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
|
|
PLEX_L4_VIP: "192.168.1.57" # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
|
|
# SECOND CONSUMER since 2026-10-04: catalog/pihole.yaml reads this knob to answer
|
|
# plex-vip.${BASE_DOMAIN} on the LAN, which is the local half of the single hostname Plex
|
|
# advertises to plex.tv. Without that per-host answer the zone wildcard sends the name to
|
|
# ${GATEWAY_VIP}, where nothing listens on 32400. So this value is now typed into THREE places
|
|
# that must agree: this knob, the WAN port-forward on the SOHO router (off-repo), and PiHole's
|
|
# answer -- and only the first two were true before.
|
|
|
|
|
|
# The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
|
|
# peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
|
|
# from `world` is an open resolver and an amplification reflector, so the allow is scoped to this
|
|
# CIDR, not to Cilium's `world` entity the way the unifi/crafty game-port allows are. It is the
|
|
# same /24 the node IPs and ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} come out of.
|
|
#
|
|
# dev -> 192.168.1.0/24 (Proxmox LAN; nodes .50/.51/.52)
|
|
# prod -> 192.168.2.0/24 (prod LAN; nodes .10/.11/.12)
|
|
LAN_CIDR: "192.168.1.0/24" # ingress scope for the PiHole DNS listener
|
|
|
|
# --- Remote-access VPN client subnet (platform/vpn-ingress, WireGuard) --------------------------
|
|
# The tunnel subnet handed to REMOTE VPN clients. `.1` is the server (the hostNetwork DaemonSet,
|
|
# on every node); `.2`-`.254` are devices, allocated one /32 each in
|
|
# platform/vpn-ingress/peers.yaml. Consumed twice in the pod's config-assembly script: to derive
|
|
# the server's own address, and as the `-s` match of the MASQUERADE rule that makes LAN replies
|
|
# routable.
|
|
#
|
|
# IT IS A KNOB RATHER THAN A LITERAL FOR ONE REASON: it is the only address range in this cluster
|
|
# that must be checked against EVERY OTHER range at once, so it belongs where those are declared.
|
|
# 10.90.0.0/24 was chosen to collide with none of them:
|
|
# 10.0.0.0/16 Cilium cluster-pool pod CIDR (${CLUSTER_POD_CIDR})
|
|
# 10.43.0.0/16 k3s service CIDR (measured live 2026-10-05: kubernetes svc
|
|
# clusterIP 10.43.0.1)
|
|
# 10.72.192.0/18 the PIA tunnel's pushed addresses (platform/vpn-egress)
|
|
# 192.168.1.0/24 dev LAN / pfSense LAN (${LAN_CIDR} base default)
|
|
# 192.168.2.0/24 prod LAN (${LAN_CIDR} on prod)
|
|
#
|
|
# NO PER-ENVIRONMENT OVERRIDE, and that is deliberate rather than an omission: unlike every VIP
|
|
# above, this range is NOT drawn from the LAN. It is private to the tunnel, so the same value is
|
|
# correct in both environments and a prod patch would be a no-op that implies otherwise.
|
|
#
|
|
# IF YOU CHANGE IT, every already-issued client profile breaks (its `Address` is from this range)
|
|
# and must be re-rendered — so treat it as fixed once the first device is enrolled.
|
|
WG_CLIENT_CIDR: "10.90.0.0/24" # remote VPN clients; .1 = server. NOT a LAN range
|
|
|
|
# --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
|
|
# THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
|
|
# (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
|
|
# (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
|
|
# `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
|
|
# declared in the ServiceDefinition, because the path is identical in every environment and only
|
|
# the ADDRESS is not.
|
|
#
|
|
# IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
|
|
# OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
|
|
# vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
|
|
# only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
|
|
# TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
|
|
# values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
|
|
# 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
|
|
# this stays a knob only because the renderer reads it per-environment.
|
|
#
|
|
# *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
|
|
# immutable — measured by server dry-run 2026-10-07:
|
|
# spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
|
|
# so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
|
|
# deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
|
|
BULK_NFS_SERVER: "192.168.2.30"
|
|
|
|
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
|
|
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
|
|
# nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
|
|
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR —
|
|
# and, measured 2026-09-05 from a pod INSIDE prod, it CANNOT reach it anyway (prod is on the SOHO
|
|
# network 192.168.2.0/24, which is pfSense's WAN, while this MinIO sits inside pfSense's LAN:
|
|
# `curl http://192.168.1.9:9000/minio/health/live` timed out after 15s). Prod uses Cloudflare R2.
|
|
S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
|
|
|
|
# S3 region. Every S3 client REQUIRES one; MinIO ignores whatever it is; Cloudflare R2 demands
|
|
# exactly `auto` (its `head-bucket` reports the physical hint `WNAM`, which is NOT a usable region
|
|
# string). This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-
|
|
# accident for MinIO and wrong for anything else, which is why it is a knob now. The base default
|
|
# is dev's LIVE value, deliberately: dev's Velero has been signing requests with `us-east-1`
|
|
# against that MinIO since bootstrap, and this repo's rule is that base == the known-good dev
|
|
# value. Prod overrides to `auto`.
|
|
S3_BACKUP_REGION: "us-east-1"
|
|
|
|
# --- storage (ADR-0003) ---
|
|
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
|
# Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
|
|
# replica is written into. It must be the path the NODE's own data disk is actually mounted at —
|
|
# roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
|
|
# fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
|
|
# Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
|
|
# working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
|
|
# on it — changing dev's value would orphan every replica, so the dev overlay restates it
|
|
# verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
|
|
# dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
|
|
# neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
|
|
LONGHORN_DATA_PATH: "/var/lib/longhorn"
|
|
|
|
# --- observability sizing (docs/10) ---
|
|
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
|
|
MONITORING_PROMETHEUS_PVC_SIZE: "10Gi"
|
|
# SIZE-based retention cap for Prometheus (Prometheus *iB units, NOT the K8s `Gi` of the PVC knob).
|
|
# MUST stay below MONITORING_PROMETHEUS_PVC_SIZE — leaves headroom for the WAL + head + compaction
|
|
# scratch (retentionSize counts only persisted blocks). Prevents a busy TSDB from filling /prometheus
|
|
# (a full disk wedges all WAL commits and makes every absent()/==0 alert fire falsely). Base 10Gi PVC.
|
|
MONITORING_PROMETHEUS_RETENTION_SIZE: "8GiB"
|
|
MONITORING_LOKI_PVC_SIZE: "10Gi"
|
|
|
|
# --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture;
|
|
# the dev overlay softens them for bootstrap (Audit / Ignore / 1) so a cold cluster can
|
|
# come up before policies are proven and admission is HA. ---
|
|
KYVERNO_VALIDATION_ACTION: "Enforce" # dev: Audit (observe before enforcing)
|
|
KYVERNO_FORCE_FAILURE_POLICY_IGNORE: "false" # dev: "true" (chart forceFailurePolicyIgnore = fail-open)
|
|
KYVERNO_ADMISSION_REPLICAS: "3" # dev: 1 (no HA needed to iterate)
|
|
|
|
# --- egress / VPN (Phase 3, ADR-0015) — node-level host OpenVPN tunnel steered by Cilium Egress
|
|
# Gateway (platform/vpn-egress/egress-steering.yaml). No in-cluster gateway pod. ---
|
|
PIA_REGION: "SE Stockholm" # PF-capable PIA region (Sweden). Under ADR-0015 the tunnel is
|
|
# host-side (Ansible vault / the node's .ovpn per docs/07), so this
|
|
# is now a reference default only — retained, not substituted into
|
|
# any manifest. ("SE Stockholm" is PF-capable; NOT "SE Streaming
|
|
# Optimized" / a US region, which lack PF.)
|
|
CLUSTER_POD_CIDR: "10.0.0.0/16" # Documents the Cilium cluster-pool pod CIDR that the egress-steering
|
|
# CiliumEgressGatewayPolicy carves out of the tunnel via excludedCIDRs
|
|
# (kept on the LOCAL datapath, never SNAT'd out tun0). The policy
|
|
# currently excludes the whole /8 RFC1918 range; this /16 records the
|
|
# actual pod allocations (Cilium assigns /24 per node from 10.0.0.0 up).
|
|
|
|
# --- identity / auth (ADR-0014) ---
|
|
KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
OAUTH2_PROXY_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
|
|
# --- backup / DR (ADR-0003, R-STATE-03) ---
|
|
VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ
|
|
VELERO_RETENTION: "168h" # dev short | prod long
|
|
|
|
# ==========================================================================================
|
|
# OBJECT-STORE TENANCY — the three knobs that keep two clusters and two backup tools from
|
|
# writing into each other's data. THESE ARE SAFETY KNOBS, not preferences.
|
|
# ==========================================================================================
|
|
# Nothing about the object layout was per-environment before this: Velero's bucket was the
|
|
# hardcoded literal `velero` with no prefix, and every CNPG Cluster used the hardcoded
|
|
# `s3://cnpg` with barman's `serverName` defaulting to the CLUSTER NAME — and BOTH environments
|
|
# have a keycloak-db, a nextcloud-db and a semaphore-db. Point two clusters at one bucket under
|
|
# that layout and:
|
|
# (a) each cluster's Velero ADOPTS the other's backups and TTL-DELETES them on retention; and
|
|
# (b) two different Postgres servers stream WAL into ONE Barman prefix, which does not merely
|
|
# mix files — it CORRUPTS THE CHAIN and destroys PITR for both. That is the worse half:
|
|
# dev's DR restore is PROVEN (R-STATE-03) and this would silently un-prove it.
|
|
#
|
|
# WHY THERE IS NO SINGLE `S3_BACKUP_BUCKET`. The two consumers take differently-SHAPED values:
|
|
# the Velero chart wants `bucket` and `prefix` as separate fields, while CNPG's
|
|
# barmanObjectStore.destinationPath is ONE full s3:// URL whose first path segment IS the bucket.
|
|
# CNPG's bucket therefore already lives inside CNPG_BACKUP_PATH; adding a second knob that had to
|
|
# hold the same bucket name a second time would be a rotation waiting to drift — the exact class
|
|
# of duplication this repo has been bitten by. Three knobs, each shaped for its own consumer, and
|
|
# no two of them can contradict each other.
|
|
#
|
|
# BASE = DEV'S LIVE LAYOUT, BYTE-FOR-BYTE, AND MUST STAY THAT WAY. Dev's existing backups are at
|
|
# `s3://velero/` (bucket root, no prefix) and `s3://cnpg/<clusterName>/` on that MinIO. Giving dev
|
|
# a prefix, or renaming its buckets, would ORPHAN the very backups its proven restore depends on —
|
|
# the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default` and its LONGHORN_DATA_PATH stays
|
|
# /var/lib/longhorn. Verified: with an EMPTY prefix the Velero chart omits `objectStorage.prefix`
|
|
# from the rendered BackupStorageLocation entirely (helm template, chart 12.1.0), so dev's live
|
|
# BSL is unchanged by this parameterization rather than merely equivalent to it.
|
|
#
|
|
# PROD, which has no history to preserve, gets ONE bucket with TWO disjoint top-level prefixes:
|
|
#
|
|
# s3://k3s-prod-backup/velero/… Velero (kopia repo + backup metadata)
|
|
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG WAL + base backups
|
|
#
|
|
# The operator created exactly one R2 bucket, so velero-vs-barman separation has to come from the
|
|
# prefix rather than the bucket. The BUCKET ROOT IS DELIBERATELY LEFT EMPTY: if a prefix ever
|
|
# renders wrong, its objects land somewhere obviously bogus instead of silently colliding with a
|
|
# real tenant's data. Dev↔prod isolation is doubly independent — different endpoint AND different
|
|
# bucket — so neither cluster can reach the other's objects even with a mis-substituted prefix.
|
|
VELERO_BACKUP_BUCKET: "velero" # dev: the `velero` MinIO bucket | prod: k3s-prod-backup
|
|
VELERO_BACKUP_PREFIX: "" # dev: bucket root (live layout) | prod: "velero"
|
|
CNPG_BACKUP_PATH: "s3://cnpg" # dev: the `cnpg` MinIO bucket root | prod: s3://k3s-prod-backup/barman
|
|
|
|
# ------------------------------------------------------------------------------------------------
|
|
# WHICH CREDENTIAL SECRET THE BACKUP CONSUMERS READ (added 2026-09-07 — the per-env credential split)
|
|
# ------------------------------------------------------------------------------------------------
|
|
# The three knobs above say WHERE the backups go. This one says WHICH KEY signs the requests, and
|
|
# it exists because the endpoint moved per-environment while the credential did not.
|
|
#
|
|
# THE BUG IT FIXES. The four backup-credential Secrets — velero-cloud.sops.yaml and the three
|
|
# <db>-backup-credentials.sops.yaml — are SHARED BASE manifests: platform/ and apps/ are rendered
|
|
# by the same Flux tiers on both clusters, and cluster-vars substitution is the ONLY thing that
|
|
# differs (R-ENV-02). So writing prod's R2 token into those four files did not GIVE prod a
|
|
# credential, it REPLACED DEV'S: dev's Velero and all three dev CNPG Clusters would have started
|
|
# signing MinIO requests with an R2 key id. For Velero that is a broken BSL; for CNPG it is worse,
|
|
# because `archive_mode=on` means Postgres REFUSES TO RECYCLE a WAL segment it could not archive,
|
|
# so a 403 loop is a monotonically filling pgdata PVC ending in a PANIC — the same failure prod
|
|
# spent its whole bring-up in with a placeholder endpoint. One shared Secret also means rotating
|
|
# either environment's key silently breaks the other; that is the structural reason for the split,
|
|
# independent of dev's eventual decommission.
|
|
#
|
|
# THE MECHANISM. Each credential is now TWO Secret objects, both applied to BOTH clusters (they are
|
|
# base resources; nothing here forks a manifest per environment). The consumer names one:
|
|
#
|
|
# velero: credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
|
# CNPG: s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
|
#
|
|
# ONE knob, not four, ON PURPOSE. Four independent secret-name knobs could be updated three-
|
|
# quarters of the way, leaving one consumer pointed at the OTHER environment's key with nothing
|
|
# rendering wrong — and StrictPostBuildSubstitutions cannot catch that, because it only fails on an
|
|
# UNDEFINED variable, never on a defined-but-stale one. With a single suffix the four move together
|
|
# or not at all. The cost is that the rendered Secret name is not greppable from the consumer; the
|
|
# names are spelled out in full in each kustomization.yaml beside the resources.
|
|
#
|
|
# THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names,
|
|
# which are the objects that already exist and that the live dev cluster is running. A default
|
|
# naming a Secret that does not exist would be worse than the shared-Secret state it replaces:
|
|
# Velero's pod would sit in ContainerCreating forever on a missing `existingSecret` volume, and a
|
|
# CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — i.e. the disk-full clock
|
|
# again — so an unresolvable default breaks the referencing cluster HARDER than a wrong key does.
|
|
# Empty is also the value dev must carry, so base and dev agree, exactly like VELERO_BACKUP_PREFIX.
|
|
#
|
|
# An EMPTY VALUE IS SAFE UNDER StrictPostBuildSubstitutions: strict mode fails on a variable that is
|
|
# not DEFINED, not on one defined as "". VELERO_BACKUP_PREFIX above has shipped empty since the
|
|
# gate was enabled and proves it. Both overlays state this knob explicitly — dev "" and prod
|
|
# "-prod" — so neither inherits it silently.
|
|
#
|
|
# ADDING A NEW BACKUP CREDENTIAL: create BOTH halves (`x-credentials.sops.yaml` and
|
|
# `x-credentials-prod.sops.yaml`), list both in the kustomization, and reference
|
|
# `x-credentials${BACKUP_CREDENTIALS_SUFFIX}`. Never a single shared Secret again.
|
|
BACKUP_CREDENTIALS_SUFFIX: "" # dev: "" (MinIO creds) | prod: "-prod" (R2 creds)
|
|
|
|
# --- placement ---
|
|
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
|
|
PLATFORM_NODE_SELECTOR_VALUE: "linux"
|