262 lines
19 KiB
YAML
262 lines
19 KiB
YAML
---
|
|
# cluster-vars — the overlay knob surface (R-ENV-02).
|
|
#
|
|
# This ConfigMap holds BASE DEFAULTS. Flux `postBuild.substituteFrom` injects these into every
|
|
# `${TOKEN}` in platform/ and apps/ manifests at reconcile time. dev/ and prod/ overlays STRATEGIC-
|
|
# MERGE PATCH this single ConfigMap — that patch is the *entire* difference between environments.
|
|
#
|
|
# Add a knob here (with a base default) before referencing `${KNOB}` in a component manifest, then
|
|
# give each overlay its env-scoped value. Never fork a base manifest to differ per environment.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: cluster-vars
|
|
namespace: flux-system
|
|
data:
|
|
# --- identity of the environment ---
|
|
CLUSTER_ENV: "base" # dev | prod (overlays set this)
|
|
|
|
# --- DNS / TLS ---
|
|
# THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
|
|
# the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
|
|
# ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
|
|
#
|
|
# THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
|
|
# hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
|
|
# authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
|
|
# (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
|
|
# See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
|
|
# subdomain, so the two clusters can no longer describe the same name.
|
|
#
|
|
# dev -> home.keithjackson.dev (UNCHANGED — see below)
|
|
# prod -> prod.home.keithjackson.dev
|
|
#
|
|
# DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
|
|
# `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
|
|
# relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
|
|
# the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
|
|
# (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
|
|
# GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
|
|
# it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
|
|
# tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
|
|
BASE_DOMAIN: "home.keithjackson.dev"
|
|
|
|
# ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
|
|
# EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
|
|
# so it cannot be created, updated or deleted regardless of TXT ownership.
|
|
#
|
|
# DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
|
|
# apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
|
|
# that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
|
|
# record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
|
|
# from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
|
|
#
|
|
# PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
|
|
# (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
|
|
# chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
|
|
# — it can never delete another environment's records — and the fallback is to set prod's value
|
|
# back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
|
|
EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
|
|
|
|
# ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
|
|
# neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
|
|
# record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
|
|
# `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
|
|
# one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
|
|
#
|
|
# DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
|
|
# under owner `default`; changing dev's value would make dev's own external-dns stop recognising
|
|
# them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
|
|
#
|
|
# THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
|
|
# forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
|
|
# this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
|
|
# visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
|
|
EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
|
|
|
|
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
|
|
GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
|
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
|
|
# ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
|
|
# takes the pool's first free IP, and coredns-custom answers
|
|
# *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
|
|
# hairpins through the Gateway (correct SNI -> wildcard cert).
|
|
# Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
|
|
# long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
|
|
# forgets this knob at least lands on a REAL network). A prod overlay
|
|
# MUST still set the prod LAN VIP alongside the prod L2 pool range.
|
|
|
|
# --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply -------------
|
|
# These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint:
|
|
# `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24)
|
|
# — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values
|
|
# (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the
|
|
# dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block
|
|
# before `flux bootstrap --path=clusters/prod`.
|
|
#
|
|
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
|
|
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
|
|
# (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
|
|
# velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
|
|
# ship the literal string into the cluster.
|
|
|
|
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
|
|
#
|
|
# *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
|
|
# 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
|
|
# control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
|
|
# live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
|
|
# own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
|
|
# Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
|
|
# outage you would notice. A virtual address that follows the control plane is the fix, and it is
|
|
# what kube-vip was stood up to provide.
|
|
#
|
|
# dev -> 192.168.1.49 (= kube_vip_address, group_vars/all.yml; base default, below)
|
|
# prod -> 192.168.2.9 (= kube_vip_address, group_vars/k3s_prod/vars.yml)
|
|
#
|
|
# THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
|
|
# unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
|
|
# move. Read it as "the API host Cilium dials", which is what it always meant.
|
|
#
|
|
# [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
|
|
# (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
|
|
# already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
|
|
# the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
|
|
# already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
|
|
# block at the top of group_vars/all.yml.
|
|
#
|
|
# Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
|
|
# in both envs, so it is NOT a knob.
|
|
#
|
|
# TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
|
|
# both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
|
|
# https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
|
|
K3S_API_HOST: "192.168.1.49"
|
|
|
|
# CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
|
|
# L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
|
|
# tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in
|
|
# practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's
|
|
# first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and
|
|
# the flagged block in clusters/prod/kustomization.yaml.)
|
|
CILIUM_LB_POOL_START: "192.168.1.53"
|
|
CILIUM_LB_POOL_STOP: "192.168.1.60"
|
|
|
|
# Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the
|
|
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
|
|
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
|
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
|
|
|
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05). Consumed by BOTH backup paths: every CNPG
|
|
# Cluster's barmanObjectStore.endpointURL (keycloak-db, nextcloud-db, semaphore-db) and Velero's
|
|
# backupStorageLocation s3Url. Full URL including scheme (and port, if not the scheme default).
|
|
#
|
|
# OPERATOR DECISION 2026-09-05: **Cloudflare R2**, replacing the on-Proxmox MinIO — "for the time
|
|
# being until a better on-prem backup solution is created". Two reasons beyond cost:
|
|
# 1. The k3s prod cluster is on the SOHO network (pfSense's WAN) and CANNOT reach the MinIO inside
|
|
# pfSense's LAN — measured, from a pod: connection timed out. R2 needs no firewall change.
|
|
# 2. That MinIO runs on the SAME Proxmox host as dev's VMs, so dev's "off-cluster" backup shared a
|
|
# physical failure domain with dev. One host loss took the cluster AND its backups. R2 is
|
|
# genuinely off-site.
|
|
# Measured to justify the cost: dev's whole estate is 4.0 GiB Velero (6,264 objects, 7d retention) +
|
|
# 211 MiB CNPG (1,755 objects) — pennies per month, and R2 egress is free, so restore REHEARSALS
|
|
# stay free and R-STATE-03 has no cost disincentive to stay proven.
|
|
#
|
|
# The MinIO copy is NOT deleted — it stays as a historical/local copy (3-2-1). Restoring from it
|
|
# means pointing this knob back at http://192.168.1.9:9000 temporarily.
|
|
S3_BACKUP_ENDPOINT: "https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com"
|
|
|
|
# S3 region. Required by every S3 client, IGNORED by MinIO, and must be exactly `auto` for R2.
|
|
# This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-accident for
|
|
# MinIO and wrong for R2, which is why it is a knob now.
|
|
S3_BACKUP_REGION: "auto"
|
|
|
|
# --- storage (ADR-0003) ---
|
|
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
|
# Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
|
|
# replica is written into. It must be the path the NODE's own data disk is actually mounted at —
|
|
# roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
|
|
# fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
|
|
# Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
|
|
# working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
|
|
# on it — changing dev's value would orphan every replica, so the dev overlay restates it
|
|
# verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
|
|
# dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
|
|
# neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
|
|
LONGHORN_DATA_PATH: "/var/lib/longhorn"
|
|
|
|
# --- observability sizing (docs/10) ---
|
|
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
|
|
MONITORING_PROMETHEUS_PVC_SIZE: "10Gi"
|
|
# SIZE-based retention cap for Prometheus (Prometheus *iB units, NOT the K8s `Gi` of the PVC knob).
|
|
# MUST stay below MONITORING_PROMETHEUS_PVC_SIZE — leaves headroom for the WAL + head + compaction
|
|
# scratch (retentionSize counts only persisted blocks). Prevents a busy TSDB from filling /prometheus
|
|
# (a full disk wedges all WAL commits and makes every absent()/==0 alert fire falsely). Base 10Gi PVC.
|
|
MONITORING_PROMETHEUS_RETENTION_SIZE: "8GiB"
|
|
MONITORING_LOKI_PVC_SIZE: "10Gi"
|
|
|
|
# --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture;
|
|
# the dev overlay softens them for bootstrap (Audit / Ignore / 1) so a cold cluster can
|
|
# come up before policies are proven and admission is HA. ---
|
|
KYVERNO_VALIDATION_ACTION: "Enforce" # dev: Audit (observe before enforcing)
|
|
KYVERNO_FORCE_FAILURE_POLICY_IGNORE: "false" # dev: "true" (chart forceFailurePolicyIgnore = fail-open)
|
|
KYVERNO_ADMISSION_REPLICAS: "3" # dev: 1 (no HA needed to iterate)
|
|
|
|
# --- egress / VPN (Phase 3, ADR-0015) — node-level host OpenVPN tunnel steered by Cilium Egress
|
|
# Gateway (platform/vpn-egress/egress-steering.yaml). No in-cluster gateway pod. ---
|
|
PIA_REGION: "SE Stockholm" # PF-capable PIA region (Sweden). Under ADR-0015 the tunnel is
|
|
# host-side (Ansible vault / the node's .ovpn per docs/07), so this
|
|
# is now a reference default only — retained, not substituted into
|
|
# any manifest. ("SE Stockholm" is PF-capable; NOT "SE Streaming
|
|
# Optimized" / a US region, which lack PF.)
|
|
CLUSTER_POD_CIDR: "10.0.0.0/16" # Documents the Cilium cluster-pool pod CIDR that the egress-steering
|
|
# CiliumEgressGatewayPolicy carves out of the tunnel via excludedCIDRs
|
|
# (kept on the LOCAL datapath, never SNAT'd out tun0). The policy
|
|
# currently excludes the whole /8 RFC1918 range; this /16 records the
|
|
# actual pod allocations (Cilium assigns /24 per node from 10.0.0.0 up).
|
|
|
|
# --- identity / auth (ADR-0014) ---
|
|
KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
OAUTH2_PROXY_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
|
|
# --- backup / DR (ADR-0003, R-STATE-03) ---
|
|
VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ
|
|
VELERO_RETENTION: "168h" # dev short | prod long
|
|
|
|
# BACKUP TENANCY — the two knobs that keep two clusters sharing ONE MinIO from destroying each
|
|
# other's backups. Added 2026-09-05 when the operator ruled prod uses the SAME endpoint as dev.
|
|
#
|
|
# Sharing an endpoint is fine. Sharing a PATH is not, and nothing here was per-environment before:
|
|
#
|
|
# VELERO — bucket `velero`, no prefix, and BOTH clusters run schedules named `daily` and
|
|
# `daily-resources-only`. Each cluster's Velero SYNCS backups it finds in the BSL into
|
|
# itself as Backup CRs and GARBAGE-COLLECTS them when their TTL expires. Prod would
|
|
# therefore adopt dev's backups and delete them on schedule.
|
|
# CNPG — destinationPath `s3://cnpg` with CNPG appending serverName, which defaults to the
|
|
# CLUSTER NAME. Both environments have `keycloak-db`, `nextcloud-db`, `semaphore-db`,
|
|
# so prod's Postgres would archive WAL into the exact prefix dev's does. Two servers
|
|
# interleaving WAL into one Barman stream CORRUPTS THE CHAIN and destroys PITR — and
|
|
# dev's DR restore is PROVEN (R-STATE-03). This is the worst of the two.
|
|
#
|
|
# Both environments are now NAMESPACED SYMMETRICALLY, which was not possible an hour ago and is
|
|
# worth explaining, because the earlier reasoning was the opposite:
|
|
#
|
|
# While the target was the MinIO, dev's base value HAD to be the bucket root — its existing
|
|
# backups lived there and giving dev a prefix would have ORPHANED the very backups its proven
|
|
# restore depends on (the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default`).
|
|
#
|
|
# Moving to R2 starts BOTH clusters on an EMPTY bucket, so there is no history to orphan and no
|
|
# reason to keep the asymmetry. `dev` and `prod` is clearer, greppable, and leaves the bucket root
|
|
# free — so an unset/misrendered prefix lands somewhere obviously wrong instead of silently
|
|
# colliding with a real environment's backups.
|
|
#
|
|
# *** THESE TWO VALUES AND S3_BACKUP_ENDPOINT MUST CHANGE TOGETHER. *** Setting a `dev` prefix while
|
|
# the endpoint is still the MinIO would orphan dev's existing MinIO backups for no benefit. They are
|
|
# one change; do not split the commit.
|
|
VELERO_BACKUP_PREFIX: "dev" # dev: "dev" | prod: "prod"
|
|
CNPG_BACKUP_PATH: "s3://cnpg/dev" # dev: s3://cnpg/dev | prod: s3://cnpg/prod
|
|
|
|
# --- placement ---
|
|
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
|
|
PLATFORM_NODE_SELECTOR_VALUE: "linux"
|