jakman85andClaude Opus 5
6f61c9622d
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."
THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.
SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join
target and a tls-san. infra/tofu/outputs.tf GENERATES this
name, so renaming would ripple into tofu and the runbooks.
k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium
consumes. Derived, not restated, so an environment overrides
ONE line and the VIP, the SAN and the CNI endpoint move
together. Prod needed no new override.
THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.
Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".
VERIFIED BY ME, not just reported:
renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP)
cilium template consumes k3s_apiserver_endpoint, not k3s_api_host
cold join still `--server https://{{ k3s_api_host }}:6443` — the founder
order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
gate full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.
ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.
APPLY ORDER, and this is the operator's step — nothing was applied:
PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
incident), it will not proceed past an unready pod, existing datapath survives an
agent restart, and .9 is already live before the change lands.
DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.
NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.
Refs R-PLAT-01, R-NET-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
296 lines
21 KiB
YAML
296 lines
21 KiB
YAML
---
|
|
# cluster-vars — the overlay knob surface (R-ENV-02).
|
|
#
|
|
# This ConfigMap holds BASE DEFAULTS. Flux `postBuild.substituteFrom` injects these into every
|
|
# `${TOKEN}` in platform/ and apps/ manifests at reconcile time. dev/ and prod/ overlays STRATEGIC-
|
|
# MERGE PATCH this single ConfigMap — that patch is the *entire* difference between environments.
|
|
#
|
|
# Add a knob here (with a base default) before referencing `${KNOB}` in a component manifest, then
|
|
# give each overlay its env-scoped value. Never fork a base manifest to differ per environment.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: cluster-vars
|
|
namespace: flux-system
|
|
data:
|
|
# --- identity of the environment ---
|
|
CLUSTER_ENV: "base" # dev | prod (overlays set this)
|
|
|
|
# --- DNS / TLS ---
|
|
# THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
|
|
# the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
|
|
# ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
|
|
#
|
|
# THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
|
|
# hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
|
|
# authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
|
|
# (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
|
|
# See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
|
|
# suffix, so the two clusters can no longer describe the same name.
|
|
#
|
|
# dev -> dev.home.keithjackson.dev (a subdomain; = this base default)
|
|
# prod -> home.keithjackson.dev (THE APEX — prod IS the apex, no `prod.` label)
|
|
#
|
|
# OPERATOR DECISION 2026-09-04 (supersedes the interim split committed earlier the same day, which
|
|
# had prod on `prod.home.` with apex CNAMEs pointing at it). PROD IS THE APEX. That deletes the
|
|
# whole CNAME layer — `<svc>.home.keithjackson.dev` is prod's real name, not an alias — and it is
|
|
# DEV that carries a label. Convention is unchanged from GATEWAY_VIP / LONGHORN_DATA_PATH: the base
|
|
# default is the DEV value (dev restates it verbatim) and prod overrides it. What changed is which
|
|
# string dev holds.
|
|
#
|
|
# DEV'S VALUE IS A MIGRATION, NOT A VARIABLE CHANGE. The live dev cluster is serving the APEX right
|
|
# now: its wildcard cert, every public DNS record, its Keycloak issuer, ~12 OIDC clients' redirect
|
|
# URIs, kubectl SSO, and gitea.home.keithjackson.dev — which IS this repo's GitOps source AND the
|
|
# push target. Flipping this value on dev without first standing up the new host set STRANDS DEV:
|
|
# Flux's GitRepository url moves to gitea.dev.home.keithjackson.dev, which has no DNS record and no
|
|
# cert, so dev stops reconciling and cannot be fixed by pushing. ORDERED CUTOVER, operator present:
|
|
# infra/PROD-PROVISIONING.md §10.
|
|
#
|
|
# DEV MUST VACATE THE APEX BEFORE PROD PUBLISHES THERE. Both clusters describing
|
|
# `*.home.keithjackson.dev` at once is the 2026-09-04 incident precondition, re-created.
|
|
BASE_DOMAIN: "dev.home.keithjackson.dev"
|
|
|
|
# ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
|
|
# EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
|
|
# so it cannot be created, updated or deleted regardless of TXT ownership.
|
|
#
|
|
# BOTH ENVIRONMENTS USE THE CLOUDFLARE **ZONE**, `keithjackson.dev`. This value is NOT a per-
|
|
# environment isolation knob and must not be narrowed to either environment's BASE_DOMAIN.
|
|
#
|
|
# FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) — the Cloudflare provider uses --domain-filter for ZONE
|
|
# SELECTION as well as name filtering, so a filter that is a STRICT SUBDOMAIN of the hosted zone
|
|
# matches NO zone: ExternalDNS generates the record and then skips it with "no hosted zone matching
|
|
# record DNS Name", publishing NOTHING. This has been tried and reverted more than once (an agent
|
|
# narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in review). DO NOT "harden"
|
|
# it — the change is not defence in depth, it is a total outage of DNS publication.
|
|
#
|
|
# WHERE ISOLATION ACTUALLY COMES FROM, and it is sufficient:
|
|
# (1) the two environments request DISJOINT hostname sets (dev.home.… vs the apex), and
|
|
# (2) EXTERNALDNS_TXT_OWNER_ID differs, so neither registry ever claims the other's records.
|
|
# That pairing is what the 2026-09-04 incident needed; a domain filter cannot provide it, because
|
|
# both environments live in the one zone either way.
|
|
EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
|
|
|
|
# ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
|
|
# neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
|
|
# record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
|
|
# `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
|
|
# one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
|
|
#
|
|
# dev -> `dev` prod -> `prod` base default -> a non-colliding sentinel (below)
|
|
#
|
|
# DEV IS `dev`, NOT `default` (changed 2026-09-04 with the move to dev.home.keithjackson.dev). Until
|
|
# this migration dev HAD to render `default`, because its LIVE apex records were already registered
|
|
# in Cloudflare's TXT registry under that owner and renaming would have orphaned every one of them.
|
|
# THAT CONSTRAINT NO LONGER BINDS: dev is publishing an ENTIRELY NEW record set under
|
|
# *.dev.home.keithjackson.dev, with no pre-existing registry entries to recognise, so the owner id
|
|
# is now free to be chosen on its merits. Two reasons to take `dev`:
|
|
# - `default` is THE VALUE YOU GET BY FORGETTING (it is the external-dns chart default, and both
|
|
# clusters silently shared it, which IS the incident). An environment whose identity is the
|
|
# forget-value can never be told apart from a misconfigured newcomer. No environment should
|
|
# hold it.
|
|
# - It makes vacating the apex an EXPLICIT, ORDERED, VERIFIABLE step. Dev's old apex records stay
|
|
# owned by `default`, which no environment now claims, so nothing deletes them implicitly on
|
|
# some future reconcile — they are retired deliberately, and prod's publish is gated on that
|
|
# retirement being confirmed in Cloudflare (infra/PROD-PROVISIONING.md §10) rather than on a
|
|
# race between two reconcile loops.
|
|
#
|
|
# PROD STAYS `prod` AND MUST NOT BECOME `default`, even though prod now serves the apex names that
|
|
# dev's `default`-owned records currently occupy. Giving prod `default` would hand prod ownership of
|
|
# the LIVE dev cluster's records while dev is still serving them, and under `policy: sync` prod
|
|
# would delete the ones no prod source claims. That is the 2026-09-04 incident, re-armed, with the
|
|
# roles swapped. Prod inherits the apex NAMES; it must never inherit the apex OWNER ID.
|
|
#
|
|
# THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default` and not any real
|
|
# environment's id. A new environment that forgets this knob must not silently inherit another's
|
|
# identity — that is precisely the incident. With this sentinel such a cluster owns nothing, so it
|
|
# can DELETE nothing; it would publish under a visibly wrong owner id (grep the TXT records)
|
|
# instead of destroying a live environment's DNS.
|
|
EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
|
|
|
|
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
|
|
GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
|
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
|
|
# ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
|
|
# takes the pool's first free IP, and coredns-custom answers
|
|
# *.${BASE_DOMAIN} with THIS so pod -> public-hostname traffic
|
|
# hairpins through the Gateway (correct SNI -> wildcard cert).
|
|
# Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
|
|
# long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
|
|
# forgets this knob at least lands on a REAL network). A prod overlay
|
|
# MUST still set the prod LAN VIP alongside the prod L2 pool range.
|
|
|
|
# --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply -------------
|
|
# These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint:
|
|
# `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24)
|
|
# — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values
|
|
# (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the
|
|
# dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block
|
|
# before `flux bootstrap --path=clusters/prod`.
|
|
#
|
|
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
|
|
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
|
|
# (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
|
|
# velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
|
|
# ship the literal string into the cluster.
|
|
|
|
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
|
|
#
|
|
# *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
|
|
# 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
|
|
# control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
|
|
# live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
|
|
# own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
|
|
# Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
|
|
# outage you would notice. A virtual address that follows the control plane is the fix, and it is
|
|
# what kube-vip was stood up to provide.
|
|
#
|
|
# dev -> 192.168.1.49 (= kube_vip_address, group_vars/all.yml; base default, below)
|
|
# prod -> 192.168.2.9 (= kube_vip_address, group_vars/k3s_prod/vars.yml)
|
|
#
|
|
# THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
|
|
# unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
|
|
# move. Read it as "the API host Cilium dials", which is what it always meant.
|
|
#
|
|
# [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
|
|
# (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
|
|
# already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
|
|
# the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
|
|
# already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
|
|
# block at the top of group_vars/all.yml.
|
|
#
|
|
# Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
|
|
# in both envs, so it is NOT a knob.
|
|
#
|
|
# TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
|
|
# both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
|
|
# https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
|
|
K3S_API_HOST: "192.168.1.49"
|
|
|
|
# CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
|
|
# L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
|
|
# tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in
|
|
# practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's
|
|
# first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and
|
|
# the flagged block in clusters/prod/kustomization.yaml.)
|
|
CILIUM_LB_POOL_START: "192.168.1.53"
|
|
CILIUM_LB_POOL_STOP: "192.168.1.60"
|
|
|
|
# Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the
|
|
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
|
|
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
|
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
|
|
|
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05). Consumed by BOTH backup paths: every CNPG
|
|
# Cluster's barmanObjectStore.endpointURL (keycloak-db, nextcloud-db, semaphore-db) and Velero's
|
|
# backupStorageLocation s3Url. Full URL including scheme (and port, if not the scheme default).
|
|
#
|
|
# OPERATOR DECISION 2026-09-05: **Cloudflare R2**, replacing the on-Proxmox MinIO — "for the time
|
|
# being until a better on-prem backup solution is created". Two reasons beyond cost:
|
|
# 1. The k3s prod cluster is on the SOHO network (pfSense's WAN) and CANNOT reach the MinIO inside
|
|
# pfSense's LAN — measured, from a pod: connection timed out. R2 needs no firewall change.
|
|
# 2. That MinIO runs on the SAME Proxmox host as dev's VMs, so dev's "off-cluster" backup shared a
|
|
# physical failure domain with dev. One host loss took the cluster AND its backups. R2 is
|
|
# genuinely off-site.
|
|
# Measured to justify the cost: dev's whole estate is 4.0 GiB Velero (6,264 objects, 7d retention) +
|
|
# 211 MiB CNPG (1,755 objects) — pennies per month, and R2 egress is free, so restore REHEARSALS
|
|
# stay free and R-STATE-03 has no cost disincentive to stay proven.
|
|
#
|
|
# The MinIO copy is NOT deleted — it stays as a historical/local copy (3-2-1). Restoring from it
|
|
# means pointing this knob back at http://192.168.1.9:9000 temporarily.
|
|
S3_BACKUP_ENDPOINT: "https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com"
|
|
|
|
# S3 region. Required by every S3 client, IGNORED by MinIO, and must be exactly `auto` for R2.
|
|
# This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-accident for
|
|
# MinIO and wrong for R2, which is why it is a knob now.
|
|
S3_BACKUP_REGION: "auto"
|
|
|
|
# --- storage (ADR-0003) ---
|
|
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
|
# Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
|
|
# replica is written into. It must be the path the NODE's own data disk is actually mounted at —
|
|
# roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
|
|
# fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
|
|
# Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
|
|
# working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
|
|
# on it — changing dev's value would orphan every replica, so the dev overlay restates it
|
|
# verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
|
|
# dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
|
|
# neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
|
|
LONGHORN_DATA_PATH: "/var/lib/longhorn"
|
|
|
|
# --- observability sizing (docs/10) ---
|
|
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
|
|
MONITORING_PROMETHEUS_PVC_SIZE: "10Gi"
|
|
# SIZE-based retention cap for Prometheus (Prometheus *iB units, NOT the K8s `Gi` of the PVC knob).
|
|
# MUST stay below MONITORING_PROMETHEUS_PVC_SIZE — leaves headroom for the WAL + head + compaction
|
|
# scratch (retentionSize counts only persisted blocks). Prevents a busy TSDB from filling /prometheus
|
|
# (a full disk wedges all WAL commits and makes every absent()/==0 alert fire falsely). Base 10Gi PVC.
|
|
MONITORING_PROMETHEUS_RETENTION_SIZE: "8GiB"
|
|
MONITORING_LOKI_PVC_SIZE: "10Gi"
|
|
|
|
# --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture;
|
|
# the dev overlay softens them for bootstrap (Audit / Ignore / 1) so a cold cluster can
|
|
# come up before policies are proven and admission is HA. ---
|
|
KYVERNO_VALIDATION_ACTION: "Enforce" # dev: Audit (observe before enforcing)
|
|
KYVERNO_FORCE_FAILURE_POLICY_IGNORE: "false" # dev: "true" (chart forceFailurePolicyIgnore = fail-open)
|
|
KYVERNO_ADMISSION_REPLICAS: "3" # dev: 1 (no HA needed to iterate)
|
|
|
|
# --- egress / VPN (Phase 3, ADR-0015) — node-level host OpenVPN tunnel steered by Cilium Egress
|
|
# Gateway (platform/vpn-egress/egress-steering.yaml). No in-cluster gateway pod. ---
|
|
PIA_REGION: "SE Stockholm" # PF-capable PIA region (Sweden). Under ADR-0015 the tunnel is
|
|
# host-side (Ansible vault / the node's .ovpn per docs/07), so this
|
|
# is now a reference default only — retained, not substituted into
|
|
# any manifest. ("SE Stockholm" is PF-capable; NOT "SE Streaming
|
|
# Optimized" / a US region, which lack PF.)
|
|
CLUSTER_POD_CIDR: "10.0.0.0/16" # Documents the Cilium cluster-pool pod CIDR that the egress-steering
|
|
# CiliumEgressGatewayPolicy carves out of the tunnel via excludedCIDRs
|
|
# (kept on the LOCAL datapath, never SNAT'd out tun0). The policy
|
|
# currently excludes the whole /8 RFC1918 range; this /16 records the
|
|
# actual pod allocations (Cilium assigns /24 per node from 10.0.0.0 up).
|
|
|
|
# --- identity / auth (ADR-0014) ---
|
|
KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
OAUTH2_PROXY_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
|
|
# --- backup / DR (ADR-0003, R-STATE-03) ---
|
|
VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ
|
|
VELERO_RETENTION: "168h" # dev short | prod long
|
|
|
|
# BACKUP TENANCY — the two knobs that keep two clusters sharing ONE MinIO from destroying each
|
|
# other's backups. Added 2026-09-05 when the operator ruled prod uses the SAME endpoint as dev.
|
|
#
|
|
# Sharing an endpoint is fine. Sharing a PATH is not, and nothing here was per-environment before:
|
|
#
|
|
# VELERO — bucket `velero`, no prefix, and BOTH clusters run schedules named `daily` and
|
|
# `daily-resources-only`. Each cluster's Velero SYNCS backups it finds in the BSL into
|
|
# itself as Backup CRs and GARBAGE-COLLECTS them when their TTL expires. Prod would
|
|
# therefore adopt dev's backups and delete them on schedule.
|
|
# CNPG — destinationPath `s3://cnpg` with CNPG appending serverName, which defaults to the
|
|
# CLUSTER NAME. Both environments have `keycloak-db`, `nextcloud-db`, `semaphore-db`,
|
|
# so prod's Postgres would archive WAL into the exact prefix dev's does. Two servers
|
|
# interleaving WAL into one Barman stream CORRUPTS THE CHAIN and destroys PITR — and
|
|
# dev's DR restore is PROVEN (R-STATE-03). This is the worst of the two.
|
|
#
|
|
# Both environments are now NAMESPACED SYMMETRICALLY, which was not possible an hour ago and is
|
|
# worth explaining, because the earlier reasoning was the opposite:
|
|
#
|
|
# While the target was the MinIO, dev's base value HAD to be the bucket root — its existing
|
|
# backups lived there and giving dev a prefix would have ORPHANED the very backups its proven
|
|
# restore depends on (the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default`).
|
|
#
|
|
# Moving to R2 starts BOTH clusters on an EMPTY bucket, so there is no history to orphan and no
|
|
# reason to keep the asymmetry. `dev` and `prod` is clearer, greppable, and leaves the bucket root
|
|
# free — so an unset/misrendered prefix lands somewhere obviously wrong instead of silently
|
|
# colliding with a real environment's backups.
|
|
#
|
|
# *** THESE TWO VALUES AND S3_BACKUP_ENDPOINT MUST CHANGE TOGETHER. *** Setting a `dev` prefix while
|
|
# the endpoint is still the MinIO would orphan dev's existing MinIO backups for no benefit. They are
|
|
# one change; do not split the commit.
|
|
VELERO_BACKUP_PREFIX: "dev" # dev: "dev" | prod: "prod"
|
|
CNPG_BACKUP_PATH: "s3://cnpg/dev" # dev: s3://cnpg/dev | prod: s3://cnpg/prod
|
|
|
|
# --- placement ---
|
|
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
|
|
PLATFORM_NODE_SELECTOR_VALUE: "linux"
|