jakman85andClaude Opus 5
b80e8eef82
Operator ruling: "the S3 backup location is the same as Dev." Prod's
S3_BACKUP_ENDPOINT is now http://192.168.1.9:9000 and the TBD-PROD-* placeholder
is retired. Set EXPLICITLY rather than by omitting the op — an omitted op would
inherit the same value silently, and a shared backup target must be visible in
the overlay that owns it.
SHARING THE ENDPOINT WAS NOT SAFE AS AUTHORED. Nothing about the backup PATHS
was per-environment, and enabling prod against dev's MinIO would have been the
2026-09-04 external-dns incident again, in a worse place:
VELERO — bucket `velero`, NO prefix, and both clusters run schedules named
`daily` / `daily-resources-only`. Velero SYNCS backups it finds in its BSL
into itself as Backup CRs and GARBAGE-COLLECTS them at TTL, so prod would
have adopted dev's backups and then deleted them on schedule.
CNPG — the worse one. destinationPath `s3://cnpg` with CNPG appending
serverName, which defaults to the CLUSTER NAME. Both environments have
keycloak-db, nextcloud-db and semaphore-db, so prod's Postgres would archive
WAL into the EXACT prefix dev's does. Two servers interleaving WAL into one
Barman stream CORRUPTS THE CHAIN and destroys PITR — and dev's restore is
PROVEN (R-STATE-03). Sharing an endpoint is fine; sharing a path is not.
Two knobs, authored once, environments differing only by scoped overrides
(R-ENV-02):
VELERO_BACKUP_PREFIX dev "" | prod "prod"
CNPG_BACKUP_PATH dev "s3://cnpg" | prod "s3://cnpg/prod"
THE BASE VALUES ARE DEV'S CURRENT LAYOUT ON PURPOSE. Dev's existing backups live
at the bucket root and at s3://cnpg; giving dev a prefix would ORPHAN the very
backups its proven restore depends on — the same reasoning that keeps dev's
EXTERNALDNS_TXT_OWNER_ID at `default`. Prod has no history to preserve.
*** PROD STILL HAS NO BACKUP COVERAGE — the endpoint is not reachable. ***
Measured from a throwaway pod INSIDE the prod cluster (the workstation proves
nothing about pod egress, and a workstation-side reading has already misled this
project once): curl http://192.168.1.9:9000/minio/health/live → connection timed
out after 15s. pfSense does not route prod's pod network to the dev LAN's MinIO.
OPERATOR ACTION REQUIRED. velero stays suspended; note apps-wave-0 applies GREEN
regardless — the token IS defined so strict substitution succeeds — while every
write fails at RUNTIME. A green apps-wave-0 does not mean backups work.
New `standup.sh s3check` step probes the endpoint from a throwaway in-cluster
pod and fails closed, with `--s3-endpoint <url>` to test a CANDIDATE before
authoring it into cluster-vars. That is how the unreachability above was
established rather than assumed.
VERIFIED: full CI gate PASS. flux-postbuild-substitution 254 -> 260 assertions,
and neither new knob appears in the dead-knob advisory list, so both are really
consumed. Rendered both overlays: all three CNPG destinationPaths and Velero's
prefix carry the tokens; dev resolves to its existing layout, prod to the nested
paths.
DELIVERY: prod reconciles the bootstrap Gitea's main at c8c1d29. This branch also
carries the DNS inversion, which must NOT reach prod. Delivering needs a
c8c1d29-based branch with only these commits — no urgency while the routing is
still closed.
Refs R-STATE-03, R-ENV-02, R-IDP-05.
Co-Authored-By: Claude Opus 5 <[email protected]>
240 lines
17 KiB
YAML
240 lines
17 KiB
YAML
---
|
|
# cluster-vars — the overlay knob surface (R-ENV-02).
|
|
#
|
|
# This ConfigMap holds BASE DEFAULTS. Flux `postBuild.substituteFrom` injects these into every
|
|
# `${TOKEN}` in platform/ and apps/ manifests at reconcile time. dev/ and prod/ overlays STRATEGIC-
|
|
# MERGE PATCH this single ConfigMap — that patch is the *entire* difference between environments.
|
|
#
|
|
# Add a knob here (with a base default) before referencing `${KNOB}` in a component manifest, then
|
|
# give each overlay its env-scoped value. Never fork a base manifest to differ per environment.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: cluster-vars
|
|
namespace: flux-system
|
|
data:
|
|
# --- identity of the environment ---
|
|
CLUSTER_ENV: "base" # dev | prod (overlays set this)
|
|
|
|
# --- DNS / TLS ---
|
|
# THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
|
|
# the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
|
|
# ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
|
|
#
|
|
# THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
|
|
# hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
|
|
# authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
|
|
# (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
|
|
# See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
|
|
# suffix, so the two clusters can no longer describe the same name.
|
|
#
|
|
# dev -> dev.home.keithjackson.dev (a subdomain; = this base default)
|
|
# prod -> home.keithjackson.dev (THE APEX — prod IS the apex, no `prod.` label)
|
|
#
|
|
# OPERATOR DECISION 2026-09-04 (supersedes the interim split committed earlier the same day, which
|
|
# had prod on `prod.home.` with apex CNAMEs pointing at it). PROD IS THE APEX. That deletes the
|
|
# whole CNAME layer — `<svc>.home.keithjackson.dev` is prod's real name, not an alias — and it is
|
|
# DEV that carries a label. Convention is unchanged from GATEWAY_VIP / LONGHORN_DATA_PATH: the base
|
|
# default is the DEV value (dev restates it verbatim) and prod overrides it. What changed is which
|
|
# string dev holds.
|
|
#
|
|
# DEV'S VALUE IS A MIGRATION, NOT A VARIABLE CHANGE. The live dev cluster is serving the APEX right
|
|
# now: its wildcard cert, every public DNS record, its Keycloak issuer, ~12 OIDC clients' redirect
|
|
# URIs, kubectl SSO, and gitea.home.keithjackson.dev — which IS this repo's GitOps source AND the
|
|
# push target. Flipping this value on dev without first standing up the new host set STRANDS DEV:
|
|
# Flux's GitRepository url moves to gitea.dev.home.keithjackson.dev, which has no DNS record and no
|
|
# cert, so dev stops reconciling and cannot be fixed by pushing. ORDERED CUTOVER, operator present:
|
|
# infra/PROD-PROVISIONING.md §10.
|
|
#
|
|
# DEV MUST VACATE THE APEX BEFORE PROD PUBLISHES THERE. Both clusters describing
|
|
# `*.home.keithjackson.dev` at once is the 2026-09-04 incident precondition, re-created.
|
|
BASE_DOMAIN: "dev.home.keithjackson.dev"
|
|
|
|
# ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
|
|
# EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
|
|
# so it cannot be created, updated or deleted regardless of TXT ownership.
|
|
#
|
|
# BOTH ENVIRONMENTS USE THE CLOUDFLARE **ZONE**, `keithjackson.dev`. This value is NOT a per-
|
|
# environment isolation knob and must not be narrowed to either environment's BASE_DOMAIN.
|
|
#
|
|
# FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) — the Cloudflare provider uses --domain-filter for ZONE
|
|
# SELECTION as well as name filtering, so a filter that is a STRICT SUBDOMAIN of the hosted zone
|
|
# matches NO zone: ExternalDNS generates the record and then skips it with "no hosted zone matching
|
|
# record DNS Name", publishing NOTHING. This has been tried and reverted more than once (an agent
|
|
# narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in review). DO NOT "harden"
|
|
# it — the change is not defence in depth, it is a total outage of DNS publication.
|
|
#
|
|
# WHERE ISOLATION ACTUALLY COMES FROM, and it is sufficient:
|
|
# (1) the two environments request DISJOINT hostname sets (dev.home.… vs the apex), and
|
|
# (2) EXTERNALDNS_TXT_OWNER_ID differs, so neither registry ever claims the other's records.
|
|
# That pairing is what the 2026-09-04 incident needed; a domain filter cannot provide it, because
|
|
# both environments live in the one zone either way.
|
|
EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
|
|
|
|
# ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
|
|
# neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
|
|
# record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
|
|
# `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
|
|
# one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
|
|
#
|
|
# dev -> `dev` prod -> `prod` base default -> a non-colliding sentinel (below)
|
|
#
|
|
# DEV IS `dev`, NOT `default` (changed 2026-09-04 with the move to dev.home.keithjackson.dev). Until
|
|
# this migration dev HAD to render `default`, because its LIVE apex records were already registered
|
|
# in Cloudflare's TXT registry under that owner and renaming would have orphaned every one of them.
|
|
# THAT CONSTRAINT NO LONGER BINDS: dev is publishing an ENTIRELY NEW record set under
|
|
# *.dev.home.keithjackson.dev, with no pre-existing registry entries to recognise, so the owner id
|
|
# is now free to be chosen on its merits. Two reasons to take `dev`:
|
|
# - `default` is THE VALUE YOU GET BY FORGETTING (it is the external-dns chart default, and both
|
|
# clusters silently shared it, which IS the incident). An environment whose identity is the
|
|
# forget-value can never be told apart from a misconfigured newcomer. No environment should
|
|
# hold it.
|
|
# - It makes vacating the apex an EXPLICIT, ORDERED, VERIFIABLE step. Dev's old apex records stay
|
|
# owned by `default`, which no environment now claims, so nothing deletes them implicitly on
|
|
# some future reconcile — they are retired deliberately, and prod's publish is gated on that
|
|
# retirement being confirmed in Cloudflare (infra/PROD-PROVISIONING.md §10) rather than on a
|
|
# race between two reconcile loops.
|
|
#
|
|
# PROD STAYS `prod` AND MUST NOT BECOME `default`, even though prod now serves the apex names that
|
|
# dev's `default`-owned records currently occupy. Giving prod `default` would hand prod ownership of
|
|
# the LIVE dev cluster's records while dev is still serving them, and under `policy: sync` prod
|
|
# would delete the ones no prod source claims. That is the 2026-09-04 incident, re-armed, with the
|
|
# roles swapped. Prod inherits the apex NAMES; it must never inherit the apex OWNER ID.
|
|
#
|
|
# THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default` and not any real
|
|
# environment's id. A new environment that forgets this knob must not silently inherit another's
|
|
# identity — that is precisely the incident. With this sentinel such a cluster owns nothing, so it
|
|
# can DELETE nothing; it would publish under a visibly wrong owner id (grep the TXT records)
|
|
# instead of destroying a live environment's DNS.
|
|
EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
|
|
|
|
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
|
|
GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
|
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
|
|
# ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
|
|
# takes the pool's first free IP, and coredns-custom answers
|
|
# *.${BASE_DOMAIN} with THIS so pod -> public-hostname traffic
|
|
# hairpins through the Gateway (correct SNI -> wildcard cert).
|
|
# Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
|
|
# long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
|
|
# forgets this knob at least lands on a REAL network). A prod overlay
|
|
# MUST still set the prod LAN VIP alongside the prod L2 pool range.
|
|
|
|
# --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply -------------
|
|
# These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint:
|
|
# `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24)
|
|
# — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values
|
|
# (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the
|
|
# dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block
|
|
# before `flux bootstrap --path=clusters/prod`.
|
|
#
|
|
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
|
|
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
|
|
# (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
|
|
# velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
|
|
# ship the literal string into the cluster.
|
|
|
|
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
|
|
# [parity] MUST equal the Ansible bootstrap's k3s_api_host (infra/ansible/roles/cilium/defaults/
|
|
# main.yml -> cilium-values.yaml.j2, already inventory-parameterized). Wrong value = Cilium cannot
|
|
# reach the API server on adopt and the CNI breaks. Port stays 6443 in both envs, so it is NOT a knob.
|
|
K3S_API_HOST: "192.168.1.50"
|
|
|
|
# CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
|
|
# L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
|
|
# tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in
|
|
# practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's
|
|
# first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and
|
|
# the flagged block in clusters/prod/kustomization.yaml.)
|
|
CILIUM_LB_POOL_START: "192.168.1.53"
|
|
CILIUM_LB_POOL_STOP: "192.168.1.60"
|
|
|
|
# Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the
|
|
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
|
|
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
|
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
|
|
|
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
|
|
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
|
|
# nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
|
|
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR.
|
|
S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
|
|
|
|
# --- storage (ADR-0003) ---
|
|
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
|
# Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
|
|
# replica is written into. It must be the path the NODE's own data disk is actually mounted at —
|
|
# roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
|
|
# fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
|
|
# Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
|
|
# working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
|
|
# on it — changing dev's value would orphan every replica, so the dev overlay restates it
|
|
# verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
|
|
# dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
|
|
# neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
|
|
LONGHORN_DATA_PATH: "/var/lib/longhorn"
|
|
|
|
# --- observability sizing (docs/10) ---
|
|
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
|
|
MONITORING_PROMETHEUS_PVC_SIZE: "10Gi"
|
|
# SIZE-based retention cap for Prometheus (Prometheus *iB units, NOT the K8s `Gi` of the PVC knob).
|
|
# MUST stay below MONITORING_PROMETHEUS_PVC_SIZE — leaves headroom for the WAL + head + compaction
|
|
# scratch (retentionSize counts only persisted blocks). Prevents a busy TSDB from filling /prometheus
|
|
# (a full disk wedges all WAL commits and makes every absent()/==0 alert fire falsely). Base 10Gi PVC.
|
|
MONITORING_PROMETHEUS_RETENTION_SIZE: "8GiB"
|
|
MONITORING_LOKI_PVC_SIZE: "10Gi"
|
|
|
|
# --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture;
|
|
# the dev overlay softens them for bootstrap (Audit / Ignore / 1) so a cold cluster can
|
|
# come up before policies are proven and admission is HA. ---
|
|
KYVERNO_VALIDATION_ACTION: "Enforce" # dev: Audit (observe before enforcing)
|
|
KYVERNO_FORCE_FAILURE_POLICY_IGNORE: "false" # dev: "true" (chart forceFailurePolicyIgnore = fail-open)
|
|
KYVERNO_ADMISSION_REPLICAS: "3" # dev: 1 (no HA needed to iterate)
|
|
|
|
# --- egress / VPN (Phase 3, ADR-0015) — node-level host OpenVPN tunnel steered by Cilium Egress
|
|
# Gateway (platform/vpn-egress/egress-steering.yaml). No in-cluster gateway pod. ---
|
|
PIA_REGION: "SE Stockholm" # PF-capable PIA region (Sweden). Under ADR-0015 the tunnel is
|
|
# host-side (Ansible vault / the node's .ovpn per docs/07), so this
|
|
# is now a reference default only — retained, not substituted into
|
|
# any manifest. ("SE Stockholm" is PF-capable; NOT "SE Streaming
|
|
# Optimized" / a US region, which lack PF.)
|
|
CLUSTER_POD_CIDR: "10.0.0.0/16" # Documents the Cilium cluster-pool pod CIDR that the egress-steering
|
|
# CiliumEgressGatewayPolicy carves out of the tunnel via excludedCIDRs
|
|
# (kept on the LOCAL datapath, never SNAT'd out tun0). The policy
|
|
# currently excludes the whole /8 RFC1918 range; this /16 records the
|
|
# actual pod allocations (Cilium assigns /24 per node from 10.0.0.0 up).
|
|
|
|
# --- identity / auth (ADR-0014) ---
|
|
KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
OAUTH2_PROXY_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
|
|
# --- backup / DR (ADR-0003, R-STATE-03) ---
|
|
VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ
|
|
VELERO_RETENTION: "168h" # dev short | prod long
|
|
|
|
# BACKUP TENANCY — the two knobs that keep two clusters sharing ONE MinIO from destroying each
|
|
# other's backups. Added 2026-09-05 when the operator ruled prod uses the SAME endpoint as dev.
|
|
#
|
|
# Sharing an endpoint is fine. Sharing a PATH is not, and nothing here was per-environment before:
|
|
#
|
|
# VELERO — bucket `velero`, no prefix, and BOTH clusters run schedules named `daily` and
|
|
# `daily-resources-only`. Each cluster's Velero SYNCS backups it finds in the BSL into
|
|
# itself as Backup CRs and GARBAGE-COLLECTS them when their TTL expires. Prod would
|
|
# therefore adopt dev's backups and delete them on schedule.
|
|
# CNPG — destinationPath `s3://cnpg` with CNPG appending serverName, which defaults to the
|
|
# CLUSTER NAME. Both environments have `keycloak-db`, `nextcloud-db`, `semaphore-db`,
|
|
# so prod's Postgres would archive WAL into the exact prefix dev's does. Two servers
|
|
# interleaving WAL into one Barman stream CORRUPTS THE CHAIN and destroys PITR — and
|
|
# dev's DR restore is PROVEN (R-STATE-03). This is the worst of the two.
|
|
#
|
|
# THE BASE VALUES ARE DEV'S CURRENT LAYOUT ON PURPOSE, and must stay that way: dev's existing
|
|
# backups live at the bucket root / at `s3://cnpg`. Giving dev a prefix would ORPHAN the very
|
|
# backups its proven restore depends on — the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays
|
|
# `default`. Prod, which has no history to preserve, takes the namespaced paths.
|
|
VELERO_BACKUP_PREFIX: "" # dev: bucket root (existing layout) | prod: "prod"
|
|
CNPG_BACKUP_PATH: "s3://cnpg" # dev: bucket root (existing layout) | prod: "s3://cnpg/prod"
|
|
|
|
# --- placement ---
|
|
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
|
|
PLATFORM_NODE_SELECTOR_VALUE: "linux"
|