--- # cluster-vars — the overlay knob surface (R-ENV-02). # # This ConfigMap holds BASE DEFAULTS. Flux `postBuild.substituteFrom` injects these into every # `${TOKEN}` in platform/ and apps/ manifests at reconcile time. dev/ and prod/ overlays STRATEGIC- # MERGE PATCH this single ConfigMap — that patch is the *entire* difference between environments. # # Add a knob here (with a base default) before referencing `${KNOB}` in a component manifest, then # give each overlay its env-scoped value. Never fork a base manifest to differ per environment. apiVersion: v1 kind: ConfigMap metadata: name: cluster-vars namespace: flux-system data: # --- identity of the environment --- CLUSTER_ENV: "base" # dev | prod (overlays set this) # --- DNS / TLS --- # THE platform hostname suffix. Every public hostname the cluster serves is `.${BASE_DOMAIN}`, # the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with # ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it. # # THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every # hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM. # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN # suffix, so the two clusters can no longer describe the same name. # # dev -> dev.home.keithjackson.dev (a subdomain; = this base default) # prod -> home.keithjackson.dev (THE APEX — prod IS the apex, no `prod.` label) # # OPERATOR DECISION 2026-09-04 (supersedes the interim split committed earlier the same day, which # had prod on `prod.home.` with apex CNAMEs pointing at it). PROD IS THE APEX. That deletes the # whole CNAME layer — `.home.keithjackson.dev` is prod's real name, not an alias — and it is # DEV that carries a label. Convention is unchanged from GATEWAY_VIP / LONGHORN_DATA_PATH: the base # default is the DEV value (dev restates it verbatim) and prod overrides it. What changed is which # string dev holds. # # DEV'S VALUE IS A MIGRATION, NOT A VARIABLE CHANGE. The live dev cluster is serving the APEX right # now: its wildcard cert, every public DNS record, its Keycloak issuer, ~12 OIDC clients' redirect # URIs, kubectl SSO, and gitea.home.keithjackson.dev — which IS this repo's GitOps source AND the # push target. Flipping this value on dev without first standing up the new host set STRANDS DEV: # Flux's GitRepository url moves to gitea.dev.home.keithjackson.dev, which has no DNS record and no # cert, so dev stops reconciling and cannot be fixed by pushing. ORDERED CUTOVER, operator present: # infra/PROD-PROVISIONING.md §10. # # DEV MUST VACATE THE APEX BEFORE PROD PUBLISHES THERE. Both clusters describing # `*.home.keithjackson.dev` at once is the 2026-09-04 incident precondition, re-created. BASE_DOMAIN: "dev.home.keithjackson.dev" # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered, # so it cannot be created, updated or deleted regardless of TXT ownership. # # BOTH ENVIRONMENTS USE THE CLOUDFLARE **ZONE**, `keithjackson.dev`. This value is NOT a per- # environment isolation knob and must not be narrowed to either environment's BASE_DOMAIN. # # FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) — the Cloudflare provider uses --domain-filter for ZONE # SELECTION as well as name filtering, so a filter that is a STRICT SUBDOMAIN of the hosted zone # matches NO zone: ExternalDNS generates the record and then skips it with "no hosted zone matching # record DNS Name", publishing NOTHING. This has been tried and reverted more than once (an agent # narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in review). DO NOT "harden" # it — the change is not defence in depth, it is a total outage of DNS publication. # # WHERE ISOLATION ACTUALLY COMES FROM, and it is sufficient: # (1) the two environments request DISJOINT hostname sets (dev.home.… vs the apex), and # (2) EXTERNALDNS_TXT_OWNER_ID differs, so neither registry ever claims the other's records. # That pairing is what the 2026-09-04 incident needed; a domain filter cannot provide it, because # both environments live in the one zone either way. EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev" # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT: # neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a # record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on # one zone therefore garbage-collect each other. Each environment MUST have a distinct value. # # dev -> `dev` prod -> `prod` base default -> a non-colliding sentinel (below) # # DEV IS `dev`, NOT `default` (changed 2026-09-04 with the move to dev.home.keithjackson.dev). Until # this migration dev HAD to render `default`, because its LIVE apex records were already registered # in Cloudflare's TXT registry under that owner and renaming would have orphaned every one of them. # THAT CONSTRAINT NO LONGER BINDS: dev is publishing an ENTIRELY NEW record set under # *.dev.home.keithjackson.dev, with no pre-existing registry entries to recognise, so the owner id # is now free to be chosen on its merits. Two reasons to take `dev`: # - `default` is THE VALUE YOU GET BY FORGETTING (it is the external-dns chart default, and both # clusters silently shared it, which IS the incident). An environment whose identity is the # forget-value can never be told apart from a misconfigured newcomer. No environment should # hold it. # - It makes vacating the apex an EXPLICIT, ORDERED, VERIFIABLE step. Dev's old apex records stay # owned by `default`, which no environment now claims, so nothing deletes them implicitly on # some future reconcile — they are retired deliberately, and prod's publish is gated on that # retirement being confirmed in Cloudflare (infra/PROD-PROVISIONING.md §10) rather than on a # race between two reconcile loops. # # PROD STAYS `prod` AND MUST NOT BECOME `default`, even though prod now serves the apex names that # dev's `default`-owned records currently occupy. Giving prod `default` would hand prod ownership of # the LIVE dev cluster's records while dev is still serving them, and under `policy: sync` prod # would delete the ones no prod source claims. That is the 2026-09-04 incident, re-armed, with the # roles swapped. Prod inherits the apex NAMES; it must never inherit the apex OWNER ID. # # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default` and not any real # environment's id. A new environment that forgets this knob must not silently inherit another's # identity — that is precisely the incident. With this sentinel such a cluster owns nothing, so it # can DELETE nothing; it would publish under a visibly wrong owner id (grep the TXT records) # instead of destroying a live environment's DNS. EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars" ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006) GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of # ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway # takes the pool's first free IP, and coredns-custom answers # *.${BASE_DOMAIN} with THIS so pod -> public-hostname traffic # hairpins through the Gateway (correct SNI -> wildcard cert). # Base default = the DEV LAN VIP (was 10.73.0.240, an address on the # long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that # forgets this knob at least lands on a REAL network). A prod overlay # MUST still set the prod LAN VIP alongside the prod L2 pool range. # --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply ------------- # These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint: # `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24) # — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values # (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the # dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block # before `flux bootstrap --path=clusters/prod`. # # Substitution coverage (verify before adding a knob here): every path that consumes these declares # postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config # (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT), # velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would # ship the literal string into the cluster. # k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml). # [parity] MUST equal the Ansible bootstrap's k3s_api_host (infra/ansible/roles/cilium/defaults/ # main.yml -> cilium-values.yaml.j2, already inventory-parameterized). Wrong value = Cilium cannot # reach the API server on adopt and the CNI breaks. Port stays 6443 in both envs, so it is NOT a knob. K3S_API_HOST: "192.168.1.50" # CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range # L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the # tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in # practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's # first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and # the flagged block in clusters/prod/kustomization.yaml.) CILIUM_LB_POOL_START: "192.168.1.53" CILIUM_LB_POOL_STOP: "192.168.1.60" # Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the # HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address: UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi) CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style. # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db, # nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port. # Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR. S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000" # --- storage (ADR-0003) --- LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes) # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every # replica is written into. It must be the path the NODE's own data disk is actually mounted at — # roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE # fact stated in two places, so [parity] holds per environment or replicas land on the OS disk. # Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good # working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes # on it — changing dev's value would orphan every replica, so the dev overlay restates it # verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data # dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the # neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root. LONGHORN_DATA_PATH: "/var/lib/longhorn" # --- observability sizing (docs/10) --- MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long MONITORING_PROMETHEUS_PVC_SIZE: "10Gi" # SIZE-based retention cap for Prometheus (Prometheus *iB units, NOT the K8s `Gi` of the PVC knob). # MUST stay below MONITORING_PROMETHEUS_PVC_SIZE — leaves headroom for the WAL + head + compaction # scratch (retentionSize counts only persisted blocks). Prevents a busy TSDB from filling /prometheus # (a full disk wedges all WAL commits and makes every absent()/==0 alert fire falsely). Base 10Gi PVC. MONITORING_PROMETHEUS_RETENTION_SIZE: "8GiB" MONITORING_LOKI_PVC_SIZE: "10Gi" # --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture; # the dev overlay softens them for bootstrap (Audit / Ignore / 1) so a cold cluster can # come up before policies are proven and admission is HA. --- KYVERNO_VALIDATION_ACTION: "Enforce" # dev: Audit (observe before enforcing) KYVERNO_FORCE_FAILURE_POLICY_IGNORE: "false" # dev: "true" (chart forceFailurePolicyIgnore = fail-open) KYVERNO_ADMISSION_REPLICAS: "3" # dev: 1 (no HA needed to iterate) # --- egress / VPN (Phase 3, ADR-0015) — node-level host OpenVPN tunnel steered by Cilium Egress # Gateway (platform/vpn-egress/egress-steering.yaml). No in-cluster gateway pod. --- PIA_REGION: "SE Stockholm" # PF-capable PIA region (Sweden). Under ADR-0015 the tunnel is # host-side (Ansible vault / the node's .ovpn per docs/07), so this # is now a reference default only — retained, not substituted into # any manifest. ("SE Stockholm" is PF-capable; NOT "SE Streaming # Optimized" / a US region, which lack PF.) CLUSTER_POD_CIDR: "10.0.0.0/16" # Documents the Cilium cluster-pool pod CIDR that the egress-steering # CiliumEgressGatewayPolicy carves out of the tunnel via excludedCIDRs # (kept on the LOCAL datapath, never SNAT'd out tun0). The policy # currently excludes the whole /8 RFC1918 range; this /16 records the # actual pod allocations (Cilium assigns /24 per node from 10.0.0.0 up). # --- identity / auth (ADR-0014) --- KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA) OAUTH2_PROXY_REPLICAS: "1" # dev 1 | prod 2 (HA) # --- backup / DR (ADR-0003, R-STATE-03) --- VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ VELERO_RETENTION: "168h" # dev short | prod long # --- placement --- PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles PLATFORM_NODE_SELECTOR_VALUE: "linux"