--- # cluster-vars — the overlay knob surface (R-ENV-02). # # This ConfigMap holds BASE DEFAULTS. Flux `postBuild.substituteFrom` injects these into every # `${TOKEN}` in platform/ and apps/ manifests at reconcile time. dev/ and prod/ overlays STRATEGIC- # MERGE PATCH this single ConfigMap — that patch is the *entire* difference between environments. # # Add a knob here (with a base default) before referencing `${KNOB}` in a component manifest, then # give each overlay its env-scoped value. Never fork a base manifest to differ per environment. apiVersion: v1 kind: ConfigMap metadata: name: cluster-vars namespace: flux-system data: # --- identity of the environment --- CLUSTER_ENV: "base" # dev | prod (overlays set this) # --- DNS / TLS --- # THE platform hostname suffix. Every public hostname the cluster serves is `.${BASE_DOMAIN}`, # the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with # ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it. # # THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every # hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM. # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN # subdomain, so the two clusters can no longer describe the same name. # # dev -> home.keithjackson.dev (UNCHANGED — see below) # prod -> prod.home.keithjackson.dev # # DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on # `dev.home.keithjackson.dev` with apex CNAMEs `.home` -> `.prod.home`, but moving dev # relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and # the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2 # (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as # GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates # it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre- # tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths). BASE_DOMAIN: "home.keithjackson.dev" # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered, # so it cannot be created, updated or deleted regardless of TXT ownership. # # DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev` # apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter # that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching # record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged # from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically. # # PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE # (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned # chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED # — it can never delete another environment's records — and the fallback is to set prod's value # back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard. EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev" # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT: # neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a # record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on # one zone therefore garbage-collect each other. Each environment MUST have a distinct value. # # DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare # under owner `default`; changing dev's value would make dev's own external-dns stop recognising # them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it. # # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that # forgets this knob must not silently inherit dev's identity — that is precisely the incident. With # this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a # visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS. EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars" ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006) GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of # ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway # takes the pool's first free IP, and coredns-custom answers # *.home.keithjackson.dev with THIS so pod -> public-hostname traffic # hairpins through the Gateway (correct SNI -> wildcard cert). # Base default = the DEV LAN VIP (was 10.73.0.240, an address on the # long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that # forgets this knob at least lands on a REAL network). A prod overlay # MUST still set the prod LAN VIP alongside the prod L2 pool range. # --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply ------------- # These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint: # `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24) # — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values # (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the # dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block # before `flux bootstrap --path=clusters/prod`. # # Substitution coverage (verify before adding a knob here): every path that consumes these declares # postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config # (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT), # velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would # ship the literal string into the cluster. # k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml). # # *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision # 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's # control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on # live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's # own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off # Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an # outage you would notice. A virtual address that follows the control plane is the fix, and it is # what kube-vip was stood up to provide. # # dev -> 192.168.1.49 (= kube_vip_address, group_vars/all.yml; base default, below) # prod -> 192.168.2.9 (= kube_vip_address, group_vars/k3s_prod/vars.yml) # # THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second, # unrelated change to every substituting Kustomization at the same time as a live CNI endpoint # move. Read it as "the API host Cilium dials", which is what it always meant. # # [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment # (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2, # already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds # the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is # already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES # block at the top of group_vars/all.yml. # # Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443 # in both envs, so it is NOT a knob. # # TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so # both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through # https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart. K3S_API_HOST: "192.168.1.49" # CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range # L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the # tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in # practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's # first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and # the flagged block in clusters/prod/kustomization.yaml.) CILIUM_LB_POOL_START: "192.168.1.53" CILIUM_LB_POOL_STOP: "192.168.1.60" # Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the # HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address: UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi) CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style. # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db, # nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port. # Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR. S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000" # --- storage (ADR-0003) --- LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes) # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every # replica is written into. It must be the path the NODE's own data disk is actually mounted at — # roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE # fact stated in two places, so [parity] holds per environment or replicas land on the OS disk. # Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good # working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes # on it — changing dev's value would orphan every replica, so the dev overlay restates it # verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data # dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the # neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root. LONGHORN_DATA_PATH: "/var/lib/longhorn" # --- observability sizing (docs/10) --- MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long MONITORING_PROMETHEUS_PVC_SIZE: "10Gi" # SIZE-based retention cap for Prometheus (Prometheus *iB units, NOT the K8s `Gi` of the PVC knob). # MUST stay below MONITORING_PROMETHEUS_PVC_SIZE — leaves headroom for the WAL + head + compaction # scratch (retentionSize counts only persisted blocks). Prevents a busy TSDB from filling /prometheus # (a full disk wedges all WAL commits and makes every absent()/==0 alert fire falsely). Base 10Gi PVC. MONITORING_PROMETHEUS_RETENTION_SIZE: "8GiB" MONITORING_LOKI_PVC_SIZE: "10Gi" # --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture; # the dev overlay softens them for bootstrap (Audit / Ignore / 1) so a cold cluster can # come up before policies are proven and admission is HA. --- KYVERNO_VALIDATION_ACTION: "Enforce" # dev: Audit (observe before enforcing) KYVERNO_FORCE_FAILURE_POLICY_IGNORE: "false" # dev: "true" (chart forceFailurePolicyIgnore = fail-open) KYVERNO_ADMISSION_REPLICAS: "3" # dev: 1 (no HA needed to iterate) # --- egress / VPN (Phase 3, ADR-0015) — node-level host OpenVPN tunnel steered by Cilium Egress # Gateway (platform/vpn-egress/egress-steering.yaml). No in-cluster gateway pod. --- PIA_REGION: "SE Stockholm" # PF-capable PIA region (Sweden). Under ADR-0015 the tunnel is # host-side (Ansible vault / the node's .ovpn per docs/07), so this # is now a reference default only — retained, not substituted into # any manifest. ("SE Stockholm" is PF-capable; NOT "SE Streaming # Optimized" / a US region, which lack PF.) CLUSTER_POD_CIDR: "10.0.0.0/16" # Documents the Cilium cluster-pool pod CIDR that the egress-steering # CiliumEgressGatewayPolicy carves out of the tunnel via excludedCIDRs # (kept on the LOCAL datapath, never SNAT'd out tun0). The policy # currently excludes the whole /8 RFC1918 range; this /16 records the # actual pod allocations (Cilium assigns /24 per node from 10.0.0.0 up). # --- identity / auth (ADR-0014) --- KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA) OAUTH2_PROXY_REPLICAS: "1" # dev 1 | prod 2 (HA) # --- backup / DR (ADR-0003, R-STATE-03) --- VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ VELERO_RETENTION: "168h" # dev short | prod long # --- placement --- PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles PLATFORM_NODE_SELECTOR_VALUE: "linux"