--- # cluster-vars — the overlay knob surface (R-ENV-02). # # This ConfigMap holds BASE DEFAULTS. Flux `postBuild.substituteFrom` injects these into every # `${TOKEN}` in platform/ and apps/ manifests at reconcile time. dev/ and prod/ overlays STRATEGIC- # MERGE PATCH this single ConfigMap — that patch is the *entire* difference between environments. # # Add a knob here (with a base default) before referencing `${KNOB}` in a component manifest, then # give each overlay its env-scoped value. Never fork a base manifest to differ per environment. apiVersion: v1 kind: ConfigMap metadata: name: cluster-vars namespace: flux-system data: # --- identity of the environment --- CLUSTER_ENV: "base" # dev | prod (overlays set this) # --- DNS / TLS --- # THE platform hostname suffix. Every public hostname the cluster serves is `.${BASE_DOMAIN}`, # the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with # ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it. # # THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every # hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM. # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN # suffix, so the two clusters can no longer describe the same name. # # dev -> dev.home.keithjackson.dev (a subdomain; = this base default) # prod -> home.keithjackson.dev (THE APEX — prod IS the apex, no `prod.` label) # # OPERATOR DECISION 2026-09-04 (supersedes the interim split committed earlier the same day, which # had prod on `prod.home.` with apex CNAMEs pointing at it). PROD IS THE APEX. That deletes the # whole CNAME layer — `.home.keithjackson.dev` is prod's real name, not an alias — and it is # DEV that carries a label. Convention is unchanged from GATEWAY_VIP / LONGHORN_DATA_PATH: the base # default is the DEV value (dev restates it verbatim) and prod overrides it. What changed is which # string dev holds. # # DEV'S VALUE IS A MIGRATION, NOT A VARIABLE CHANGE. The live dev cluster is serving the APEX right # now: its wildcard cert, every public DNS record, its Keycloak issuer, ~12 OIDC clients' redirect # URIs, kubectl SSO, and gitea.home.keithjackson.dev — which IS this repo's GitOps source AND the # push target. Flipping this value on dev without first standing up the new host set STRANDS DEV: # Flux's GitRepository url moves to gitea.dev.home.keithjackson.dev, which has no DNS record and no # cert, so dev stops reconciling and cannot be fixed by pushing. ORDERED CUTOVER, operator present: # infra/PROD-PROVISIONING.md §10. # # DEV MUST VACATE THE APEX BEFORE PROD PUBLISHES THERE. Both clusters describing # `*.home.keithjackson.dev` at once is the 2026-09-04 incident precondition, re-created. BASE_DOMAIN: "dev.home.keithjackson.dev" # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered, # so it cannot be created, updated or deleted regardless of TXT ownership. # # BOTH ENVIRONMENTS USE THE CLOUDFLARE **ZONE**, `keithjackson.dev`. This value is NOT a per- # environment isolation knob and must not be narrowed to either environment's BASE_DOMAIN. # # FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) — the Cloudflare provider uses --domain-filter for ZONE # SELECTION as well as name filtering, so a filter that is a STRICT SUBDOMAIN of the hosted zone # matches NO zone: ExternalDNS generates the record and then skips it with "no hosted zone matching # record DNS Name", publishing NOTHING. This has been tried and reverted more than once (an agent # narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in review). DO NOT "harden" # it — the change is not defence in depth, it is a total outage of DNS publication. # # WHERE ISOLATION ACTUALLY COMES FROM, and it is sufficient: # (1) the two environments request DISJOINT hostname sets (dev.home.… vs the apex), and # (2) EXTERNALDNS_TXT_OWNER_ID differs, so neither registry ever claims the other's records. # That pairing is what the 2026-09-04 incident needed; a domain filter cannot provide it, because # both environments live in the one zone either way. EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev" # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT: # neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a # record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on # one zone therefore garbage-collect each other. Each environment MUST have a distinct value. # # dev -> `dev` prod -> `prod` base default -> a non-colliding sentinel (below) # # DEV IS `dev`, NOT `default` (changed 2026-09-04 with the move to dev.home.keithjackson.dev). Until # this migration dev HAD to render `default`, because its LIVE apex records were already registered # in Cloudflare's TXT registry under that owner and renaming would have orphaned every one of them. # THAT CONSTRAINT NO LONGER BINDS: dev is publishing an ENTIRELY NEW record set under # *.dev.home.keithjackson.dev, with no pre-existing registry entries to recognise, so the owner id # is now free to be chosen on its merits. Two reasons to take `dev`: # - `default` is THE VALUE YOU GET BY FORGETTING (it is the external-dns chart default, and both # clusters silently shared it, which IS the incident). An environment whose identity is the # forget-value can never be told apart from a misconfigured newcomer. No environment should # hold it. # - It makes vacating the apex an EXPLICIT, ORDERED, VERIFIABLE step. Dev's old apex records stay # owned by `default`, which no environment now claims, so nothing deletes them implicitly on # some future reconcile — they are retired deliberately, and prod's publish is gated on that # retirement being confirmed in Cloudflare (infra/PROD-PROVISIONING.md §10) rather than on a # race between two reconcile loops. # # PROD STAYS `prod` AND MUST NOT BECOME `default`, even though prod now serves the apex names that # dev's `default`-owned records currently occupy. Giving prod `default` would hand prod ownership of # the LIVE dev cluster's records while dev is still serving them, and under `policy: sync` prod # would delete the ones no prod source claims. That is the 2026-09-04 incident, re-armed, with the # roles swapped. Prod inherits the apex NAMES; it must never inherit the apex OWNER ID. # # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default` and not any real # environment's id. A new environment that forgets this knob must not silently inherit another's # identity — that is precisely the incident. With this sentinel such a cluster owns nothing, so it # can DELETE nothing; it would publish under a visibly wrong owner id (grep the TXT records) # instead of destroying a live environment's DNS. EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars" ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006) GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of # ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway # takes the pool's first free IP, and coredns-custom answers # *.${BASE_DOMAIN} with THIS so pod -> public-hostname traffic # hairpins through the Gateway (correct SNI -> wildcard cert). # Base default = the DEV LAN VIP (was 10.73.0.240, an address on the # long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that # forgets this knob at least lands on a REAL network). A prod overlay # MUST still set the prod LAN VIP alongside the prod L2 pool range. # --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply ------------- # These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint: # `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24) # — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values # (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the # dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block # before `flux bootstrap --path=clusters/prod`. # # Substitution coverage (verify before adding a knob here): every path that consumes these declares # postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config # (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT), # velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would # ship the literal string into the cluster. # k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml). # # *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision # 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's # control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on # live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's # own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off # Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an # outage you would notice. A virtual address that follows the control plane is the fix, and it is # what kube-vip was stood up to provide. # # dev -> 192.168.1.49 (= kube_vip_address, group_vars/all.yml; base default, below) # prod -> 192.168.2.9 (= kube_vip_address, group_vars/k3s_prod/vars.yml) # # THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second, # unrelated change to every substituting Kustomization at the same time as a live CNI endpoint # move. Read it as "the API host Cilium dials", which is what it always meant. # # [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment # (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2, # already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds # the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is # already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES # block at the top of group_vars/all.yml. # # Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443 # in both envs, so it is NOT a knob. # # TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so # both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through # https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart. K3S_API_HOST: "192.168.1.49" # CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range # L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the # tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in # practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's # first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and # the flagged block in clusters/prod/kustomization.yaml.) CILIUM_LB_POOL_START: "192.168.1.53" CILIUM_LB_POOL_STOP: "192.168.1.60" # Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the # HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address: UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi) CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports # Off-cluster S3 backup target (R-STATE-03, R-IDP-05). Consumed by BOTH backup paths: every CNPG # Cluster's barmanObjectStore.endpointURL (keycloak-db, nextcloud-db, semaphore-db) and Velero's # backupStorageLocation s3Url. Full URL including scheme (and port, if not the scheme default). # # OPERATOR DECISION 2026-09-05: **Cloudflare R2**, replacing the on-Proxmox MinIO — "for the time # being until a better on-prem backup solution is created". Two reasons beyond cost: # 1. The k3s prod cluster is on the SOHO network (pfSense's WAN) and CANNOT reach the MinIO inside # pfSense's LAN — measured, from a pod: connection timed out. R2 needs no firewall change. # 2. That MinIO runs on the SAME Proxmox host as dev's VMs, so dev's "off-cluster" backup shared a # physical failure domain with dev. One host loss took the cluster AND its backups. R2 is # genuinely off-site. # Measured to justify the cost: dev's whole estate is 4.0 GiB Velero (6,264 objects, 7d retention) + # 211 MiB CNPG (1,755 objects) — pennies per month, and R2 egress is free, so restore REHEARSALS # stay free and R-STATE-03 has no cost disincentive to stay proven. # # The MinIO copy is NOT deleted — it stays as a historical/local copy (3-2-1). Restoring from it # means pointing this knob back at http://192.168.1.9:9000 temporarily. S3_BACKUP_ENDPOINT: "https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com" # S3 region. Required by every S3 client, IGNORED by MinIO, and must be exactly `auto` for R2. # This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-accident for # MinIO and wrong for R2, which is why it is a knob now. S3_BACKUP_REGION: "auto" # --- storage (ADR-0003) --- LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes) # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every # replica is written into. It must be the path the NODE's own data disk is actually mounted at — # roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE # fact stated in two places, so [parity] holds per environment or replicas land on the OS disk. # Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good # working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes # on it — changing dev's value would orphan every replica, so the dev overlay restates it # verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data # dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the # neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root. LONGHORN_DATA_PATH: "/var/lib/longhorn" # --- observability sizing (docs/10) --- MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long MONITORING_PROMETHEUS_PVC_SIZE: "10Gi" # SIZE-based retention cap for Prometheus (Prometheus *iB units, NOT the K8s `Gi` of the PVC knob). # MUST stay below MONITORING_PROMETHEUS_PVC_SIZE — leaves headroom for the WAL + head + compaction # scratch (retentionSize counts only persisted blocks). Prevents a busy TSDB from filling /prometheus # (a full disk wedges all WAL commits and makes every absent()/==0 alert fire falsely). Base 10Gi PVC. MONITORING_PROMETHEUS_RETENTION_SIZE: "8GiB" MONITORING_LOKI_PVC_SIZE: "10Gi" # --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture; # the dev overlay softens them for bootstrap (Audit / Ignore / 1) so a cold cluster can # come up before policies are proven and admission is HA. --- KYVERNO_VALIDATION_ACTION: "Enforce" # dev: Audit (observe before enforcing) KYVERNO_FORCE_FAILURE_POLICY_IGNORE: "false" # dev: "true" (chart forceFailurePolicyIgnore = fail-open) KYVERNO_ADMISSION_REPLICAS: "3" # dev: 1 (no HA needed to iterate) # --- egress / VPN (Phase 3, ADR-0015) — node-level host OpenVPN tunnel steered by Cilium Egress # Gateway (platform/vpn-egress/egress-steering.yaml). No in-cluster gateway pod. --- PIA_REGION: "SE Stockholm" # PF-capable PIA region (Sweden). Under ADR-0015 the tunnel is # host-side (Ansible vault / the node's .ovpn per docs/07), so this # is now a reference default only — retained, not substituted into # any manifest. ("SE Stockholm" is PF-capable; NOT "SE Streaming # Optimized" / a US region, which lack PF.) CLUSTER_POD_CIDR: "10.0.0.0/16" # Documents the Cilium cluster-pool pod CIDR that the egress-steering # CiliumEgressGatewayPolicy carves out of the tunnel via excludedCIDRs # (kept on the LOCAL datapath, never SNAT'd out tun0). The policy # currently excludes the whole /8 RFC1918 range; this /16 records the # actual pod allocations (Cilium assigns /24 per node from 10.0.0.0 up). # --- identity / auth (ADR-0014) --- KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA) OAUTH2_PROXY_REPLICAS: "1" # dev 1 | prod 2 (HA) # --- backup / DR (ADR-0003, R-STATE-03) --- VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ VELERO_RETENTION: "168h" # dev short | prod long # BACKUP TENANCY — the two knobs that keep two clusters sharing ONE MinIO from destroying each # other's backups. Added 2026-09-05 when the operator ruled prod uses the SAME endpoint as dev. # # Sharing an endpoint is fine. Sharing a PATH is not, and nothing here was per-environment before: # # VELERO — bucket `velero`, no prefix, and BOTH clusters run schedules named `daily` and # `daily-resources-only`. Each cluster's Velero SYNCS backups it finds in the BSL into # itself as Backup CRs and GARBAGE-COLLECTS them when their TTL expires. Prod would # therefore adopt dev's backups and delete them on schedule. # CNPG — destinationPath `s3://cnpg` with CNPG appending serverName, which defaults to the # CLUSTER NAME. Both environments have `keycloak-db`, `nextcloud-db`, `semaphore-db`, # so prod's Postgres would archive WAL into the exact prefix dev's does. Two servers # interleaving WAL into one Barman stream CORRUPTS THE CHAIN and destroys PITR — and # dev's DR restore is PROVEN (R-STATE-03). This is the worst of the two. # # Both environments are now NAMESPACED SYMMETRICALLY, which was not possible an hour ago and is # worth explaining, because the earlier reasoning was the opposite: # # While the target was the MinIO, dev's base value HAD to be the bucket root — its existing # backups lived there and giving dev a prefix would have ORPHANED the very backups its proven # restore depends on (the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default`). # # Moving to R2 starts BOTH clusters on an EMPTY bucket, so there is no history to orphan and no # reason to keep the asymmetry. `dev` and `prod` is clearer, greppable, and leaves the bucket root # free — so an unset/misrendered prefix lands somewhere obviously wrong instead of silently # colliding with a real environment's backups. # # *** THESE TWO VALUES AND S3_BACKUP_ENDPOINT MUST CHANGE TOGETHER. *** Setting a `dev` prefix while # the endpoint is still the MinIO would orphan dev's existing MinIO backups for no benefit. They are # one change; do not split the commit. VELERO_BACKUP_PREFIX: "dev" # dev: "dev" | prod: "prod" CNPG_BACKUP_PATH: "s3://cnpg/dev" # dev: s3://cnpg/dev | prod: s3://cnpg/prod # --- placement --- PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles PLATFORM_NODE_SELECTOR_VALUE: "linux"