jakman85andClaude Opus 5
9137bbd0c3
*** DO NOT MERGE TO main WHILE THE OPERATOR IS AWAY. MERGING IS THE MIGRATION. ***
Operator simplified the architecture, inverting c8c1d29:
dev -> *.dev.home.keithjackson.dev (was the apex)
prod -> *.home.keithjackson.dev (the apex; no prod. subdomain, no CNAME layer)
WHY MERGING IS DANGEROUS, and this is the finding that matters most here:
clusters/dev/sync.yaml self-reconciles ./clusters/dev from `main` every 10 minutes. Merging
these files EXECUTES the cutover on the live dev cluster, unattended. And because the same
commit moves dev's GitRepository url to gitea.dev.home.keithjackson.dev, dev STRANDS — it
stops reconciling and cannot be fixed by pushing, because the broken thing IS the path Flux
reads pushes through. Banners added at the top of clusters/dev/kustomization.yaml and beside
the url itself. That url cannot be tokenized: the root Kustomization declares no
substituteFrom, so `gitea.${BASE_DOMAIN}` would ship the literal token.
Verified dev is currently safe: this work is NOT on origin/main, dev is reconciling
main@4809199, and its live BASE_DOMAIN is still home.keithjackson.dev. Deliberately NOT
pushed to the bootstrap Gitea either — prod reconciles from there, and prod on `prod.home.`
cannot collide with dev on the apex, so that is the safer interim for prod.
VALUES
BASE_DOMAIN TXT_OWNER_ID DOMAIN_FILTER
base dev.home.keithjackson.dev sentinel keithjackson.dev
dev dev.home.keithjackson.dev dev keithjackson.dev
prod home.keithjackson.dev prod keithjackson.dev
Ansible seam inverted too: role default k3s_base_domain = dev's subdomain, group_vars/k3s_prod
overrides to the apex.
dev's txtOwnerId moves `default` -> `dev`. The old value was FORCED (its live apex records
were already registered under `default`, so renaming would have orphaned them). On a new
suffix dev publishes a brand-new record set with no prior registry, so the id is chosen on
merits — and `default` has none: it is what you get by FORGETTING the flag, which is exactly
how both clusters came to share it and delete each other's records. Operationally it also
means dev's old apex records stay owned by a now-unclaimed id, so nothing deletes them
implicitly and vacating the apex becomes an explicit sequenceable step rather than a race
between two reconcile loops. PROD MUST NEVER BE GIVEN `default`: prod inherits the apex
NAMES, so `default` is the one value that would let it adopt and policy:sync-delete dev's
still-live apex records — the same incident with the roles swapped.
DOMAIN_FILTER stays the ZONE for both, with F-DNS1 written into all three files as a
do-not-retry: a subdomain filter drops the parent zone and publishes NOTHING.
VERIFICATION — the property changed, and so did the method. "Dev renders byte-identically" no
longer applies because dev is deliberately migrating. Instead: apply the INVERSE substitution
to the new render and demand byte-identity with the baseline. 145 changed line pairs, every
one classified: 143 hostname-move, 2 declared owner-id, ZERO unexplained, ZERO bare-apex
remnants. cilium and velero show 0 changes and provably contain no hostname, so they are not
blind spots. Prod: 0 occurrences of prod.home., 0 of dev.home., 143 apex names. Certs: dev
*.dev.home.keithjackson.dev, prod *.home.keithjackson.dev.
THREE positive controls, all fire — including one that proves the two checks are not
redundant: injecting a single un-moved hostname SURVIVES the inverse-substitution check and is
caught only by the independent bare-apex scan. That is why both exist.
Ansible probe (connection: local, zero SSH): dev nodes -> keycloak.dev.home..., prod nodes ->
keycloak.home.... Used roles: + --tags probe rather than include_vars, because include_vars
outranks group_vars and would have FALSELY shown prod inheriting dev's value.
Gates all PASS: l0, l1, render-golden (10 goldens), lint, substitution (254 assertions),
render.py --selftest, spec-immutability. Catalog ServiceDefinitions edited, never rendered apps/.
RUNBOOK rewritten in §10 + §10a with the ordering that avoids stranding: D0 snapshot escape
routes -> D1 break the automatic trigger -> D2 publish and PROVE gitea.dev.home + the new
wildcard cert WHILE dev still serves the apex -> D3 merge -> D4 release reconciliation
deliberately -> D5 Keycloak issuer + ~12 clients (headlamp's is SOPS, by hand) -> D6
one-server-at-a-time k3s restart + SSO re-issue (service restart only, NO reboots) -> D7
retire old apex records -> D8 verify the apex is empty AGAINST CLOUDFLARE, not pfSense —
the gate on everything after. Then P1 unsuspend prod.
ACME: prod's wildcard is EXACTLY the name set dev holds today, so prod's orders share dev's
5-per-week duplicate-certificate bucket. Order after D8, delete dev's old Certificate rather
than let it renew, rehearse on staging.
The 6 L2 chainsaw tests hardcoding the apex are enumerated but deliberately NOT edited — while
dev is live on the apex they are correct; they need per-environment parameterisation.
Serves: R-ENV-02, R-DNS-01/02/03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
218 lines
16 KiB
YAML
218 lines
16 KiB
YAML
---
|
|
# cluster-vars — the overlay knob surface (R-ENV-02).
|
|
#
|
|
# This ConfigMap holds BASE DEFAULTS. Flux `postBuild.substituteFrom` injects these into every
|
|
# `${TOKEN}` in platform/ and apps/ manifests at reconcile time. dev/ and prod/ overlays STRATEGIC-
|
|
# MERGE PATCH this single ConfigMap — that patch is the *entire* difference between environments.
|
|
#
|
|
# Add a knob here (with a base default) before referencing `${KNOB}` in a component manifest, then
|
|
# give each overlay its env-scoped value. Never fork a base manifest to differ per environment.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: cluster-vars
|
|
namespace: flux-system
|
|
data:
|
|
# --- identity of the environment ---
|
|
CLUSTER_ENV: "base" # dev | prod (overlays set this)
|
|
|
|
# --- DNS / TLS ---
|
|
# THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
|
|
# the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
|
|
# ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
|
|
#
|
|
# THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
|
|
# hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
|
|
# authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
|
|
# (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
|
|
# See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
|
|
# suffix, so the two clusters can no longer describe the same name.
|
|
#
|
|
# dev -> dev.home.keithjackson.dev (a subdomain; = this base default)
|
|
# prod -> home.keithjackson.dev (THE APEX — prod IS the apex, no `prod.` label)
|
|
#
|
|
# OPERATOR DECISION 2026-09-04 (supersedes the interim split committed earlier the same day, which
|
|
# had prod on `prod.home.` with apex CNAMEs pointing at it). PROD IS THE APEX. That deletes the
|
|
# whole CNAME layer — `<svc>.home.keithjackson.dev` is prod's real name, not an alias — and it is
|
|
# DEV that carries a label. Convention is unchanged from GATEWAY_VIP / LONGHORN_DATA_PATH: the base
|
|
# default is the DEV value (dev restates it verbatim) and prod overrides it. What changed is which
|
|
# string dev holds.
|
|
#
|
|
# DEV'S VALUE IS A MIGRATION, NOT A VARIABLE CHANGE. The live dev cluster is serving the APEX right
|
|
# now: its wildcard cert, every public DNS record, its Keycloak issuer, ~12 OIDC clients' redirect
|
|
# URIs, kubectl SSO, and gitea.home.keithjackson.dev — which IS this repo's GitOps source AND the
|
|
# push target. Flipping this value on dev without first standing up the new host set STRANDS DEV:
|
|
# Flux's GitRepository url moves to gitea.dev.home.keithjackson.dev, which has no DNS record and no
|
|
# cert, so dev stops reconciling and cannot be fixed by pushing. ORDERED CUTOVER, operator present:
|
|
# infra/PROD-PROVISIONING.md §10.
|
|
#
|
|
# DEV MUST VACATE THE APEX BEFORE PROD PUBLISHES THERE. Both clusters describing
|
|
# `*.home.keithjackson.dev` at once is the 2026-09-04 incident precondition, re-created.
|
|
BASE_DOMAIN: "dev.home.keithjackson.dev"
|
|
|
|
# ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
|
|
# EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
|
|
# so it cannot be created, updated or deleted regardless of TXT ownership.
|
|
#
|
|
# BOTH ENVIRONMENTS USE THE CLOUDFLARE **ZONE**, `keithjackson.dev`. This value is NOT a per-
|
|
# environment isolation knob and must not be narrowed to either environment's BASE_DOMAIN.
|
|
#
|
|
# FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) — the Cloudflare provider uses --domain-filter for ZONE
|
|
# SELECTION as well as name filtering, so a filter that is a STRICT SUBDOMAIN of the hosted zone
|
|
# matches NO zone: ExternalDNS generates the record and then skips it with "no hosted zone matching
|
|
# record DNS Name", publishing NOTHING. This has been tried and reverted more than once (an agent
|
|
# narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in review). DO NOT "harden"
|
|
# it — the change is not defence in depth, it is a total outage of DNS publication.
|
|
#
|
|
# WHERE ISOLATION ACTUALLY COMES FROM, and it is sufficient:
|
|
# (1) the two environments request DISJOINT hostname sets (dev.home.… vs the apex), and
|
|
# (2) EXTERNALDNS_TXT_OWNER_ID differs, so neither registry ever claims the other's records.
|
|
# That pairing is what the 2026-09-04 incident needed; a domain filter cannot provide it, because
|
|
# both environments live in the one zone either way.
|
|
EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
|
|
|
|
# ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
|
|
# neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
|
|
# record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
|
|
# `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
|
|
# one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
|
|
#
|
|
# dev -> `dev` prod -> `prod` base default -> a non-colliding sentinel (below)
|
|
#
|
|
# DEV IS `dev`, NOT `default` (changed 2026-09-04 with the move to dev.home.keithjackson.dev). Until
|
|
# this migration dev HAD to render `default`, because its LIVE apex records were already registered
|
|
# in Cloudflare's TXT registry under that owner and renaming would have orphaned every one of them.
|
|
# THAT CONSTRAINT NO LONGER BINDS: dev is publishing an ENTIRELY NEW record set under
|
|
# *.dev.home.keithjackson.dev, with no pre-existing registry entries to recognise, so the owner id
|
|
# is now free to be chosen on its merits. Two reasons to take `dev`:
|
|
# - `default` is THE VALUE YOU GET BY FORGETTING (it is the external-dns chart default, and both
|
|
# clusters silently shared it, which IS the incident). An environment whose identity is the
|
|
# forget-value can never be told apart from a misconfigured newcomer. No environment should
|
|
# hold it.
|
|
# - It makes vacating the apex an EXPLICIT, ORDERED, VERIFIABLE step. Dev's old apex records stay
|
|
# owned by `default`, which no environment now claims, so nothing deletes them implicitly on
|
|
# some future reconcile — they are retired deliberately, and prod's publish is gated on that
|
|
# retirement being confirmed in Cloudflare (infra/PROD-PROVISIONING.md §10) rather than on a
|
|
# race between two reconcile loops.
|
|
#
|
|
# PROD STAYS `prod` AND MUST NOT BECOME `default`, even though prod now serves the apex names that
|
|
# dev's `default`-owned records currently occupy. Giving prod `default` would hand prod ownership of
|
|
# the LIVE dev cluster's records while dev is still serving them, and under `policy: sync` prod
|
|
# would delete the ones no prod source claims. That is the 2026-09-04 incident, re-armed, with the
|
|
# roles swapped. Prod inherits the apex NAMES; it must never inherit the apex OWNER ID.
|
|
#
|
|
# THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default` and not any real
|
|
# environment's id. A new environment that forgets this knob must not silently inherit another's
|
|
# identity — that is precisely the incident. With this sentinel such a cluster owns nothing, so it
|
|
# can DELETE nothing; it would publish under a visibly wrong owner id (grep the TXT records)
|
|
# instead of destroying a live environment's DNS.
|
|
EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
|
|
|
|
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
|
|
GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
|
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
|
|
# ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
|
|
# takes the pool's first free IP, and coredns-custom answers
|
|
# *.${BASE_DOMAIN} with THIS so pod -> public-hostname traffic
|
|
# hairpins through the Gateway (correct SNI -> wildcard cert).
|
|
# Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
|
|
# long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
|
|
# forgets this knob at least lands on a REAL network). A prod overlay
|
|
# MUST still set the prod LAN VIP alongside the prod L2 pool range.
|
|
|
|
# --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply -------------
|
|
# These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint:
|
|
# `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24)
|
|
# — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values
|
|
# (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the
|
|
# dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block
|
|
# before `flux bootstrap --path=clusters/prod`.
|
|
#
|
|
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
|
|
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
|
|
# (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
|
|
# velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
|
|
# ship the literal string into the cluster.
|
|
|
|
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
|
|
# [parity] MUST equal the Ansible bootstrap's k3s_api_host (infra/ansible/roles/cilium/defaults/
|
|
# main.yml -> cilium-values.yaml.j2, already inventory-parameterized). Wrong value = Cilium cannot
|
|
# reach the API server on adopt and the CNI breaks. Port stays 6443 in both envs, so it is NOT a knob.
|
|
K3S_API_HOST: "192.168.1.50"
|
|
|
|
# CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
|
|
# L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
|
|
# tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in
|
|
# practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's
|
|
# first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and
|
|
# the flagged block in clusters/prod/kustomization.yaml.)
|
|
CILIUM_LB_POOL_START: "192.168.1.53"
|
|
CILIUM_LB_POOL_STOP: "192.168.1.60"
|
|
|
|
# Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the
|
|
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
|
|
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
|
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
|
|
|
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
|
|
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
|
|
# nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
|
|
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR.
|
|
S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
|
|
|
|
# --- storage (ADR-0003) ---
|
|
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
|
# Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
|
|
# replica is written into. It must be the path the NODE's own data disk is actually mounted at —
|
|
# roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
|
|
# fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
|
|
# Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
|
|
# working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
|
|
# on it — changing dev's value would orphan every replica, so the dev overlay restates it
|
|
# verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
|
|
# dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
|
|
# neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
|
|
LONGHORN_DATA_PATH: "/var/lib/longhorn"
|
|
|
|
# --- observability sizing (docs/10) ---
|
|
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
|
|
MONITORING_PROMETHEUS_PVC_SIZE: "10Gi"
|
|
# SIZE-based retention cap for Prometheus (Prometheus *iB units, NOT the K8s `Gi` of the PVC knob).
|
|
# MUST stay below MONITORING_PROMETHEUS_PVC_SIZE — leaves headroom for the WAL + head + compaction
|
|
# scratch (retentionSize counts only persisted blocks). Prevents a busy TSDB from filling /prometheus
|
|
# (a full disk wedges all WAL commits and makes every absent()/==0 alert fire falsely). Base 10Gi PVC.
|
|
MONITORING_PROMETHEUS_RETENTION_SIZE: "8GiB"
|
|
MONITORING_LOKI_PVC_SIZE: "10Gi"
|
|
|
|
# --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture;
|
|
# the dev overlay softens them for bootstrap (Audit / Ignore / 1) so a cold cluster can
|
|
# come up before policies are proven and admission is HA. ---
|
|
KYVERNO_VALIDATION_ACTION: "Enforce" # dev: Audit (observe before enforcing)
|
|
KYVERNO_FORCE_FAILURE_POLICY_IGNORE: "false" # dev: "true" (chart forceFailurePolicyIgnore = fail-open)
|
|
KYVERNO_ADMISSION_REPLICAS: "3" # dev: 1 (no HA needed to iterate)
|
|
|
|
# --- egress / VPN (Phase 3, ADR-0015) — node-level host OpenVPN tunnel steered by Cilium Egress
|
|
# Gateway (platform/vpn-egress/egress-steering.yaml). No in-cluster gateway pod. ---
|
|
PIA_REGION: "SE Stockholm" # PF-capable PIA region (Sweden). Under ADR-0015 the tunnel is
|
|
# host-side (Ansible vault / the node's .ovpn per docs/07), so this
|
|
# is now a reference default only — retained, not substituted into
|
|
# any manifest. ("SE Stockholm" is PF-capable; NOT "SE Streaming
|
|
# Optimized" / a US region, which lack PF.)
|
|
CLUSTER_POD_CIDR: "10.0.0.0/16" # Documents the Cilium cluster-pool pod CIDR that the egress-steering
|
|
# CiliumEgressGatewayPolicy carves out of the tunnel via excludedCIDRs
|
|
# (kept on the LOCAL datapath, never SNAT'd out tun0). The policy
|
|
# currently excludes the whole /8 RFC1918 range; this /16 records the
|
|
# actual pod allocations (Cilium assigns /24 per node from 10.0.0.0 up).
|
|
|
|
# --- identity / auth (ADR-0014) ---
|
|
KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
OAUTH2_PROXY_REPLICAS: "1" # dev 1 | prod 2 (HA)
|
|
|
|
# --- backup / DR (ADR-0003, R-STATE-03) ---
|
|
VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ
|
|
VELERO_RETENTION: "168h" # dev short | prod long
|
|
|
|
# --- placement ---
|
|
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
|
|
PLATFORM_NODE_SELECTOR_VALUE: "linux"
|