feat(env): tokenize hardcoded dev LAN addresses into cluster-vars (R-ENV-02)
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).
New cluster-vars knobs (base default = dev literal, per the file's existing convention):
K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml
CILIUM_LB_POOL_STOP "
UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml
CRAFTY_L4_VIP platform/crafty/service-l4.yaml
S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
semaphore}-db.yaml, and platform/velero/helmrelease.yaml
Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.
clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.
PROD ADDRESS PLAN (operator, 2026-08-31):
nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
Cilium LB pool .20-.30
CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.
Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.
DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.
Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
e3c44368c8
commit
3f980d2a17
17 files changed
+269
-52
No files matched your search
@@ -23,7 +23,21 @@ spec:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./apps/vms
|
||||
# NO postBuild.substituteFrom: Flux's envsubst would run over the ENTIRE manifest, including the
|
||||
# VM cloud-init shell scripts, and mangle every `${...}` (e.g. ${UOS_URL}, ${PIPESTATUS[0]}) —
|
||||
# corrupting the in-guest install. VM manifests reference no cluster-vars, so substitution is
|
||||
# unneeded here. A future VM that needs a cluster-var must escape its shell `${...}` as `$${...}`.
|
||||
# postBuild.substituteFrom ENABLED 2026-08-31 (R-ENV-02). It was previously omitted because Flux's
|
||||
# envsubst runs over the ENTIRE manifest — including VM cloud-init shell scripts — and would mangle
|
||||
# every `${...}`; the omission was safe only while no VM manifest referenced a cluster-var. It now
|
||||
# does: apps/vms/unifi/50-service-l4.yaml pins the UniFi L4 VIP from ${UNIFI_L4_VIP} instead of the
|
||||
# hardcoded dev-LAN literal that would have shipped 192.168.1.54 onto the PROD LAN.
|
||||
#
|
||||
# The escape contract the old comment prescribed is now in force and MUST be honoured by every VM
|
||||
# manifest under this path (verified with `flux envsubst`):
|
||||
# `${VAR}` -> substituted (empty when unset) => escape shell braces as `$${VAR}`, which
|
||||
# envsubst emits back as a literal `${VAR}`.
|
||||
# `$VAR` -> left alone by envsubst => bare shell vars need NO escaping.
|
||||
# `$(cmd)` -> left alone by envsubst => command substitution needs NO escaping.
|
||||
# `${ARR[0]}` (e.g. ${PIPESTATUS[0]}) -> HARD ERROR "missing closing brace", which fails this
|
||||
# whole Kustomization. NEVER use an indexed expansion here, escaped or not.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
@@ -21,13 +21,56 @@ data:
|
||||
DNS_ZONE: "keithjackson.dev" # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
|
||||
# ExternalDNS domain-filter must be the zone, not the subdomain
|
||||
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
|
||||
GATEWAY_VIP: "10.73.0.240" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
||||
# ingress VIP of the `shared` Gateway (first IP of the Cilium
|
||||
# LB IP pool 10.73.0.240-.250, platform-config/cilium/l2-ip-pool.yaml).
|
||||
# coredns-custom answers *.home.keithjackson.dev with THIS so pod->
|
||||
# public-hostname traffic hairpins through the Gateway (correct SNI ->
|
||||
# wildcard cert). Base default = dev VIP; a prod overlay must set the
|
||||
# prod LAN VIP alongside the prod L2 pool range when hardware is built.
|
||||
GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
||||
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
|
||||
# ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
|
||||
# takes the pool's first free IP, and coredns-custom answers
|
||||
# *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
|
||||
# hairpins through the Gateway (correct SNI -> wildcard cert).
|
||||
# Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
|
||||
# long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
|
||||
# forgets this knob at least lands on a REAL network). A prod overlay
|
||||
# MUST still set the prod LAN VIP alongside the prod L2 pool range.
|
||||
|
||||
# --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply -------------
|
||||
# These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint:
|
||||
# `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24)
|
||||
# — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values
|
||||
# (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the
|
||||
# dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block
|
||||
# before `flux bootstrap --path=clusters/prod`.
|
||||
#
|
||||
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
|
||||
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
|
||||
# (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
|
||||
# velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
|
||||
# ship the literal string into the cluster.
|
||||
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
|
||||
# [parity] MUST equal the Ansible bootstrap's k3s_api_host (infra/ansible/roles/cilium/defaults/
|
||||
# main.yml -> cilium-values.yaml.j2, already inventory-parameterized). Wrong value = Cilium cannot
|
||||
# reach the API server on adopt and the CNI breaks. Port stays 6443 in both envs, so it is NOT a knob.
|
||||
K3S_API_HOST: "192.168.1.50"
|
||||
|
||||
# CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
|
||||
# L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
|
||||
# tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in
|
||||
# practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's
|
||||
# first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and
|
||||
# the flagged block in clusters/prod/kustomization.yaml.)
|
||||
CILIUM_LB_POOL_START: "192.168.1.53"
|
||||
CILIUM_LB_POOL_STOP: "192.168.1.60"
|
||||
|
||||
# Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the
|
||||
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
|
||||
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
||||
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
||||
|
||||
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
|
||||
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
|
||||
# nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
|
||||
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR.
|
||||
S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
|
||||
|
||||
# --- storage (ADR-0003) ---
|
||||
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
||||
|
||||
Reference in new issue
Block a user