jakman85andClaude Opus 5
3f980d2a17
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).
New cluster-vars knobs (base default = dev literal, per the file's existing convention):
K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml
CILIUM_LB_POOL_STOP "
UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml
CRAFTY_L4_VIP platform/crafty/service-l4.yaml
S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
semaphore}-db.yaml, and platform/velero/helmrelease.yaml
Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.
clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.
PROD ADDRESS PLAN (operator, 2026-08-31):
nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
Cilium LB pool .20-.30
CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.
Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.
DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.
Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
236 lines
10 KiB
YAML
236 lines
10 KiB
YAML
---
|
|
# clusters/dev — the ACTIVE build target overlay.
|
|
#
|
|
# `flux bootstrap ... --path=clusters/dev` points the dev cluster's Flux here. It reuses the base
|
|
# reconciliation graph and patches the shared cluster-vars ConfigMap with dev-scoped values (R-ENV-02).
|
|
# It also adds a self-managing ROOT Kustomization (sync.yaml) so the whole graph reconciles from git
|
|
# on every push — no more manual `kubectl apply -k` for cluster-vars/overlay changes (F-OPS1).
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- ../base
|
|
- sync.yaml # self-managing root Kustomization → ./clusters/dev
|
|
patches:
|
|
# --- SOPS decryption + bootstrap source (PM-inlined from bootstrap/flux/patches/, 2026-08-16) ---
|
|
# Inlined (not cross-dir file refs) to stay within kustomize's default load restrictions.
|
|
# Realises the commented decryption TODOs in clusters/base and sets the real source URL.
|
|
# The `sops-age` Secret is created out-of-band (bootstrap/sops/install-age-key.sh) — name only.
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: platform
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# SOPS decryption on EACH app wave (the app tier is now apps-wave-0..N, not a single `apps`
|
|
# Kustomization). Same stanza the retired `apps` Kustomization received — one per wave, by
|
|
# name. Add a patch here whenever `render.py --emit-waves` introduces a new deepest wave
|
|
# (DESIGN-waves.md cutover note). The `sops-age` Secret is created out-of-band (name only).
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-0
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-1
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-2
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# oauth2-proxy moved to its own post-platform Flux Kustomization (clusters/base/oauth2-proxy.yaml).
|
|
# It carries oauth2-proxy-credentials.sops.yaml, previously decrypted via the `platform`
|
|
# Kustomization; now that this tier owns the encrypted Secret it needs its own decryption stanza.
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: oauth2-proxy
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# headlamp moved to its own post-platform Flux Kustomization (clusters/base/headlamp.yaml).
|
|
# It carries headlamp-oidc-credentials.sops.yaml, previously decrypted via the `platform`
|
|
# Kustomization; now that this tier owns the encrypted Secret it needs its own decryption stanza.
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: headlamp
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# crafty has its own post-platform Flux Kustomization (clusters/base/crafty.yaml). It carries
|
|
# crafty-oauth2-proxy-credentials.sops.yaml, so it needs its own SOPS decryption stanza (same as
|
|
# headlamp/oauth2-proxy). The `sops-age` Secret is created out-of-band (name only).
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: crafty
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# velero (backup controller) has its own post-platform Flux Kustomization (clusters/base/backup.yaml).
|
|
# It carries velero-cloud.sops.yaml (the scoped MinIO `cloud` credential), so it needs its own SOPS
|
|
# decryption stanza (same as headlamp/oauth2-proxy/crafty). The `sops-age` Secret is out-of-band (name only).
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: velero
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: source.toolkit.fluxcd.io/v1
|
|
kind: GitRepository
|
|
metadata:
|
|
name: flux-system
|
|
namespace: flux-system
|
|
spec:
|
|
# Phase-6 cutover done: source is now the in-cluster Gitea; bootstrap host-Gitea retained
|
|
# stopped as recovery. PUBLIC repo => Flux needs no creds; Flux resolves the host in-cluster
|
|
# via coredns -> Gateway VIP, and the LE-prod wildcard cert is publicly trusted so NO
|
|
# secretRef/caCert is needed. Recovery: restart bootstrap Gitea (192.168.2.103:3000) and
|
|
# revert this url (docs/05).
|
|
url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
|
ref:
|
|
branch: main
|
|
# --- dev-scoped cluster-vars (R-ENV-02) ---
|
|
- target:
|
|
kind: ConfigMap
|
|
name: cluster-vars
|
|
patch: |-
|
|
- op: replace
|
|
path: /data/CLUSTER_ENV
|
|
value: "dev"
|
|
# DNS-01 via LE PRODUCTION: in-cluster OIDC clients (oauth2-proxy, Grafana) must trust the
|
|
# wildcard cert for SSO to work; the staging root isn't in any trust store. Prod issuer is
|
|
# registered + Ready; one wildcard cert is well within LE rate limits — ADR-0006, R-TLS-01/03.
|
|
- op: replace
|
|
path: /data/ACME_CLUSTER_ISSUER
|
|
value: "letsencrypt-prod"
|
|
# --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
|
|
# EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
|
|
# equals the base default. These values are what the LIVE dev cluster is running right now, so
|
|
# the substituted output is byte-identical to the pre-parameterization hardcoded manifests
|
|
# (verified 2026-08-31). Never let this block drift from the live cluster.
|
|
#
|
|
# Split-horizon DNS target: the Cilium-L2 VIP of the `shared` Gateway on the Proxmox LAN
|
|
# (= CILIUM_LB_POOL_START, the pool's first address). coredns-custom answers
|
|
# *.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
|
|
- op: replace
|
|
path: /data/GATEWAY_VIP
|
|
value: "192.168.1.53"
|
|
# k3s API endpoint Cilium's kube-proxy replacement dials — the primary control-plane node
|
|
# (k3s-1). [parity] equals the Ansible inventory's k3s_api_host for dev.
|
|
- op: replace
|
|
path: /data/K3S_API_HOST
|
|
value: "192.168.1.50"
|
|
# Cilium LB IP pool = the tofu reservation cilium_l2_vip_range .53-.60, outside the node IPs
|
|
# (.50/.51/.52) and outside the DHCP scope. .53 Gateway, .54 unifi, .55 crafty; .56-.60 free.
|
|
- op: replace
|
|
path: /data/CILIUM_LB_POOL_START
|
|
value: "192.168.1.53"
|
|
- op: replace
|
|
path: /data/CILIUM_LB_POOL_STOP
|
|
value: "192.168.1.60"
|
|
# Pinned L4 VIPs out of that pool (lbipam.cilium.io/ips). Changing either MOVES a live device-
|
|
# facing endpoint: unifi is the inform/STUN target every adopted UniFi device holds.
|
|
- op: replace
|
|
path: /data/UNIFI_L4_VIP
|
|
value: "192.168.1.54"
|
|
- op: replace
|
|
path: /data/CRAFTY_L4_VIP
|
|
value: "192.168.1.55"
|
|
# Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
|
|
# barmanObjectStore and Velero's backupStorageLocation write to.
|
|
- op: replace
|
|
path: /data/S3_BACKUP_ENDPOINT
|
|
value: "http://192.168.1.9:9000"
|
|
# dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
|
|
- op: replace
|
|
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
|
value: "3"
|
|
# smaller retention / volumes to keep the dev workstation footprint light.
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_RETENTION
|
|
value: "3d"
|
|
# 5Gi was too tight: 3d of this cluster's series volume (~1.2Gi/day of blocks) plus a WAL that
|
|
# ballooned past 1Gi overran the disk, wedging WAL commits and firing every absent()/==0 alert
|
|
# falsely. Bumped to 8Gi (still lighter than base 10Gi) so the size-cap below has real headroom.
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_PVC_SIZE
|
|
value: "8Gi"
|
|
# Size-cap kept well below the 8Gi PVC (headroom for WAL + head + compaction scratch) so
|
|
# Prometheus prunes oldest blocks by size and never fills the disk on this young cluster.
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
|
|
value: "5GiB"
|
|
- op: replace
|
|
path: /data/MONITORING_LOKI_PVC_SIZE
|
|
value: "5Gi"
|
|
# single-replica control-plane addons in dev (no HA needed to iterate).
|
|
- op: replace
|
|
path: /data/KEYCLOAK_REPLICAS
|
|
value: "1"
|
|
- op: replace
|
|
path: /data/OAUTH2_PROXY_REPLICAS
|
|
value: "1"
|
|
- op: replace
|
|
path: /data/VELERO_RETENTION
|
|
value: "72h"
|
|
# dev-softened Kyverno for cold-start (base defaults Enforce/Fail/3 = prod posture).
|
|
# Not credential-driven — bootstrap-safety knobs so a cold cluster comes up before
|
|
# admission is proven/HA; flip dev to the strict posture once green.
|
|
- op: replace
|
|
path: /data/KYVERNO_VALIDATION_ACTION
|
|
value: "Audit"
|
|
- op: replace
|
|
path: /data/KYVERNO_FORCE_FAILURE_POLICY_IGNORE
|
|
value: "true"
|
|
# 2 (not base-default 1): the webhook is fail-closed, so a single-pod restart 502s every
|
|
# admission-gated write cluster-wide. Two replicas give a fail-closed webhook availability
|
|
# headroom during a restart/upgrade while staying dev-modest (prod runs 3).
|
|
- op: replace
|
|
path: /data/KYVERNO_ADMISSION_REPLICAS
|
|
value: "2"
|