Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
666a60776a981c21d895e1fd079e286135939640
home-network-k3s-impl/clusters/prod/kustomization.yaml
T
jakman85andClaude Opus 5 666a60776a fix(flux): close the substitution race — platform dependsOn root + strict substitution (P0/P1) 
P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix
for the incident, not the feature gate. Verified against kustomize-controller v1.4.0
`kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION-
gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent
cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision
N is exactly what writes a newly-added cluster-vars knob.

Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked
the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all
11 substituting Kustomizations chain transitively through it (platform-config -> platform;
the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config).
The "not chaining through platform" set is EMPTY, so this one edge protects the whole
graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/
sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready
on apply success without health-gating its children.

Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest
anywhere under clusters/<env> — including clusters/base — now stalls platform and
everything downstream, where today it would not. Fail-closed, per doctrine.

P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it
matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README).
There is no gotk-components.yaml in git, and the live kustomize-controller Deployment
carries NO Flux ownership labels — it was installed imperatively by `flux install` from
bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT
the controllers. Nothing under clusters/** can reach a controller flag.

So it uses Flux's documented mechanism for hand-installed clusters: a new
bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a
JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to
kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a
container's args is atomic and a merge patch would replace all five install-generated
args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the
patch, ASSERT the gate string is present so a future Flux bump that breaks the patch
target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is
vendored, so components can never drift from the pinned FLUX_VERSION.

Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true
would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial
Deployment as a Flux-managed resource (a future prune would delete kustomize-controller).

ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive
on a git push — the controller Deployment is outside Flux's inventory, so an operator must
re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is
level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled
BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race
turns a transient empty render into a transient hard Kustomization failure.

ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had
no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own
sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left
platform blocked forever on a flux-system Kustomization that does not exist, and nothing
downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev.

Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7):
prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs
the prod age-key decision, a security choice not a copy), and no GitRepository source URL
patch (needs a decision on whether prod tracks the same branch as dev, which would make a
dev-facing merge reconcile prod).

Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS.

Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 01:45:04 -05:00

127 lines
7.1 KiB
YAML
Raw Blame History

---
# clusters/prod — DESIGN-ONLY overlay. NOT a build target until roadmap Phase 7 opens prod
# (spec roadmap/phases.md, AGENTS.md §6: "Never build directly on production").
#
# When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
# here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
# values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
#
# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
# done here: prod is design-only and its root object is the prod bring-up owner's call.)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
- sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
patches:
- target:
kind: ConfigMap
name: cluster-vars
patch: |-
- op: replace
path: /data/CLUSTER_ENV
value: "prod"
# trusted, rate-limited real certs in prod — ADR-0006.
- op: replace
path: /data/ACME_CLUSTER_ISSUER
value: "letsencrypt-prod"
- op: replace
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
value: "3"
# longer retention / larger volumes on prod NVMe.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION
value: "30d"
- op: replace
path: /data/MONITORING_PROMETHEUS_PVC_SIZE
value: "50Gi"
# Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
# size cap before 30d if series volume is high, keeping the disk from filling either way.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
value: "45GiB"
- op: replace
path: /data/MONITORING_LOKI_PVC_SIZE
value: "50Gi"
# HA control-plane addons in prod.
- op: replace
path: /data/KEYCLOAK_REPLICAS
value: "2"
- op: replace
path: /data/OAUTH2_PROXY_REPLICAS
value: "2"
- op: replace
path: /data/VELERO_RETENTION
value: "720h"
# ==========================================================================================
# PROD LAN ADDRESSING (192.168.2.0/24). Nodes: k3s-prod-1/2/3 = .10 / .11 / .12.
# ==========================================================================================
# These knobs were HARDCODED to the dev LAN inside base manifests until 2026-08-31; prod would
# have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN. They are knobs now, and
# this overlay supplies every one of them. Anything the operator has NOT yet decided stays an
# INVALID `TBD-PROD-*` placeholder on purpose: base defaults are the DEV values, so an OMITTED
# op means prod silently inherits a dev address — far worse than a failed apply. `TBD-PROD-*`
# is not a valid IP/URL, so Cilium CRD validation / LB-IPAM / CNPG / Velero reject it LOUDLY.
#
# ------------------------------------------------------------------------------------------
# ADDRESS PLAN (operator, 2026-08-31) — see infra/PROD-PROVISIONING.md §4 for the full table.
# ------------------------------------------------------------------------------------------
# nodes 192.168.2.10 / .11 / .12 (Ansible inventory)
# control-plane VIP 192.168.2.9 (Ansible group_vars/k3s_prod — NOT here)
# ingress Gateway VIP 192.168.2.20 (GATEWAY_VIP, below)
# Cilium LB pool 192.168.2.20 - .30 (CILIUM_LB_POOL_*, below)
#
# CONTAINMENT INVARIANT SATISFIED: GATEWAY_VIP (.20) IS CILIUM_LB_POOL_START, mirroring dev
# (.53 of .53-.60). This matters because the `shared` Gateway (platform/gateway/gateway.yaml)
# pins NOTHING — no `lbipam.cilium.io/ips`, no spec.addresses (unlike platform/crafty and
# apps/vms/unifi, which DO pin) — so Cilium LB-IPAM assigns it the FIRST FREE address of a
# matching pool block. Had GATEWAY_VIP stayed outside every pool (the original .21-.30
# proposal), the Gateway would have come up on .21 while split-horizon coredns-custom kept
# answering *.home.keithjackson.dev with .20 — an address nothing announces — blackholing
# in-cluster hairpin traffic (OIDC clients -> Keycloak/Grafana) while looking healthy.
# KEEP GATEWAY_VIP == CILIUM_LB_POOL_START when changing either.
# The control-plane VIP (.9) is deliberately OUTSIDE this pool: kube-vip floats it, and
# LB-IPAM must never be able to hand it to a Service.
- op: replace
path: /data/GATEWAY_VIP
value: "192.168.2.20"
# Cilium LB IP pool, operator-supplied: 192.168.2.20-.30 (START == GATEWAY_VIP, see above).
# Mirror this range into the prod tofu reservation (cilium_l2_vip_range) if prod ever gets one.
- op: replace
path: /data/CILIUM_LB_POOL_START
value: "192.168.2.20"
- op: replace
path: /data/CILIUM_LB_POOL_STOP
value: "192.168.2.30"
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
# k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10. It is owned
# Ansible-side (prod inventory k3s_api_host) and MUST equal that value — [parity] contract in
# platform/cilium/helmrelease.yaml. Not yet decided.
- op: replace
path: /data/K3S_API_HOST
value: "192.168.2.10"
# Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
# prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
# they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
# collide with the Gateway VIP or the node IPs (.10/.11/.12).
- op: replace
path: /data/UNIFI_L4_VIP
value: "192.168.2.21"
- op: replace
path: /data/CRAFTY_L4_VIP
value: "192.168.2.22"
# Off-cluster DR target. Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the
# network — a prod backup written there is not off-cluster DR and would mix two environments'
# PITR streams in one bucket. Operator supplies the prod endpoint (full URL incl. scheme+port).
- op: replace
path: /data/S3_BACKUP_ENDPOINT
value: "TBD-PROD-S3-BACKUP-ENDPOINT"
# TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.
Reference in new issue View Git Blame Copy Permalink