jakman85andClaude Opus 5
6b4da27f84
Operator decisions, 2026-09-01:
1. Prod uses the SAME SOPS age key as dev.
2. Prod tracks the SAME BRANCH as dev (`main`). Dev is a temporary instance and will be
decommissioned once prod is deemed stable.
SOPS: eight `decryption: {provider: sops, secretRef: sops-age}` patches mirroring dev
exactly — platform, apps-wave-0/1/2, oauth2-proxy, headlamp, crafty, velero. The sops-age
Secret is still created OUT OF BAND on prod (bootstrap/sops/install-age-key.sh); the
overlay references it by name only, never inlines key material.
RECORDED CONSEQUENCE of the shared branch: once prod is live, EVERY MERGE TO main
RECONCILES BOTH CLUSTERS. There is no promotion step and no soak window — dev and prod
move together. That is a deliberate consequence of dev being temporary, not an oversight,
but it removes the "prove it on dev first" safety net that the last few days have leaned
on heavily. Worth revisiting if dev outlives its expected life.
*** BOOTSTRAP BLOCKER recorded, NOT papered over *** — the source URL cannot work on a
cold prod cluster, for two independent reasons found while wiring this:
1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER — the one slated for
decommission. Prod syncing from it means prod's source of truth lives inside the
cluster being retired. Gitea must move to prod (or off-cluster) BEFORE dev is torn
down. This makes Gitea migration the FIRST task of the decommission plan, not the last.
2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP} via a template
plugin. On prod that is 192.168.2.20 — PROD's own gateway, where no Gitea listens. Prod
Flux would resolve gitea.home.keithjackson.dev to itself and never reach a repository.
This is the same split-horizon mechanism that makes dev work, inverted against us.
Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
mirror) with a cutover to the in-cluster URL once Gitea actually runs on prod — precisely
the sequence docs/05 describes for dev's own Phase-6 cutover. The branch is settled and
correct; only the host is blocked. Documented at the patch site and in
PROD-PROVISIONING.md §7.
Gate: MERGE GATE: PASS.
Serves: R-SEC-01, R-GITOPS-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
250 lines
11 KiB
YAML
250 lines
11 KiB
YAML
---
|
|
# clusters/prod — DESIGN-ONLY overlay. NOT a build target until roadmap Phase 7 opens prod
|
|
# (spec roadmap/phases.md, AGENTS.md §6: "Never build directly on production").
|
|
#
|
|
# When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
|
|
# here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
|
|
# values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
|
|
#
|
|
# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
|
|
# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
|
|
# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
|
|
# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
|
|
# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
|
|
# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
|
|
# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
|
|
# done here: prod is design-only and its root object is the prod bring-up owner's call.)
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- ../base
|
|
- sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
|
|
patches:
|
|
# --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) ---
|
|
# Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets.
|
|
# The `sops-age` Secret is created OUT OF BAND on the prod cluster
|
|
# (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined.
|
|
# Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave.
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: platform
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-0
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-1
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-2
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: oauth2-proxy
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: headlamp
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: crafty
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: velero
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
|
|
#
|
|
# *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. ***
|
|
#
|
|
# Two independent reasons, both consequences of the split-horizon design:
|
|
# 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
|
|
# GitOps source lives inside the very cluster slated for decommission. Gitea must move to
|
|
# prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
|
|
# 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
|
|
# answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
|
|
# 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
|
|
# gitea.home.keithjackson.dev to itself and never reach a repository.
|
|
#
|
|
# Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
|
|
# mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the
|
|
# sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7.
|
|
#
|
|
# The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
|
|
# CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
|
|
# CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
|
|
- patch: |-
|
|
apiVersion: source.toolkit.fluxcd.io/v1
|
|
kind: GitRepository
|
|
metadata:
|
|
name: flux-system
|
|
namespace: flux-system
|
|
spec:
|
|
url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
|
ref:
|
|
branch: main
|
|
- target:
|
|
kind: ConfigMap
|
|
name: cluster-vars
|
|
patch: |-
|
|
- op: replace
|
|
path: /data/CLUSTER_ENV
|
|
value: "prod"
|
|
# trusted, rate-limited real certs in prod — ADR-0006.
|
|
- op: replace
|
|
path: /data/ACME_CLUSTER_ISSUER
|
|
value: "letsencrypt-prod"
|
|
- op: replace
|
|
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
|
value: "3"
|
|
# longer retention / larger volumes on prod NVMe.
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_RETENTION
|
|
value: "30d"
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_PVC_SIZE
|
|
value: "50Gi"
|
|
# Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
|
|
# size cap before 30d if series volume is high, keeping the disk from filling either way.
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
|
|
value: "45GiB"
|
|
- op: replace
|
|
path: /data/MONITORING_LOKI_PVC_SIZE
|
|
value: "50Gi"
|
|
# HA control-plane addons in prod.
|
|
- op: replace
|
|
path: /data/KEYCLOAK_REPLICAS
|
|
value: "2"
|
|
- op: replace
|
|
path: /data/OAUTH2_PROXY_REPLICAS
|
|
value: "2"
|
|
- op: replace
|
|
path: /data/VELERO_RETENTION
|
|
value: "720h"
|
|
# ==========================================================================================
|
|
# PROD LAN ADDRESSING (192.168.2.0/24). Nodes: k3s-prod-1/2/3 = .10 / .11 / .12.
|
|
# ==========================================================================================
|
|
# These knobs were HARDCODED to the dev LAN inside base manifests until 2026-08-31; prod would
|
|
# have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN. They are knobs now, and
|
|
# this overlay supplies every one of them. Anything the operator has NOT yet decided stays an
|
|
# INVALID `TBD-PROD-*` placeholder on purpose: base defaults are the DEV values, so an OMITTED
|
|
# op means prod silently inherits a dev address — far worse than a failed apply. `TBD-PROD-*`
|
|
# is not a valid IP/URL, so Cilium CRD validation / LB-IPAM / CNPG / Velero reject it LOUDLY.
|
|
#
|
|
# ------------------------------------------------------------------------------------------
|
|
# ADDRESS PLAN (operator, 2026-08-31) — see infra/PROD-PROVISIONING.md §4 for the full table.
|
|
# ------------------------------------------------------------------------------------------
|
|
# nodes 192.168.2.10 / .11 / .12 (Ansible inventory)
|
|
# control-plane VIP 192.168.2.9 (Ansible group_vars/k3s_prod — NOT here)
|
|
# ingress Gateway VIP 192.168.2.20 (GATEWAY_VIP, below)
|
|
# Cilium LB pool 192.168.2.20 - .30 (CILIUM_LB_POOL_*, below)
|
|
#
|
|
# CONTAINMENT INVARIANT SATISFIED: GATEWAY_VIP (.20) IS CILIUM_LB_POOL_START, mirroring dev
|
|
# (.53 of .53-.60). This matters because the `shared` Gateway (platform/gateway/gateway.yaml)
|
|
# pins NOTHING — no `lbipam.cilium.io/ips`, no spec.addresses (unlike platform/crafty and
|
|
# apps/vms/unifi, which DO pin) — so Cilium LB-IPAM assigns it the FIRST FREE address of a
|
|
# matching pool block. Had GATEWAY_VIP stayed outside every pool (the original .21-.30
|
|
# proposal), the Gateway would have come up on .21 while split-horizon coredns-custom kept
|
|
# answering *.home.keithjackson.dev with .20 — an address nothing announces — blackholing
|
|
# in-cluster hairpin traffic (OIDC clients -> Keycloak/Grafana) while looking healthy.
|
|
# KEEP GATEWAY_VIP == CILIUM_LB_POOL_START when changing either.
|
|
# The control-plane VIP (.9) is deliberately OUTSIDE this pool: kube-vip floats it, and
|
|
# LB-IPAM must never be able to hand it to a Service.
|
|
- op: replace
|
|
path: /data/GATEWAY_VIP
|
|
value: "192.168.2.20"
|
|
# Cilium LB IP pool, operator-supplied: 192.168.2.20-.30 (START == GATEWAY_VIP, see above).
|
|
# Mirror this range into the prod tofu reservation (cilium_l2_vip_range) if prod ever gets one.
|
|
- op: replace
|
|
path: /data/CILIUM_LB_POOL_START
|
|
value: "192.168.2.20"
|
|
- op: replace
|
|
path: /data/CILIUM_LB_POOL_STOP
|
|
value: "192.168.2.30"
|
|
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
|
|
# k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
|
|
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10. It is owned
|
|
# Ansible-side (prod inventory k3s_api_host) and MUST equal that value — [parity] contract in
|
|
# platform/cilium/helmrelease.yaml. Not yet decided.
|
|
- op: replace
|
|
path: /data/K3S_API_HOST
|
|
value: "192.168.2.10"
|
|
# Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
|
|
# prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
|
|
# they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
|
|
# collide with the Gateway VIP or the node IPs (.10/.11/.12).
|
|
- op: replace
|
|
path: /data/UNIFI_L4_VIP
|
|
value: "192.168.2.21"
|
|
- op: replace
|
|
path: /data/CRAFTY_L4_VIP
|
|
value: "192.168.2.22"
|
|
# Off-cluster DR target. Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the
|
|
# network — a prod backup written there is not off-cluster DR and would mix two environments'
|
|
# PITR streams in one bucket. Operator supplies the prod endpoint (full URL incl. scheme+port).
|
|
- op: replace
|
|
path: /data/S3_BACKUP_ENDPOINT
|
|
value: "TBD-PROD-S3-BACKUP-ENDPOINT"
|
|
# TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.
|