jakman85andClaude Opus 5
161430d915
The previous commit created the two halves. This one wires the choice, so dev renders MinIO
credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the
overlay patch is the entire difference between environments; no base manifest is forked).
cluster-vars BACKUP_CREDENTIALS_SUFFIX base "" | dev "" | prod "-prod"
velero HelmRelease credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
3x CNPG Cluster s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters
of the way, leaving one consumer reading the OTHER environment's key with nothing rendering
wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED
variable, never on a defined-but-stale one. With one suffix the four move together or not at
all. The cost is that the rendered Secret name is not greppable from the consumer, so both
kustomizations spell the two names out in full beside the resources.
THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects
that exist in every render, and that dev is running today. The alternative failure mode is
strictly worse than the shared-Secret state this replaces: velero's pod hangs in
ContainerCreating forever when `existingSecret` names a missing Secret (the volume never
mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which
under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable
default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the
value explicitly anyway, so neither inherits it silently.
Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is
untouched (it stays decryption-only, and velero stays suspended on prod).
VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and
clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform,
platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with
that environment's variables:
strict envsubst PASSES for all 8 (env x tier) combinations
dev velero existingSecret velero-cloud-credentials
dev keycloak/nextcloud/semaphore-db <db>-backup-credentials (both key refs each)
prod velero existingSecret velero-cloud-credentials-prod
prod keycloak/nextcloud/semaphore-db <db>-backup-credentials-prod (both key refs each)
both all EIGHT Secret objects present in the rendered output (dev and prod alike)
no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell
escapes in apps/wave-0, which are pre-existing and unrelated.
Read alongside the endpoints the same overlays already set, the two environments are now
isolated on credential AND target:
dev velero s3://velero (root) + cnpg s3://cnpg @ http://192.168.1.9:9000 region us-east-1
prod velero k3s-prod-backup/velero + barman prefix @ https://<acct>.r2.cloudflarestorage.com region auto
Co-Authored-By: Claude Opus 5 <[email protected]>
483 lines
29 KiB
YAML
483 lines
29 KiB
YAML
---
|
|
# clusters/prod — DESIGN-ONLY overlay. NOT a build target until roadmap Phase 7 opens prod
|
|
# (spec roadmap/phases.md, AGENTS.md §6: "Never build directly on production").
|
|
#
|
|
# When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
|
|
# here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
|
|
# values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
|
|
#
|
|
# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
|
|
# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
|
|
# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
|
|
# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
|
|
# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
|
|
# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
|
|
# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
|
|
# done here: prod is design-only and its root object is the prod bring-up owner's call.)
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- ../base
|
|
- sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
|
|
patches:
|
|
# --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) ---
|
|
# Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets.
|
|
# The `sops-age` Secret is created OUT OF BAND on the prod cluster
|
|
# (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined.
|
|
# Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave.
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: platform
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-0
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-1
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-2
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: oauth2-proxy
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: headlamp
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: crafty
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: velero
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
|
|
#
|
|
# ############################################################################################
|
|
# ### ###
|
|
# ### *** T E M P O R A R Y B O O T S T R A P S O U R C E — NOT PERMANENT *** ###
|
|
# ### ###
|
|
# ### The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside ###
|
|
# ### any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST ###
|
|
# ### be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this ###
|
|
# ### and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is ###
|
|
# ### a live single point of failure, not a settled design. See "CUTOVER" below. ###
|
|
# ### ###
|
|
# ############################################################################################
|
|
#
|
|
# WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
|
|
# unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
|
|
# Two independent reasons, both consequences of the split-horizon design:
|
|
# 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
|
|
# GitOps source lives inside the very cluster slated for decommission. Gitea must move to
|
|
# prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
|
|
# 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
|
|
# answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
|
|
# 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
|
|
# gitea.home.keithjackson.dev to itself and never reach a repository.
|
|
#
|
|
# WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
|
|
# bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
|
|
# operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
|
|
# reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
|
|
# change was needed. This is the same "local bootstrap source" pattern dev itself used before
|
|
# its Phase-6 cutover (docs/05 "Source-of-truth").
|
|
#
|
|
# PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
|
|
# for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
|
|
# the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
|
|
# .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
|
|
#
|
|
# NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
|
|
# works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
|
|
# curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
|
|
# git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git -> refs/heads/main
|
|
# If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
|
|
# THAT FIRST: a private repo here fails exactly that way.
|
|
#
|
|
# *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
|
|
# WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
|
|
# through prod's own Gateway VIP with a publicly-trusted cert.
|
|
# DO: 1. push/mirror this repo into the prod in-cluster Gitea;
|
|
# 2. change the url below to
|
|
# https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
|
# and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
|
|
# clusters/dev/kustomization.yaml, is what "done" looks like);
|
|
# 3. confirm `flux get sources git -n flux-system` reconciles the new revision;
|
|
# 4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
|
|
# exactly as dev retained its bootstrap Gitea.
|
|
# UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
|
|
# powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
|
|
# it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
|
|
#
|
|
# The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
|
|
# CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
|
|
# CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
|
|
# NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
|
|
# Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
|
|
# is the revision you intend prod to run (`git ls-remote` above, against local `main`).
|
|
- patch: |-
|
|
apiVersion: source.toolkit.fluxcd.io/v1
|
|
kind: GitRepository
|
|
metadata:
|
|
name: flux-system
|
|
namespace: flux-system
|
|
spec:
|
|
# CUTOVER DONE 2026-09-06 (operator direction: "move the repo to prod"). Prod now reconciles
|
|
# from PROD'S OWN in-cluster Gitea. The workstation bootstrap container
|
|
# (http://192.168.2.132:3000) is retired as prod's source — it was a podman container on the
|
|
# operator's workstation holding a DHCP lease on WIFI, i.e. production's source of truth was a
|
|
# laptop. It is kept only as a recovery path (bootstrap/gitea/bootstrap-gitea.sh).
|
|
#
|
|
# THIS LINE IS SELF-REFERENTIAL AND THAT IS THE WHOLE DIFFICULTY. The root Kustomization
|
|
# renders ./clusters/prod, which contains this patch — so a live `kubectl patch` of
|
|
# GitRepository.spec.url is REVERTED on the next reconcile unless the authored value moves
|
|
# too. The cutover therefore had to be: (1) push the repo to prod's Gitea, (2) fast-forward
|
|
# its `main` to the exact commit prod was already running so the source swap changed nothing
|
|
# else, (3) live-patch the url to open a window, (4) land THIS commit through that window so
|
|
# the authored value agrees. Dev hit the identical trap on 2026-09-05
|
|
# (infra/PROD-PROVISIONING.md §10a "D0/D1 as executed").
|
|
#
|
|
# Verified before the switch: a throwaway pod INSIDE prod resolved this host through
|
|
# in-cluster CoreDNS to the Gateway VIP and cloned over the public LE cert with no CA
|
|
# supplied. The workstation being able to reach it proves nothing about pod egress.
|
|
url: https://gitea.prod.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
|
ref:
|
|
branch: main
|
|
- target:
|
|
kind: ConfigMap
|
|
name: cluster-vars
|
|
patch: |-
|
|
- op: replace
|
|
path: /data/CLUSTER_ENV
|
|
value: "prod"
|
|
# trusted, rate-limited real certs in prod — ADR-0006.
|
|
- op: replace
|
|
path: /data/ACME_CLUSTER_ISSUER
|
|
value: "letsencrypt-prod"
|
|
# ==========================================================================================
|
|
# PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
|
|
# ==========================================================================================
|
|
# PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
|
|
# home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
|
|
# public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
|
|
# of which already happened:
|
|
# 1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
|
|
# default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
|
|
# 2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
|
|
# wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
|
|
# have raced each other's `_acme-challenge` TXT records on that one zone.
|
|
# Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
|
|
# record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
|
|
#
|
|
# dev -> home.keithjackson.dev (UNCHANGED — dev is live; phase 2 moves it)
|
|
# prod -> prod.home.keithjackson.dev (this knob)
|
|
#
|
|
# CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
|
|
# gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
|
|
# on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
|
|
# `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
|
|
# would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
|
|
#
|
|
# [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
|
|
# The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
|
|
# set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
|
|
# trusts a different Keycloak than the one prod actually runs.
|
|
- op: replace
|
|
path: /data/BASE_DOMAIN
|
|
value: "prod.home.keithjackson.dev"
|
|
# SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
|
|
# OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
|
|
# TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
|
|
# ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
|
|
# the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
|
|
# matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
|
|
# records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
|
|
# EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
|
|
- op: replace
|
|
path: /data/EXTERNALDNS_DOMAIN_FILTER
|
|
# MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
|
|
# 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
|
|
# record then skipped it with "no hosted zone matching record DNS Name", so prod would
|
|
# publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
|
|
# (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
|
|
# (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
|
|
# That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
|
|
value: "keithjackson.dev"
|
|
# SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
|
|
# both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
|
|
# its own and, under policy: sync, deleted the records behind them. This value MUST differ from
|
|
# dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
|
|
# cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
|
|
- op: replace
|
|
path: /data/EXTERNALDNS_TXT_OWNER_ID
|
|
value: "prod"
|
|
- op: replace
|
|
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
|
value: "3"
|
|
# Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
|
|
# the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
|
|
# and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
|
|
# (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
|
|
# measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
|
|
# its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
|
|
#
|
|
# [parity] MUST equal node_baseline_longhorn_data_path in
|
|
# infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
|
|
# directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
|
|
# path on the OS filesystem instead of the data partition and /var fills up.
|
|
- op: replace
|
|
path: /data/LONGHORN_DATA_PATH
|
|
value: "/data/longhorn"
|
|
# longer retention / larger volumes on prod NVMe.
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_RETENTION
|
|
value: "30d"
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_PVC_SIZE
|
|
value: "50Gi"
|
|
# Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
|
|
# size cap before 30d if series volume is high, keeping the disk from filling either way.
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
|
|
value: "45GiB"
|
|
- op: replace
|
|
path: /data/MONITORING_LOKI_PVC_SIZE
|
|
value: "50Gi"
|
|
# HA control-plane addons in prod.
|
|
- op: replace
|
|
path: /data/KEYCLOAK_REPLICAS
|
|
value: "2"
|
|
- op: replace
|
|
path: /data/OAUTH2_PROXY_REPLICAS
|
|
value: "2"
|
|
- op: replace
|
|
path: /data/VELERO_RETENTION
|
|
value: "720h"
|
|
# ==========================================================================================
|
|
# PROD LAN ADDRESSING (192.168.2.0/24). Nodes: k3s-prod-1/2/3 = .10 / .11 / .12.
|
|
# ==========================================================================================
|
|
# These knobs were HARDCODED to the dev LAN inside base manifests until 2026-08-31; prod would
|
|
# have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN. They are knobs now, and
|
|
# this overlay supplies every one of them. Anything the operator has NOT yet decided stays an
|
|
# INVALID `TBD-PROD-*` placeholder on purpose: base defaults are the DEV values, so an OMITTED
|
|
# op means prod silently inherits a dev address — far worse than a failed apply. `TBD-PROD-*`
|
|
# is not a valid IP/URL, so Cilium CRD validation / LB-IPAM / CNPG / Velero reject it LOUDLY.
|
|
#
|
|
# ------------------------------------------------------------------------------------------
|
|
# ADDRESS PLAN (operator, 2026-08-31) — see infra/PROD-PROVISIONING.md §4 for the full table.
|
|
# ------------------------------------------------------------------------------------------
|
|
# nodes 192.168.2.10 / .11 / .12 (Ansible inventory)
|
|
# control-plane VIP 192.168.2.9 (Ansible group_vars/k3s_prod — NOT here)
|
|
# ingress Gateway VIP 192.168.2.20 (GATEWAY_VIP, below)
|
|
# Cilium LB pool 192.168.2.20 - .30 (CILIUM_LB_POOL_*, below)
|
|
#
|
|
# CONTAINMENT INVARIANT SATISFIED: GATEWAY_VIP (.20) IS CILIUM_LB_POOL_START, mirroring dev
|
|
# (.53 of .53-.60). This matters because the `shared` Gateway (platform/gateway/gateway.yaml)
|
|
# pins NOTHING — no `lbipam.cilium.io/ips`, no spec.addresses (unlike platform/crafty and
|
|
# apps/vms/unifi, which DO pin) — so Cilium LB-IPAM assigns it the FIRST FREE address of a
|
|
# matching pool block. Had GATEWAY_VIP stayed outside every pool (the original .21-.30
|
|
# proposal), the Gateway would have come up on .21 while split-horizon coredns-custom kept
|
|
# answering *.home.keithjackson.dev with .20 — an address nothing announces — blackholing
|
|
# in-cluster hairpin traffic (OIDC clients -> Keycloak/Grafana) while looking healthy.
|
|
# KEEP GATEWAY_VIP == CILIUM_LB_POOL_START when changing either.
|
|
# The control-plane VIP (.9) is deliberately OUTSIDE this pool: kube-vip floats it, and
|
|
# LB-IPAM must never be able to hand it to a Service.
|
|
- op: replace
|
|
path: /data/GATEWAY_VIP
|
|
value: "192.168.2.20"
|
|
# Cilium LB IP pool, operator-supplied: 192.168.2.20-.30 (START == GATEWAY_VIP, see above).
|
|
# Mirror this range into the prod tofu reservation (cilium_l2_vip_range) if prod ever gets one.
|
|
- op: replace
|
|
path: /data/CILIUM_LB_POOL_START
|
|
value: "192.168.2.20"
|
|
- op: replace
|
|
path: /data/CILIUM_LB_POOL_STOP
|
|
value: "192.168.2.30"
|
|
# k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
|
|
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10.
|
|
#
|
|
# *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously
|
|
# read "Not yet decided" while shipping .10 — the founder's own address — and that is what
|
|
# the LIVE prod cluster is running: all three cilium-agents carry
|
|
# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a
|
|
# 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and
|
|
# the datapath keeps running off Cilium's cache while no service or policy update lands, with
|
|
# nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the
|
|
# control plane is, which is the whole reason kube-vip exists here.)
|
|
#
|
|
# It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from
|
|
# this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST
|
|
# equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host:
|
|
# that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap
|
|
# `--server` join needs a host that is up before any VIP is floated. Two addresses, two
|
|
# variables, deliberately.
|
|
#
|
|
# .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and
|
|
# LB-IPAM can never be handed the same IP — asserted by
|
|
# conformance/tests/l0-static/ha-control-plane-as-code.
|
|
- op: replace
|
|
path: /data/K3S_API_HOST
|
|
value: "192.168.2.9"
|
|
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
|
|
# Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
|
|
# prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
|
|
# they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
|
|
# collide with the Gateway VIP or the node IPs (.10/.11/.12).
|
|
- op: replace
|
|
path: /data/UNIFI_L4_VIP
|
|
value: "192.168.2.21"
|
|
- op: replace
|
|
path: /data/CRAFTY_L4_VIP
|
|
value: "192.168.2.22"
|
|
# ==========================================================================================
|
|
# OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
|
|
# ==========================================================================================
|
|
# Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the network and is NOT an
|
|
# option for prod, for two independent reasons:
|
|
# 1. UNREACHABLE. Prod is on the SOHO network 192.168.2.0/24 — pfSense's WAN — while that
|
|
# MinIO is inside pfSense's LAN. Measured from a pod INSIDE prod (the workstation proves
|
|
# nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live` → connection
|
|
# timed out after 15s. Reaching it would need a WAN→LAN rule that does not exist.
|
|
# 2. NOT OFF-CLUSTER DR. It runs on the same Proxmox host as dev's VMs, so it already shares
|
|
# a physical failure domain with dev. R2 is genuinely off-site, and its free egress means
|
|
# restore REHEARSALS cost nothing — R-STATE-03 has no financial disincentive to stay proven.
|
|
#
|
|
# THIS UNBLOCKS AN ACTIVE INCIDENT, not just a roadmap item. All three prod CNPG Clusters ship
|
|
# `archive_mode=on` with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, so barman-cloud-wal-archive
|
|
# exits 4 on every segment and Postgres — correctly — REFUSES TO RECYCLE WAL. Verified live
|
|
# 2026-09-07: keycloak-db / nextcloud-db / semaphore-db all report
|
|
# ContinuousArchiving=False/ContinuousArchivingFailing, and keycloak-db's 5 GiB PVC is at
|
|
# 579 MiB used / 4360 MiB free and climbing monotonically. A disk-full PANIC there takes the
|
|
# cluster's SSO database down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and
|
|
# forward-auth. Fixing the endpoint lets Postgres drain the whole backlog.
|
|
#
|
|
# ENDPOINT IS ACCOUNT-SCOPED, so it stays a prod-overlay value even though it is not a LAN
|
|
# address: this is the operator's Cloudflare account id, confirmed from inside a prod pod
|
|
# (`head-bucket` on k3s-prod-backup → BucketRegion WNAM; PUT and DELETE both succeed).
|
|
# It is NOT put in base, because base is dev's live MinIO and dev is deliberately STAYING
|
|
# there — moving dev to R2 would reset its PITR window and orphan the backups its PROVEN
|
|
# restore depends on. Two environments, two targets, isolated by URL *and* by bucket.
|
|
- op: replace
|
|
path: /data/S3_BACKUP_ENDPOINT
|
|
value: "https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com"
|
|
# R2 requires the region string to be exactly `auto`. (`head-bucket` answers
|
|
# `{"BucketRegion":"WNAM"}` — that is a PHYSICAL LOCATION HINT, not a signing region; using it
|
|
# here produces AuthorizationHeaderMalformed.) Dev keeps the base `us-east-1` its live MinIO
|
|
# has always been signed with.
|
|
- op: replace
|
|
path: /data/S3_BACKUP_REGION
|
|
value: "auto"
|
|
# ------------------------------------------------------------------------------------------
|
|
# OBJECT TENANCY INSIDE THE ONE BUCKET. The operator created a SINGLE R2 bucket, so velero and
|
|
# barman cannot be separated by bucket the way they are on dev's MinIO (`velero` and `cnpg`).
|
|
# They are separated by disjoint TOP-LEVEL PREFIXES instead:
|
|
#
|
|
# s3://k3s-prod-backup/velero/… Velero: kopia repo + backup metadata
|
|
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG: WAL + base backups
|
|
#
|
|
# The bucket ROOT is left empty on purpose — a misrendered prefix then lands somewhere
|
|
# obviously wrong instead of silently colliding with a live tenant. Velero's retention TTL
|
|
# deletes objects under its own prefix only, so it can never reap a WAL segment; and barman's
|
|
# per-cluster serverName subdirectory keeps the three Postgres servers' chains apart, which
|
|
# matters because interleaving two servers' WAL into one prefix corrupts the chain outright.
|
|
# Dev cannot be reached from any of these: different endpoint AND different bucket names.
|
|
# ------------------------------------------------------------------------------------------
|
|
- op: replace
|
|
path: /data/VELERO_BACKUP_BUCKET
|
|
value: "k3s-prod-backup"
|
|
- op: replace
|
|
path: /data/VELERO_BACKUP_PREFIX
|
|
value: "velero"
|
|
- op: replace
|
|
path: /data/CNPG_BACKUP_PATH
|
|
value: "s3://k3s-prod-backup/barman"
|
|
# ------------------------------------------------------------------------------------------
|
|
# WHICH CREDENTIAL SIGNS THE REQUESTS TO THAT BUCKET (per-env credential split, 2026-09-07).
|
|
# ------------------------------------------------------------------------------------------
|
|
# The knobs above moved the ENDPOINT and the OBJECT LAYOUT per environment. The credential did
|
|
# not move with them, and it could not: the four backup-credential Secrets are SHARED BASE
|
|
# manifests, applied to both clusters by the same Flux tiers. Writing the R2 token into them
|
|
# gave prod a working key by TAKING DEV'S AWAY — dev would have signed MinIO requests with an
|
|
# R2 key id, and under archive_mode=on that is a filling pgdata PVC, not a missing backup.
|
|
#
|
|
# So each credential is now two objects, and this suffix selects prod's:
|
|
# velero-cloud-credentials-prod the R2 token in the AWS INI `cloud` shape
|
|
# <db>-backup-credentials-prod the same token as ACCESS_KEY_ID / ACCESS_SECRET_KEY
|
|
# for keycloak-db, nextcloud-db and semaphore-db. Both halves ship to both clusters (they are
|
|
# base resources); ONLY this value decides which one each consumer reads. It is a SUFFIX rather
|
|
# than four secret-name knobs deliberately — four could be updated three-quarters of the way,
|
|
# leaving one consumer on dev's MinIO key with nothing rendering wrong, and strict-mode
|
|
# substitution cannot catch a defined-but-stale variable.
|
|
#
|
|
# THE R2 TOKEN IS THE ONE PROVEN FROM INSIDE PROD (chore/r2-credentials 0a6a862): head-bucket on
|
|
# k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Rotating it now
|
|
# touches ONLY the four `-prod` files; dev's MinIO keys are untouched objects. That isolation is
|
|
# the structural reason for the split, independent of dev's eventual decommission.
|
|
- op: replace
|
|
path: /data/BACKUP_CREDENTIALS_SUFFIX
|
|
value: "-prod"
|
|
# TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.
|