jakman85andClaude Opus 5
6f61c9622d
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."
THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.
SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join
target and a tls-san. infra/tofu/outputs.tf GENERATES this
name, so renaming would ripple into tofu and the runbooks.
k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium
consumes. Derived, not restated, so an environment overrides
ONE line and the VIP, the SAN and the CNI endpoint move
together. Prod needed no new override.
THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.
Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".
VERIFIED BY ME, not just reported:
renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP)
cilium template consumes k3s_apiserver_endpoint, not k3s_api_host
cold join still `--server https://{{ k3s_api_host }}:6443` — the founder
order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
gate full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.
ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.
APPLY ORDER, and this is the operator's step — nothing was applied:
PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
incident), it will not proceed past an unready pod, existing datapath survives an
agent restart, and .9 is already live before the change lands.
DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.
NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.
Refs R-PLAT-01, R-NET-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
439 lines
26 KiB
YAML
439 lines
26 KiB
YAML
---
|
|
# clusters/prod — DESIGN-ONLY overlay. NOT a build target until roadmap Phase 7 opens prod
|
|
# (spec roadmap/phases.md, AGENTS.md §6: "Never build directly on production").
|
|
#
|
|
# When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
|
|
# here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
|
|
# values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
|
|
#
|
|
# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
|
|
# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
|
|
# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
|
|
# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
|
|
# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
|
|
# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
|
|
# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
|
|
# done here: prod is design-only and its root object is the prod bring-up owner's call.)
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- ../base
|
|
- sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
|
|
patches:
|
|
# --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) ---
|
|
# Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets.
|
|
# The `sops-age` Secret is created OUT OF BAND on the prod cluster
|
|
# (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined.
|
|
# Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave.
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: platform
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-0
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-1
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-2
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: oauth2-proxy
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: headlamp
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: crafty
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: velero
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
|
|
#
|
|
# ############################################################################################
|
|
# ### ###
|
|
# ### *** T E M P O R A R Y B O O T S T R A P S O U R C E — NOT PERMANENT *** ###
|
|
# ### ###
|
|
# ### The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside ###
|
|
# ### any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST ###
|
|
# ### be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this ###
|
|
# ### and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is ###
|
|
# ### a live single point of failure, not a settled design. See "CUTOVER" below. ###
|
|
# ### ###
|
|
# ############################################################################################
|
|
#
|
|
# WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
|
|
# unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
|
|
# REASONING UPDATED 2026-09-04 for the new suffix direction (prod = the apex, dev = dev.home.),
|
|
# which changes WHY the blocker holds but not THAT it holds:
|
|
# 1. SELF-HOSTING CIRCULARITY — still true, and now the only reason. The Gitea this repo is
|
|
# pushed to runs ON THE DEV CLUSTER. Prod syncing from it would put prod's GitOps source
|
|
# inside the very cluster slated for decommission. Gitea must move to prod (or off-cluster)
|
|
# BEFORE dev is retired, or prod loses its source of truth.
|
|
# NOTE the suffix change makes the two Giteas DIFFERENT HOSTS: dev's is now
|
|
# gitea.dev.home.keithjackson.dev and prod's is gitea.home.keithjackson.dev. Prod pointing at
|
|
# the apex name therefore means "prod's OWN Gitea", never dev's — which is what makes the
|
|
# cutover below a one-line change once prod actually runs Gitea.
|
|
# 2. SPLIT-HORIZON DNS — NO LONGER A BLOCKER, and the note is kept because the old reasoning is
|
|
# easy to re-derive incorrectly. platform/coredns/coredns-custom.yaml answers the whole
|
|
# ${BASE_DOMAIN} zone with ${GATEWAY_VIP}; when prod's BASE_DOMAIN was `prod.home.` that meant
|
|
# prod resolved gitea.home.keithjackson.dev to PROD's gateway (192.168.2.20) where no Gitea
|
|
# listened — a misdirection. Now that prod IS the apex, resolving gitea.home.keithjackson.dev
|
|
# to prod's own Gateway is CORRECT: that is where prod's Gitea will be served from. What
|
|
# remains is purely reason 1 — the Gitea is not there YET.
|
|
#
|
|
# WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
|
|
# bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
|
|
# operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
|
|
# reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
|
|
# change was needed. This is the same "local bootstrap source" pattern dev itself used before
|
|
# its Phase-6 cutover (docs/05 "Source-of-truth").
|
|
#
|
|
# PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
|
|
# for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
|
|
# the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
|
|
# .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
|
|
#
|
|
# NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
|
|
# works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
|
|
# curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
|
|
# git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git -> refs/heads/main
|
|
# If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
|
|
# THAT FIRST: a private repo here fails exactly that way.
|
|
#
|
|
# *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
|
|
# WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
|
|
# through prod's own Gateway VIP with a publicly-trusted cert.
|
|
# DO: 1. push/mirror this repo into the prod in-cluster Gitea;
|
|
# 2. change the url below to
|
|
# https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
|
# and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
|
|
# clusters/dev/kustomization.yaml, is what "done" looks like);
|
|
# 3. confirm `flux get sources git -n flux-system` reconciles the new revision;
|
|
# 4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
|
|
# exactly as dev retained its bootstrap Gitea.
|
|
# UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
|
|
# powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
|
|
# it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
|
|
#
|
|
# The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
|
|
# CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
|
|
# CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
|
|
# NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
|
|
# Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
|
|
# is the revision you intend prod to run (`git ls-remote` above, against local `main`).
|
|
- patch: |-
|
|
apiVersion: source.toolkit.fluxcd.io/v1
|
|
kind: GitRepository
|
|
metadata:
|
|
name: flux-system
|
|
namespace: flux-system
|
|
spec:
|
|
# TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block
|
|
# above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.
|
|
url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git
|
|
ref:
|
|
branch: main
|
|
- target:
|
|
kind: ConfigMap
|
|
name: cluster-vars
|
|
patch: |-
|
|
- op: replace
|
|
path: /data/CLUSTER_ENV
|
|
value: "prod"
|
|
# trusted, rate-limited real certs in prod — ADR-0006.
|
|
- op: replace
|
|
path: /data/ACME_CLUSTER_ISSUER
|
|
value: "letsencrypt-prod"
|
|
# ==========================================================================================
|
|
# PROD DNS IDENTITY (R-DNS-01/02/03) — PROD **IS** THE APEX.
|
|
# ==========================================================================================
|
|
# OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day
|
|
# (prod on `prod.home.` behind apex CNAMEs). The simpler target, and the one authored here:
|
|
#
|
|
# dev -> dev.home.keithjackson.dev (dev carries the label; clusters/dev)
|
|
# prod -> home.keithjackson.dev (THE APEX — this knob)
|
|
#
|
|
# There is NO `prod.` label and NO CNAME layer. `<svc>.home.keithjackson.dev` is prod's REAL
|
|
# name, so nothing has to alias anything: prod's Gitea is gitea.home.keithjackson.dev, its
|
|
# Keycloak keycloak.home.keithjackson.dev, and the wildcard cert is `*.home.keithjackson.dev`.
|
|
#
|
|
# BACKGROUND — WHY THE SUFFIXES HAD TO SPLIT AT ALL. Until 2026-09-04 every hostname was
|
|
# HARDCODED to home.keithjackson.dev in the base manifests, so prod was authored to serve the
|
|
# EXACT SAME public names as the live dev cluster out of the SAME Cloudflare zone. Two
|
|
# consequences, one of which already happened:
|
|
# 1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
|
|
# default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
|
|
# 2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
|
|
# wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
|
|
# have raced each other's `_acme-challenge` TXT records on that one zone.
|
|
#
|
|
# *** THE ORDERING CONSTRAINT THIS VALUE CREATES — READ BEFORE UNSUSPENDING ANYTHING. ***
|
|
# Prod now claims the very names the LIVE dev cluster is serving today. Both hazards above are
|
|
# therefore still armed until DEV HAS VACATED THE APEX. Prod's `platform` and its 10 downstream
|
|
# tiers are SUSPENDED for exactly this reason. The gate is not "review is done", it is:
|
|
# (a) dev is reconciling on *.dev.home.keithjackson.dev with a Ready wildcard cert, AND
|
|
# (b) dev's old apex A/CNAME records and their TXT registry entries are GONE from Cloudflare
|
|
# (verified against Cloudflare directly — the workstation's dnsmasq is NOT evidence about public
|
|
# DNS; that mistake was made on 2026-09-04), AND
|
|
# (c) dev's old `*.home.keithjackson.dev` cert/Certificate is released, because prod ordering
|
|
# that same name set while dev's still exists counts against LE's duplicate-certificate
|
|
# limit (5 per week per identical name set).
|
|
# Full ordered runbook: infra/PROD-PROVISIONING.md §10. Unsuspending before (a)-(c) re-creates
|
|
# the 2026-09-04 incident with the roles reversed.
|
|
#
|
|
# [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
|
|
# The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
|
|
# set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
|
|
# trusts a different Keycloak than the one prod actually runs.
|
|
- op: replace
|
|
path: /data/BASE_DOMAIN
|
|
value: "home.keithjackson.dev"
|
|
- op: replace
|
|
path: /data/EXTERNALDNS_DOMAIN_FILTER
|
|
# MUST be the Cloudflare ZONE. Prod now serves the apex, so `home.keithjackson.dev` looks
|
|
# like a tighter, more honest filter than the zone — IT IS NOT, IT IS AN OUTAGE. FINDING
|
|
# F-DNS1 (DECISIONS-LOG 2026-08-17) proved a STRICT SUBDOMAIN of the hosted zone drops the
|
|
# zone entirely: ExternalDNS generates the record then skips it with "no hosted zone matching
|
|
# record DNS Name", so prod publishes NOTHING. An agent narrowed this to
|
|
# prod.home.keithjackson.dev on 2026-09-04 and it was reverted in review; do not retry it.
|
|
# Isolation from dev does NOT come from this filter — it comes from
|
|
# (1) the two clusters requesting DISJOINT hostname sets (apex vs dev.home.), and
|
|
# (2) txtOwnerId: prod vs dev, so neither registry ever claims the other's records.
|
|
# That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
|
|
value: "keithjackson.dev"
|
|
# SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
|
|
# both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
|
|
# its own and, under policy: sync, deleted the records behind them.
|
|
#
|
|
# PROD MUST NOT BE GIVEN `default`, however tempting it now looks. Prod is taking over the apex
|
|
# NAMES, and dev's existing apex records are registered under owner `default` — so `default` is
|
|
# the one value that would let prod adopt them. It would also let prod SYNC-DELETE the ones no
|
|
# prod source claims, on a cluster dev is still serving. Prod inherits the apex names; it must
|
|
# never inherit the apex OWNER ID. `prod` keeps prod fail-safe: records owned by `default` are
|
|
# invisible to it, so an early unsuspend cannot destroy dev's DNS the way 2026-09-04 did — it
|
|
# simply fails to publish until those records are retired, which is the correct behaviour.
|
|
# dev is `dev` (clusters/dev, changed with dev's migration); the base default is a
|
|
# non-colliding sentinel. All three differ.
|
|
- op: replace
|
|
path: /data/EXTERNALDNS_TXT_OWNER_ID
|
|
value: "prod"
|
|
- op: replace
|
|
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
|
value: "3"
|
|
# Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
|
|
# the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
|
|
# and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
|
|
# (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
|
|
# measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
|
|
# its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
|
|
#
|
|
# [parity] MUST equal node_baseline_longhorn_data_path in
|
|
# infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
|
|
# directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
|
|
# path on the OS filesystem instead of the data partition and /var fills up.
|
|
- op: replace
|
|
path: /data/LONGHORN_DATA_PATH
|
|
value: "/data/longhorn"
|
|
# longer retention / larger volumes on prod NVMe.
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_RETENTION
|
|
value: "30d"
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_PVC_SIZE
|
|
value: "50Gi"
|
|
# Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
|
|
# size cap before 30d if series volume is high, keeping the disk from filling either way.
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
|
|
value: "45GiB"
|
|
- op: replace
|
|
path: /data/MONITORING_LOKI_PVC_SIZE
|
|
value: "50Gi"
|
|
# HA control-plane addons in prod.
|
|
- op: replace
|
|
path: /data/KEYCLOAK_REPLICAS
|
|
value: "2"
|
|
- op: replace
|
|
path: /data/OAUTH2_PROXY_REPLICAS
|
|
value: "2"
|
|
- op: replace
|
|
path: /data/VELERO_RETENTION
|
|
value: "720h"
|
|
# ==========================================================================================
|
|
# PROD LAN ADDRESSING (192.168.2.0/24). Nodes: k3s-prod-1/2/3 = .10 / .11 / .12.
|
|
# ==========================================================================================
|
|
# These knobs were HARDCODED to the dev LAN inside base manifests until 2026-08-31; prod would
|
|
# have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN. They are knobs now, and
|
|
# this overlay supplies every one of them. Anything the operator has NOT yet decided stays an
|
|
# INVALID `TBD-PROD-*` placeholder on purpose: base defaults are the DEV values, so an OMITTED
|
|
# op means prod silently inherits a dev address — far worse than a failed apply. `TBD-PROD-*`
|
|
# is not a valid IP/URL, so Cilium CRD validation / LB-IPAM / CNPG / Velero reject it LOUDLY.
|
|
#
|
|
# ------------------------------------------------------------------------------------------
|
|
# ADDRESS PLAN (operator, 2026-08-31) — see infra/PROD-PROVISIONING.md §4 for the full table.
|
|
# ------------------------------------------------------------------------------------------
|
|
# nodes 192.168.2.10 / .11 / .12 (Ansible inventory)
|
|
# control-plane VIP 192.168.2.9 (Ansible group_vars/k3s_prod — NOT here)
|
|
# ingress Gateway VIP 192.168.2.20 (GATEWAY_VIP, below)
|
|
# Cilium LB pool 192.168.2.20 - .30 (CILIUM_LB_POOL_*, below)
|
|
#
|
|
# CONTAINMENT INVARIANT SATISFIED: GATEWAY_VIP (.20) IS CILIUM_LB_POOL_START, mirroring dev
|
|
# (.53 of .53-.60). This matters because the `shared` Gateway (platform/gateway/gateway.yaml)
|
|
# pins NOTHING — no `lbipam.cilium.io/ips`, no spec.addresses (unlike platform/crafty and
|
|
# apps/vms/unifi, which DO pin) — so Cilium LB-IPAM assigns it the FIRST FREE address of a
|
|
# matching pool block. Had GATEWAY_VIP stayed outside every pool (the original .21-.30
|
|
# proposal), the Gateway would have come up on .21 while split-horizon coredns-custom kept
|
|
# answering *.home.keithjackson.dev with .20 — an address nothing announces — blackholing
|
|
# in-cluster hairpin traffic (OIDC clients -> Keycloak/Grafana) while looking healthy.
|
|
# KEEP GATEWAY_VIP == CILIUM_LB_POOL_START when changing either.
|
|
# The control-plane VIP (.9) is deliberately OUTSIDE this pool: kube-vip floats it, and
|
|
# LB-IPAM must never be able to hand it to a Service.
|
|
- op: replace
|
|
path: /data/GATEWAY_VIP
|
|
value: "192.168.2.20"
|
|
# Cilium LB IP pool, operator-supplied: 192.168.2.20-.30 (START == GATEWAY_VIP, see above).
|
|
# Mirror this range into the prod tofu reservation (cilium_l2_vip_range) if prod ever gets one.
|
|
- op: replace
|
|
path: /data/CILIUM_LB_POOL_START
|
|
value: "192.168.2.20"
|
|
- op: replace
|
|
path: /data/CILIUM_LB_POOL_STOP
|
|
value: "192.168.2.30"
|
|
# k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
|
|
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10.
|
|
#
|
|
# *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously
|
|
# read "Not yet decided" while shipping .10 — the founder's own address — and that is what
|
|
# the LIVE prod cluster is running: all three cilium-agents carry
|
|
# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a
|
|
# 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and
|
|
# the datapath keeps running off Cilium's cache while no service or policy update lands, with
|
|
# nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the
|
|
# control plane is, which is the whole reason kube-vip exists here.)
|
|
#
|
|
# It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from
|
|
# this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST
|
|
# equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host:
|
|
# that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap
|
|
# `--server` join needs a host that is up before any VIP is floated. Two addresses, two
|
|
# variables, deliberately.
|
|
#
|
|
# .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and
|
|
# LB-IPAM can never be handed the same IP — asserted by
|
|
# conformance/tests/l0-static/ha-control-plane-as-code.
|
|
- op: replace
|
|
path: /data/K3S_API_HOST
|
|
value: "192.168.2.9"
|
|
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
|
|
# Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
|
|
# prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
|
|
# they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
|
|
# collide with the Gateway VIP or the node IPs (.10/.11/.12).
|
|
- op: replace
|
|
path: /data/UNIFI_L4_VIP
|
|
value: "192.168.2.21"
|
|
- op: replace
|
|
path: /data/CRAFTY_L4_VIP
|
|
value: "192.168.2.22"
|
|
# ------------------------------------------------------------------------------------------
|
|
# OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05).
|
|
#
|
|
# THERE IS DELIBERATELY NO `S3_BACKUP_ENDPOINT` OP HERE, and that is a reversal of the rule
|
|
# stated at the top of this overlay ("an OMITTED op means prod silently inherits a dev
|
|
# address"). That rule exists because base defaults are dev's LAN addresses. R2 is not one: it
|
|
# is a single shared, environment-independent endpoint, so under R-ENV-02 it belongs in base and
|
|
# overriding it with an identical value would only create two places to change on a rotation —
|
|
# the exact drift this repo has been bitten by before. Isolation does NOT come from the
|
|
# endpoint; it comes from the two path knobs below.
|
|
#
|
|
# WHY R2 AND NOT THE MinIO the operator first named: prod is on the SOHO network
|
|
# 192.168.2.0/24, which is pfSense's WAN, while the MinIO sits inside pfSense's LAN
|
|
# 192.168.1.0/24. Measured from a pod INSIDE prod (the workstation proves nothing about pod
|
|
# egress): `curl http://192.168.1.9:9000/minio/health/live` → connection timed out after 15s.
|
|
# Reaching it would need a WAN→LAN rule that does not exist. R2 needs no firewall change, and
|
|
# unlike that MinIO — which runs on the same Proxmox host as dev's VMs — it is genuinely
|
|
# off-site. Verify prod can reach the new endpoint before resuming velero:
|
|
# bootstrap/standup.sh ... --s3-endpoint <the R2 endpoint> s3check
|
|
#
|
|
# THE PATH KNOBS ARE WHAT MAKE ONE SHARED BUCKET SAFE. Nothing about the paths was
|
|
# per-environment before 2026-09-05: Velero used one unprefixed bucket, and every CNPG Cluster
|
|
# used `s3://cnpg` with serverName defaulting to the CLUSTER NAME — and both environments have
|
|
# keycloak-db / nextcloud-db / semaphore-db. Without these knobs prod would (a) adopt and
|
|
# TTL-delete dev's Velero backups and (b) interleave two Postgres servers' WAL into ONE Barman
|
|
# stream, corrupting the chain and destroying PITR for both. See clusters/base/cluster-vars.yaml.
|
|
# ------------------------------------------------------------------------------------------
|
|
- op: replace
|
|
path: /data/VELERO_BACKUP_PREFIX
|
|
value: "prod"
|
|
# Barman objects land at s3://cnpg/prod/<clusterName>/; dev's at s3://cnpg/dev/<clusterName>/.
|
|
- op: replace
|
|
path: /data/CNPG_BACKUP_PATH
|
|
value: "s3://cnpg/prod"
|
|
# TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.
|