Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
ae37e27f140a02ebc2ecff77779d9359345384a1
home-network-k3s-impl/clusters/prod/kustomization.yaml
T
jakman85andClaude Opus 5 ae37e27f14
gate / gate (push) Has been cancelled
Details
gate / conformance (push) Has been cancelled
Details
fix(gitops): move prod's own Git source with the apex flip 
The GitRepository url is a hardcoded literal, not templated on ${BASE_DOMAIN}, so
the apex flip did not carry it. But the flip renames prod's HTTPRoute hostnames, so
external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev and CoreDNS's
split-horizon zone moves from prod.home. to home. — which would have severed prod
from its OWN GitOps source moments after the flip reconciled.

Landed in the same push as the flip deliberately: prod has not yet fetched 9161cab
(still on b9c0970, all hostnames still prod.home.), so both commits arrive in one
fetch and the url changes in the same apply that renames the routes. A live patch
would not survive — the root Kustomization renders this very file and reverts it.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 22:10:57 -05:00

569 lines
36 KiB
YAML
Raw Blame History

---
# clusters/prod — DESIGN-ONLY overlay. NOT a build target until roadmap Phase 7 opens prod
# (spec roadmap/phases.md, AGENTS.md §6: "Never build directly on production").
#
# When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
# here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
# values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
#
# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
# done here: prod is design-only and its root object is the prod bring-up owner's call.)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
- sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
patches:
# --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) ---
# Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets.
# The `sops-age` Secret is created OUT OF BAND on the prod cluster
# (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined.
# Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave.
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: platform
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-wave-0
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-wave-1
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-wave-2
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: oauth2-proxy
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: headlamp
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: crafty
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: velero
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
#
# ############################################################################################
# ### ###
# ### *** T E M P O R A R Y B O O T S T R A P S O U R C E — NOT PERMANENT *** ###
# ### ###
# ### The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside ###
# ### any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST ###
# ### be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this ###
# ### and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is ###
# ### a live single point of failure, not a settled design. See "CUTOVER" below. ###
# ### ###
# ############################################################################################
#
# WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
# unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
# Two independent reasons, both consequences of the split-horizon design:
# 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
# GitOps source lives inside the very cluster slated for decommission. Gitea must move to
# prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
# 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
# answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
# 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
# gitea.home.keithjackson.dev to itself and never reach a repository.
#
# WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
# bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
# operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
# reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
# change was needed. This is the same "local bootstrap source" pattern dev itself used before
# its Phase-6 cutover (docs/05 "Source-of-truth").
#
# PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
# for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
# the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
# .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
#
# NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
# works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
# curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
# git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git -> refs/heads/main
# If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
# THAT FIRST: a private repo here fails exactly that way.
#
# *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
# WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
# through prod's own Gateway VIP with a publicly-trusted cert.
# DO: 1. push/mirror this repo into the prod in-cluster Gitea;
# 2. change the url below to
# https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
# and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
# clusters/dev/kustomization.yaml, is what "done" looks like);
# 3. confirm `flux get sources git -n flux-system` reconciles the new revision;
# 4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
# exactly as dev retained its bootstrap Gitea.
# UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
# powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
# it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
#
# The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
# CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
# CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
# NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
# Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
# is the revision you intend prod to run (`git ls-remote` above, against local `main`).
- patch: |-
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: flux-system
namespace: flux-system
spec:
# CUTOVER DONE 2026-09-06 (operator direction: "move the repo to prod"). Prod now reconciles
# from PROD'S OWN in-cluster Gitea. The workstation bootstrap container
# (http://192.168.2.132:3000) is retired as prod's source — it was a podman container on the
# operator's workstation holding a DHCP lease on WIFI, i.e. production's source of truth was a
# laptop. It is kept only as a recovery path (bootstrap/gitea/bootstrap-gitea.sh).
#
# THIS LINE IS SELF-REFERENTIAL AND THAT IS THE WHOLE DIFFICULTY. The root Kustomization
# renders ./clusters/prod, which contains this patch — so a live `kubectl patch` of
# GitRepository.spec.url is REVERTED on the next reconcile unless the authored value moves
# too. The cutover therefore had to be: (1) push the repo to prod's Gitea, (2) fast-forward
# its `main` to the exact commit prod was already running so the source swap changed nothing
# else, (3) live-patch the url to open a window, (4) land THIS commit through that window so
# the authored value agrees. Dev hit the identical trap on 2026-09-05
# (infra/PROD-PROVISIONING.md §10a "D0/D1 as executed").
#
# Verified before the switch: a throwaway pod INSIDE prod resolved this host through
# in-cluster CoreDNS to the Gateway VIP and cloned over the public LE cert with no CA
# supplied. The workstation being able to reach it proves nothing about pod egress.
# MOVED WITH THE APEX FLIP (2026-09-13) — and it had to move in the SAME fetch.
# This URL is a hardcoded literal, NOT templated on ${BASE_DOMAIN}, so the flip did not
# carry it. But the flip DOES rename prod's own HTTPRoute hostnames, so:
# - external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev, and
# - platform/coredns/coredns-custom.yaml answers the whole ${BASE_DOMAIN} zone with
# ${GATEWAY_VIP}, so in-cluster resolution moves from prod.home. to home.
# Leaving this on the old name would therefore have severed prod from its OWN GitOps
# source moments after the flip reconciled — self-reference, the same class of trap as
# patching this url live and having the root Kustomization revert it. The fix has to land
# THROUGH THE OLD SOURCE, which is why it is in the same push as the flip rather than
# after it.
# FOLLOW-UP worth doing: point this at the in-cluster Service
# (gitea-http.gitea.svc.cluster.local) so prod's GitOps stops depending on public DNS and
# on its own Gateway at all. Deferred: it needs the TLS/credential path rethought.
url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
ref:
branch: main
- target:
kind: ConfigMap
name: cluster-vars
patch: |-
- op: replace
path: /data/CLUSTER_ENV
value: "prod"
# trusted, rate-limited real certs in prod — ADR-0006.
- op: replace
path: /data/ACME_CLUSTER_ISSUER
value: "letsencrypt-prod"
# ==========================================================================================
# PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
# ==========================================================================================
# PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
# home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
# public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
# of which already happened:
# 1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
# default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
# 2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
# wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
# have raced each other's `_acme-challenge` TXT records on that one zone.
# Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
# record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
#
# dev -> home.keithjackson.dev (UNCHANGED — dev is live; phase 2 moves it)
# prod -> prod.home.keithjackson.dev (this knob)
#
# CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
# gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
# on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
# `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
# would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
#
# [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
# The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
# set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
# trusts a different Keycloak than the one prod actually runs.
- op: replace
path: /data/BASE_DOMAIN
# APEX AS OF 2026-09-13. Prod now serves home.keithjackson.dev directly; dev has been
# decommissioned and released the name. Prerequisites completed and VERIFIED before this
# flip, in this order:
# 1. dev's 18 A records released by its own external-dns (policy=sync, sources deleted)
# 2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by hand —
# external-dns v0.21.0 removed only the `a-` prefixed form and then reported "All
# records are already up to date", so it would NEVER have cleared them. They carried
# `owner=default`, and prod runs `owner=prod`, so prod could not have claimed these
# names while they existed. Confirmed gone at the AUTHORITY (betty.ns.cloudflare.com);
# 1.1.1.1 served stale cache for minutes afterwards and must not be trusted here.
# 3. dev's `*.home.keithjackson.dev` + apex Certificate deleted, so it stops renewing.
# It was issued ~18d ago, so prod's request is #1 this week against LE's duplicate
# limit of 5 per identical name set per week.
# NOTE the zone has wildcards `*.keithjackson.dev` AND `*.home.keithjackson.dev` pointing
# at the public IP 136.51.115.252 (legacy non-k3s prod). Dev's specific records were
# SHADOWING that wildcard; prod's new specific records shadow it again. The legacy stack
# lives on `*.keithjackson.dev` and is untouched by any of this.
value: "home.keithjackson.dev"
# SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
# OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
# TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
# ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
# the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
# matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
# records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
# EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
- op: replace
path: /data/EXTERNALDNS_DOMAIN_FILTER
# MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
# 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
# record then skipped it with "no hosted zone matching record DNS Name", so prod would
# publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
# (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
# (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
# That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
value: "keithjackson.dev"
# SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
# both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
# its own and, under policy: sync, deleted the records behind them. This value MUST differ from
# dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
# cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
- op: replace
path: /data/EXTERNALDNS_TXT_OWNER_ID
value: "prod"
- op: replace
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
value: "3"
# Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
# the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
# and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
# (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
# measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
# its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
#
# [parity] MUST equal node_baseline_longhorn_data_path in
# infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
# directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
# path on the OS filesystem instead of the data partition and /var fills up.
- op: replace
path: /data/LONGHORN_DATA_PATH
value: "/data/longhorn"
# longer retention / larger volumes on prod NVMe.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION
value: "30d"
- op: replace
path: /data/MONITORING_PROMETHEUS_PVC_SIZE
value: "50Gi"
# Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
# size cap before 30d if series volume is high, keeping the disk from filling either way.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
value: "45GiB"
- op: replace
path: /data/MONITORING_LOKI_PVC_SIZE
value: "50Gi"
# HA control-plane addons in prod.
- op: replace
path: /data/KEYCLOAK_REPLICAS
value: "2"
- op: replace
path: /data/OAUTH2_PROXY_REPLICAS
value: "2"
- op: replace
path: /data/VELERO_RETENTION
value: "720h"
# ==========================================================================================
# PROD LAN ADDRESSING (192.168.2.0/24). Nodes: k3s-prod-1/2/3 = .10 / .11 / .12.
# ==========================================================================================
# These knobs were HARDCODED to the dev LAN inside base manifests until 2026-08-31; prod would
# have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN. They are knobs now, and
# this overlay supplies every one of them. Anything the operator has NOT yet decided stays an
# INVALID `TBD-PROD-*` placeholder on purpose: base defaults are the DEV values, so an OMITTED
# op means prod silently inherits a dev address — far worse than a failed apply. `TBD-PROD-*`
# is not a valid IP/URL, so Cilium CRD validation / LB-IPAM / CNPG / Velero reject it LOUDLY.
#
# ------------------------------------------------------------------------------------------
# ADDRESS PLAN (operator, 2026-08-31) — see infra/PROD-PROVISIONING.md §4 for the full table.
# ------------------------------------------------------------------------------------------
# nodes 192.168.2.10 / .11 / .12 (Ansible inventory)
# control-plane VIP 192.168.2.9 (Ansible group_vars/k3s_prod — NOT here)
# ingress Gateway VIP 192.168.2.20 (GATEWAY_VIP, below)
# Cilium LB pool 192.168.2.20 - .30 (CILIUM_LB_POOL_*, below)
#
# CONTAINMENT INVARIANT SATISFIED: GATEWAY_VIP (.20) IS CILIUM_LB_POOL_START, mirroring dev
# (.53 of .53-.60). This matters because the `shared` Gateway (platform/gateway/gateway.yaml)
# pins NOTHING — no `lbipam.cilium.io/ips`, no spec.addresses (unlike platform/crafty and
# apps/vms/unifi, which DO pin) — so Cilium LB-IPAM assigns it the FIRST FREE address of a
# matching pool block. Had GATEWAY_VIP stayed outside every pool (the original .21-.30
# proposal), the Gateway would have come up on .21 while split-horizon coredns-custom kept
# answering *.home.keithjackson.dev with .20 — an address nothing announces — blackholing
# in-cluster hairpin traffic (OIDC clients -> Keycloak/Grafana) while looking healthy.
# KEEP GATEWAY_VIP == CILIUM_LB_POOL_START when changing either.
# The control-plane VIP (.9) is deliberately OUTSIDE this pool: kube-vip floats it, and
# LB-IPAM must never be able to hand it to a Service.
- op: replace
path: /data/GATEWAY_VIP
value: "192.168.2.20"
# Cilium LB IP pool, operator-supplied: 192.168.2.20-.30 (START == GATEWAY_VIP, see above).
# Mirror this range into the prod tofu reservation (cilium_l2_vip_range) if prod ever gets one.
- op: replace
path: /data/CILIUM_LB_POOL_START
value: "192.168.2.20"
- op: replace
path: /data/CILIUM_LB_POOL_STOP
value: "192.168.2.30"
# k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10.
#
# *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously
# read "Not yet decided" while shipping .10 — the founder's own address — and that is what
# the LIVE prod cluster is running: all three cilium-agents carry
# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a
# 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and
# the datapath keeps running off Cilium's cache while no service or policy update lands, with
# nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the
# control plane is, which is the whole reason kube-vip exists here.)
#
# It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from
# this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST
# equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host:
# that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap
# `--server` join needs a host that is up before any VIP is floated. Two addresses, two
# variables, deliberately.
#
# .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and
# LB-IPAM can never be handed the same IP — asserted by
# conformance/tests/l0-static/ha-control-plane-as-code.
- op: replace
path: /data/K3S_API_HOST
value: "192.168.2.9"
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
# Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
# prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
# they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
# collide with the Gateway VIP or the node IPs (.10/.11/.12).
- op: replace
path: /data/UNIFI_L4_VIP
value: "192.168.2.21"
- op: replace
path: /data/CRAFTY_L4_VIP
value: "192.168.2.22"
# ------------------------------------------------------------------------------------------
# BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
# ------------------------------------------------------------------------------------------
# Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
# radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
#
# THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
# vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
# 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
# 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
# outcomes give DIFFERENT values here:
# port-forward 192.168.2.102:2049 -> 192.168.1.9:2049 => "192.168.2.102"
# a WAN->LAN route to 192.168.1.0/24 => "192.168.1.9"
# Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
# ever became reachable — mounts something unintended. So it stays a placeholder.
#
# WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
# NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
# over v3 was in part this decision's cost: v3 would have needed four ports and three of them
# on ephemeral numbers.
#
# WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
# Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
# the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
# address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
# 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
# addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
#
# *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
# A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
# placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
# failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
# TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
# ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
# apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
# a revision containing the bulk volumes; the dev overlay renders a working value, which is
# where these manifests are meant to be proven first.
- op: replace
path: /data/BULK_NFS_SERVER
# 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
# host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
# reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
# keeps the base value 192.168.1.9.
#
# PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
# - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
# - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
# carries no_root_squash, so a root write would have passed and proved nothing, while
# LSIO images default to PUID 911 and would have gone Ready then failed every import
# with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
# - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
# real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
# pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
# so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
value: "192.168.2.102"
# ==========================================================================================
# OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
# ==========================================================================================
# Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the network and is NOT an
# option for prod, for two independent reasons:
# 1. UNREACHABLE. Prod is on the SOHO network 192.168.2.0/24 — pfSense's WAN — while that
# MinIO is inside pfSense's LAN. Measured from a pod INSIDE prod (the workstation proves
# nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live` → connection
# timed out after 15s. Reaching it would need a WAN→LAN rule that does not exist.
# 2. NOT OFF-CLUSTER DR. It runs on the same Proxmox host as dev's VMs, so it already shares
# a physical failure domain with dev. R2 is genuinely off-site, and its free egress means
# restore REHEARSALS cost nothing — R-STATE-03 has no financial disincentive to stay proven.
#
# THIS UNBLOCKS AN ACTIVE INCIDENT, not just a roadmap item. All three prod CNPG Clusters ship
# `archive_mode=on` with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, so barman-cloud-wal-archive
# exits 4 on every segment and Postgres — correctly — REFUSES TO RECYCLE WAL. Verified live
# 2026-09-07: keycloak-db / nextcloud-db / semaphore-db all report
# ContinuousArchiving=False/ContinuousArchivingFailing, and keycloak-db's 5 GiB PVC is at
# 579 MiB used / 4360 MiB free and climbing monotonically. A disk-full PANIC there takes the
# cluster's SSO database down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and
# forward-auth. Fixing the endpoint lets Postgres drain the whole backlog.
#
# ENDPOINT IS ACCOUNT-SCOPED, so it stays a prod-overlay value even though it is not a LAN
# address: this is the operator's Cloudflare account id, confirmed from inside a prod pod
# (`head-bucket` on k3s-prod-backup → BucketRegion WNAM; PUT and DELETE both succeed).
# It is NOT put in base, because base is dev's live MinIO and dev is deliberately STAYING
# there — moving dev to R2 would reset its PITR window and orphan the backups its PROVEN
# restore depends on. Two environments, two targets, isolated by URL *and* by bucket.
- op: replace
path: /data/S3_BACKUP_ENDPOINT
value: "https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com"
# R2 requires the region string to be exactly `auto`. (`head-bucket` answers
# `{"BucketRegion":"WNAM"}` — that is a PHYSICAL LOCATION HINT, not a signing region; using it
# here produces AuthorizationHeaderMalformed.) Dev keeps the base `us-east-1` its live MinIO
# has always been signed with.
- op: replace
path: /data/S3_BACKUP_REGION
value: "auto"
# ------------------------------------------------------------------------------------------
# OBJECT TENANCY INSIDE THE ONE BUCKET. The operator created a SINGLE R2 bucket, so velero and
# barman cannot be separated by bucket the way they are on dev's MinIO (`velero` and `cnpg`).
# They are separated by disjoint TOP-LEVEL PREFIXES instead:
#
# s3://k3s-prod-backup/velero/… Velero: kopia repo + backup metadata
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG: WAL + base backups
#
# The bucket ROOT is left empty on purpose — a misrendered prefix then lands somewhere
# obviously wrong instead of silently colliding with a live tenant. Velero's retention TTL
# deletes objects under its own prefix only, so it can never reap a WAL segment; and barman's
# per-cluster serverName subdirectory keeps the three Postgres servers' chains apart, which
# matters because interleaving two servers' WAL into one prefix corrupts the chain outright.
# Dev cannot be reached from any of these: different endpoint AND different bucket names.
# ------------------------------------------------------------------------------------------
- op: replace
path: /data/VELERO_BACKUP_BUCKET
value: "k3s-prod-backup"
- op: replace
path: /data/VELERO_BACKUP_PREFIX
value: "velero"
- op: replace
path: /data/CNPG_BACKUP_PATH
value: "s3://k3s-prod-backup/barman"
# ------------------------------------------------------------------------------------------
# WHICH CREDENTIAL SIGNS THE REQUESTS TO THAT BUCKET (per-env credential split, 2026-09-07).
# ------------------------------------------------------------------------------------------
# The knobs above moved the ENDPOINT and the OBJECT LAYOUT per environment. The credential did
# not move with them, and it could not: the four backup-credential Secrets are SHARED BASE
# manifests, applied to both clusters by the same Flux tiers. Writing the R2 token into them
# gave prod a working key by TAKING DEV'S AWAY — dev would have signed MinIO requests with an
# R2 key id, and under archive_mode=on that is a filling pgdata PVC, not a missing backup.
#
# So each credential is now two objects, and this suffix selects prod's:
# velero-cloud-credentials-prod the R2 token in the AWS INI `cloud` shape
# <db>-backup-credentials-prod the same token as ACCESS_KEY_ID / ACCESS_SECRET_KEY
# for keycloak-db, nextcloud-db and semaphore-db. Both halves ship to both clusters (they are
# base resources); ONLY this value decides which one each consumer reads. It is a SUFFIX rather
# than four secret-name knobs deliberately — four could be updated three-quarters of the way,
# leaving one consumer on dev's MinIO key with nothing rendering wrong, and strict-mode
# substitution cannot catch a defined-but-stale variable.
#
# THE R2 TOKEN IS THE ONE PROVEN FROM INSIDE PROD (chore/r2-credentials 0a6a862): head-bucket on
# k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Rotating it now
# touches ONLY the four `-prod` files; dev's MinIO keys are untouched objects. That isolation is
# the structural reason for the split, independent of dev's eventual decommission.
- op: replace
path: /data/BACKUP_CREDENTIALS_SUFFIX
value: "-prod"
# TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.
Reference in new issue View Git Blame Copy Permalink