Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
8f9b2e4270c80bc93adeb9755793c1d1144f84f9
home-network-k3s-impl/clusters/dev/kustomization.yaml
T
jakman85andClaude Opus 5 32d67660a6 feat(backup): move off-cluster DR to Cloudflare R2 (operator decision 2026-09-05) 
"That's where backups will go for the time being until a better on-prem backup
solution is created." Two reasons beyond the (negligible) cost:

  1. Prod CANNOT reach the on-Proxmox MinIO. Prod is on the SOHO network
     192.168.2.0/24 — pfSense's WAN — while that MinIO sits inside pfSense's LAN.
     Measured from a pod inside prod: connection timed out after 15s. R2 needs no
     firewall change.
  2. That MinIO runs on the SAME Proxmox host as dev's VMs, so dev's
     "off-cluster" backup shared a physical failure domain with dev — one host
     loss took the cluster AND its backups. R2 is genuinely off-site.

COST, measured rather than estimated (read straight out of the buckets):
  velero  4.0 GiB / 6,264 objects  (7d retention; 8.65 GB live -> kopia dedup)
  cnpg    211 MiB / 1,755 objects  (keycloak 66Mi/331, nextcloud 87Mi/1329,
                                    semaphore 58Mi/95; ~7d of a 30d window)
Both clusters project to ~13-15 GiB and ~30k writes/month => pennies on R2, and
free egress means restore REHEARSALS cost nothing, so R-STATE-03 has no financial
disincentive to stay proven. Wasabi/Hetzner are 50-200x worse here purely because
of 1 TB minimums; Glacier tiers are a trap (90-day minimum billing against a
7-day Velero retention plus early-delete fees on 30-day WAL pruning).

*** THE R2 GOTCHA, WIRED UP FRONT: checksumAlgorithm: "" ***
velero-plugin-for-aws v1.14.2 bundles an AWS SDK that sends
x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT. R2 REJECTS
those with HTTP 400 InvalidArgument, so every backup and every kopia blob upload
would fail. Set unconditionally — MinIO ignores it, so it needs no fork.

`region` was hardcoded us-east-1 inside the Velero HelmRelease: correct-by-
accident for MinIO (which ignores it) and WRONG for R2 (which demands exactly
`auto`). Now the S3_BACKUP_REGION knob.

PREFIXES ARE NOW SYMMETRIC (dev/prod), reversing the reasoning from two commits
ago. While the target was MinIO, dev HAD to keep the bucket root or it would
orphan the backups its proven restore depends on. R2 starts BOTH clusters on an
empty bucket, so there is no history to orphan — and leaving the bucket root
unused means a misrendered prefix lands somewhere obviously wrong instead of
silently colliding with a real environment.

NEITHER OVERLAY PINS THE ENDPOINT ANY MORE — a deliberate reversal of this
repo's "prod must never inherit base" rule. That rule exists because base
defaults are dev's LAN addresses; R2 is a single environment-INDEPENDENT endpoint,
so under R-ENV-02 it belongs in base, and two copies of one URL is a rotation
waiting to drift. Isolation comes from the path knobs, not the URL. Caught while
verifying: dev's overlay DID still pin MinIO, so dev would have taken the new
`dev` prefix while writing to the old endpoint — orphaning its MinIO backups for
no benefit. That is why the endpoint and the prefixes must move in ONE commit.

*** MERGING THIS RESETS DEV'S PITR WINDOW. *** Dev's R2 prefix starts empty:
Velero writes a fresh kopia repo, CNPG takes new base backups, the 30-day Barman
window rebuilds from zero. The MinIO copy is NOT deleted and stays restorable by
pointing the endpoint back. Do not merge while dev's proven restore is the only
recovery path you rely on.

STILL BLOCKED ON THE OPERATOR: the endpoint is the loud placeholder
https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com. Creating the R2 buckets and
an API token needs Cloudflare account access I do not have (the token in this
repo is Zone:DNS scoped). Velero+R2 and CNPG-barman+R2 are also UNPROVEN here and
must be verified by an actual backup AND restore, not just an upload.

VERIFIED: full CI gate PASS. Rendered both overlays — dev dev/s3://cnpg/dev,
prod prod/s3://cnpg/prod, region auto, one shared endpoint.

Refs R-STATE-03, R-ENV-02, R-IDP-05.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 11:29:22 -05:00

340 lines
19 KiB
YAML
Raw Blame History

---
# clusters/dev — the ACTIVE build target overlay.
#
# `flux bootstrap ... --path=clusters/dev` points the dev cluster's Flux here. It reuses the base
# reconciliation graph and patches the shared cluster-vars ConfigMap with dev-scoped values (R-ENV-02).
# It also adds a self-managing ROOT Kustomization (sync.yaml) so the whole graph reconciles from git
# on every push — no more manual `kubectl apply -k` for cluster-vars/overlay changes (F-OPS1).
#
# ##############################################################################################
# ### *** DO NOT MERGE THIS FILE TO `main` UNATTENDED (as of 2026-09-04) *** ###
# ### ###
# ### This overlay now carries dev's DNS MIGRATION target: BASE_DOMAIN moves from the apex ###
# ### `home.keithjackson.dev` to `dev.home.keithjackson.dev`, and the GitRepository url ###
# ### moves with it. sync.yaml reconciles ./clusters/dev from `main` EVERY 10 MINUTES, so a ###
# ### merge to main IS the cutover — it executes on the live dev cluster with nobody present. ###
# ### Dev's Gitea is both its GitOps source and the push target: if the new host is not up ###
# ### with a Ready cert first, dev stops reconciling and pushing cannot repair it. ###
# ### ###
# ### Run the ordered runbook: infra/PROD-PROVISIONING.md §10 (steps D1-D8), operator ###
# ### present. Merging is step D3, not step one. ###
# ##############################################################################################
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
- sync.yaml # self-managing root Kustomization → ./clusters/dev
patches:
# --- SOPS decryption + bootstrap source (PM-inlined from bootstrap/flux/patches/, 2026-08-16) ---
# Inlined (not cross-dir file refs) to stay within kustomize's default load restrictions.
# Realises the commented decryption TODOs in clusters/base and sets the real source URL.
# The `sops-age` Secret is created out-of-band (bootstrap/sops/install-age-key.sh) — name only.
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: platform
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# SOPS decryption on EACH app wave (the app tier is now apps-wave-0..N, not a single `apps`
# Kustomization). Same stanza the retired `apps` Kustomization received — one per wave, by
# name. Add a patch here whenever `render.py --emit-waves` introduces a new deepest wave
# (DESIGN-waves.md cutover note). The `sops-age` Secret is created out-of-band (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-wave-0
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-wave-1
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-wave-2
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# oauth2-proxy moved to its own post-platform Flux Kustomization (clusters/base/oauth2-proxy.yaml).
# It carries oauth2-proxy-credentials.sops.yaml, previously decrypted via the `platform`
# Kustomization; now that this tier owns the encrypted Secret it needs its own decryption stanza.
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: oauth2-proxy
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# headlamp moved to its own post-platform Flux Kustomization (clusters/base/headlamp.yaml).
# It carries headlamp-oidc-credentials.sops.yaml, previously decrypted via the `platform`
# Kustomization; now that this tier owns the encrypted Secret it needs its own decryption stanza.
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: headlamp
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# crafty has its own post-platform Flux Kustomization (clusters/base/crafty.yaml). It carries
# crafty-oauth2-proxy-credentials.sops.yaml, so it needs its own SOPS decryption stanza (same as
# headlamp/oauth2-proxy). The `sops-age` Secret is created out-of-band (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: crafty
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# velero (backup controller) has its own post-platform Flux Kustomization (clusters/base/backup.yaml).
# It carries velero-cloud.sops.yaml (the scoped MinIO `cloud` credential), so it needs its own SOPS
# decryption stanza (same as headlamp/oauth2-proxy/crafty). The `sops-age` Secret is out-of-band (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: velero
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: flux-system
namespace: flux-system
spec:
# Phase-6 cutover done: source is now the in-cluster Gitea; bootstrap host-Gitea retained
# stopped as recovery. PUBLIC repo => Flux needs no creds; Flux resolves the host in-cluster
# via coredns -> Gateway VIP, and the LE-prod wildcard cert is publicly trusted so NO
# secretRef/caCert is needed. Recovery: restart bootstrap Gitea (192.168.2.103:3000) and
# revert this url (docs/05).
#
# *** THE STRAND. THIS URL IS THE MOST DANGEROUS LINE IN THE REPOSITORY. ***
# It is BOTH dev's GitOps source AND the operator's push target, and it is NOT tokenized —
# it cannot be: the `flux-system` root Kustomization that renders ./clusters/dev declares NO
# postBuild.substituteFrom, so `gitea.${BASE_DOMAIN}` here would ship the LITERAL token and
# break the source outright (the L0 flux-postbuild-substitution check asserts exactly this).
# So it is a hand-maintained literal that must be moved in lockstep with BASE_DOMAIN below.
#
# It has been moved to dev's NEW suffix as part of the 2026-09-04 migration authoring. That
# makes this file the TARGET state, NOT a safe unattended merge: dev's root Kustomization
# (sync.yaml) reconciles ./clusters/dev from `main` every 10m, so MERGING THIS TO main WHILE
# DEV IS LIVE MIGRATES DEV UNATTENDED. If gitea.dev.home.keithjackson.dev does not already
# resolve and serve a valid cert at that instant, dev stops reconciling and CANNOT BE FIXED
# BY PUSHING — the broken thing is the path Flux uses to read pushes.
# ORDERING THAT AVOIDS IT (infra/PROD-PROVISIONING.md §10, steps D1-D3): suspend dev's root
# Kustomization or park this url on the bootstrap Gitea (http://192.168.2.132:3000/...) for
# the migration window, and only restore the in-cluster url after the new host is proven.
url: https://gitea.dev.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
ref:
branch: main
# --- dev-scoped cluster-vars (R-ENV-02) ---
- target:
kind: ConfigMap
name: cluster-vars
patch: |-
- op: replace
path: /data/CLUSTER_ENV
value: "dev"
# DNS-01 via LE PRODUCTION: in-cluster OIDC clients (oauth2-proxy, Grafana) must trust the
# wildcard cert for SSO to work; the staging root isn't in any trust store. Prod issuer is
# registered + Ready; one wildcard cert is well within LE rate limits — ADR-0006, R-TLS-01/03.
- op: replace
path: /data/ACME_CLUSTER_ISSUER
value: "letsencrypt-prod"
# ==========================================================================================
# DEV DNS IDENTITY (R-DNS-01/02/03) — DEV MOVES OFF THE APEX. **NOT YET APPLIED.**
# ==========================================================================================
# OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day:
#
# dev -> dev.home.keithjackson.dev (this knob; = the base default, restated)
# prod -> home.keithjackson.dev (THE APEX — prod IS the apex, no `prod.` label,
# and therefore no apex CNAME layer at all)
#
# *** THIS VALUE IS AHEAD OF THE LIVE CLUSTER. *** The dev cluster is serving the APEX right
# now. Merging this file does not perform the migration — it DEFINES the target. The cutover is
# an ORDERED, OPERATOR-PRESENT runbook: infra/PROD-PROVISIONING.md §10.
#
# WHY IT CANNOT BE A PLAIN VARIABLE FLIP — THE STRAND. `spec.url` of the GitRepository ~20
# lines above is https://gitea.home.keithjackson.dev/... . Dev's Gitea is BOTH dev's GitOps
# source AND the operator's push target. The moment BASE_DOMAIN moves, dev's Gitea HTTPRoute,
# DNS record and wildcard cert all move to gitea.dev.home.keithjackson.dev. If that host does
# not ALREADY resolve with a valid, Ready cert, dev stops reconciling — and it CANNOT BE FIXED
# BY PUSHING, because the thing that is broken is the path Flux uses to read pushes. Recovery
# is then hands-on (point the source at the bootstrap Gitea on 192.168.2.132:3000, or repair
# in-cluster by hand). §10 states the ordering that avoids it: publish and verify the new host
# FIRST, or park the source on the bootstrap Gitea for the window.
#
# ALSO MOVED BY THIS ONE LINE, all on a live cluster: the `*.` wildcard Certificate (a fresh
# ACME order), Keycloak's issuer URL and ~12 OIDC clients' redirect URIs, kubectl SSO (a k3s
# server flag => a rolling restart + every user re-issuing their kubeconfig), coredns-custom's
# split-horizon zone, Homepage's links, and 6 L2 chainsaw tests that hardcode the apex.
- op: replace
path: /data/BASE_DOMAIN
value: "dev.home.keithjackson.dev"
# ExternalDNS scope. MUST stay the Cloudflare ZONE — NOT dev.home.keithjackson.dev. FINDING
# F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this for ZONE SELECTION, so a
# strict-subdomain filter matches no zone and publishes NOTHING ("no hosted zone matching
# record DNS Name"). Dev moving to a subdomain does NOT make narrowing safe — it makes it
# newly tempting, which is why this comment is here. Same value the retired DNS_ZONE knob held.
- op: replace
path: /data/EXTERNALDNS_DOMAIN_FILTER
value: "keithjackson.dev"
# OWNER ID: `dev` (was `default` until 2026-09-04). The old value was FORCED — dev's live apex
# records were already in Cloudflare's TXT registry under owner `default`, so renaming would
# have orphaned every one of them. THE MIGRATION DISSOLVES THAT CONSTRAINT: on the new suffix
# dev publishes a brand-new record set with no prior registry entries to recognise, so the id
# is chosen on its merits, and `default` has none — it is the value external-dns uses when you
# FORGET the flag, which is exactly how both clusters came to share it. Consequences, both
# deliberate:
# - dev's OLD apex records remain owned by `default`, which no environment now claims, so
# nothing deletes them implicitly. Retiring them is an explicit §10 step, and prod's first
# publish is gated on that retirement being CONFIRMED — not on two reconcile loops racing.
# - prod stays `prod` and must NEVER be given `default`, or prod would own the live dev
# cluster's apex records and sync-delete them. Prod inherits the apex NAMES, not its OWNER.
# dev != prod is the load-bearing invariant; the base default stays a non-colliding sentinel.
- op: replace
path: /data/EXTERNALDNS_TXT_OWNER_ID
value: "dev"
# --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
# EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
# equals the base default. These values are what the LIVE dev cluster is running right now, so
# the substituted output is byte-identical to the pre-parameterization hardcoded manifests
# (verified 2026-08-31). Never let this block drift from the live cluster.
#
# Split-horizon DNS target: the Cilium-L2 VIP of the `shared` Gateway on the Proxmox LAN
# (= CILIUM_LB_POOL_START, the pool's first address). coredns-custom answers
# *.dev.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
- op: replace
path: /data/GATEWAY_VIP
value: "192.168.1.53"
# k3s API endpoint Cilium's kube-proxy replacement dials — the primary control-plane node
# (k3s-1). [parity] equals the Ansible inventory's k3s_api_host for dev.
- op: replace
path: /data/K3S_API_HOST
value: "192.168.1.50"
# Cilium LB IP pool = the tofu reservation cilium_l2_vip_range .53-.60, outside the node IPs
# (.50/.51/.52) and outside the DHCP scope. .53 Gateway, .54 unifi, .55 crafty; .56-.60 free.
- op: replace
path: /data/CILIUM_LB_POOL_START
value: "192.168.1.53"
- op: replace
path: /data/CILIUM_LB_POOL_STOP
value: "192.168.1.60"
# Pinned L4 VIPs out of that pool (lbipam.cilium.io/ips). Changing either MOVES a live device-
# facing endpoint: unifi is the inform/STUN target every adopted UniFi device holds.
- op: replace
path: /data/UNIFI_L4_VIP
value: "192.168.1.54"
- op: replace
path: /data/CRAFTY_L4_VIP
value: "192.168.1.55"
# OFF-CLUSTER BACKUP TARGET — no op here any more, deliberately.
#
# This used to pin the on-Proxmox MinIO (http://192.168.1.9:9000). As of 2026-09-05 the operator
# moved backups to Cloudflare R2 "for the time being until a better on-prem backup solution is
# created", and R2 is a single environment-INDEPENDENT endpoint: both clusters write to the same
# buckets and are isolated by VELERO_BACKUP_PREFIX / CNPG_BACKUP_PATH (dev vs prod), not by URL.
# Under R-ENV-02 a value that does not differ per environment belongs in base and must NOT be
# restated here — two copies of one endpoint is just a rotation waiting to drift.
#
# *** MERGING THIS RESETS DEV'S PITR WINDOW. *** Dev's R2 prefix starts EMPTY, so Velero writes a
# fresh kopia repository and CNPG takes new base backups; the 30-day Barman window rebuilds from
# zero. The existing MinIO copy is NOT deleted and stays restorable — point S3_BACKUP_ENDPOINT
# back at http://192.168.1.9:9000 (and VELERO_BACKUP_PREFIX back to "") to reach it. Do not merge
# this while dev's proven restore is the only recovery path you are relying on.
# dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
- op: replace
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
value: "3"
# DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
# on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
# mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
# else would orphan every one of them (they are not recoverable from the old path by
# rescheduling). Restated explicitly even though it equals the base default, exactly like the
# LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
# its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
- op: replace
path: /data/LONGHORN_DATA_PATH
value: "/var/lib/longhorn"
# smaller retention / volumes to keep the dev workstation footprint light.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION
value: "3d"
# 5Gi was too tight: 3d of this cluster's series volume (~1.2Gi/day of blocks) plus a WAL that
# ballooned past 1Gi overran the disk, wedging WAL commits and firing every absent()/==0 alert
# falsely. Bumped to 8Gi (still lighter than base 10Gi) so the size-cap below has real headroom.
- op: replace
path: /data/MONITORING_PROMETHEUS_PVC_SIZE
value: "8Gi"
# Size-cap kept well below the 8Gi PVC (headroom for WAL + head + compaction scratch) so
# Prometheus prunes oldest blocks by size and never fills the disk on this young cluster.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
value: "5GiB"
- op: replace
path: /data/MONITORING_LOKI_PVC_SIZE
value: "5Gi"
# single-replica control-plane addons in dev (no HA needed to iterate).
- op: replace
path: /data/KEYCLOAK_REPLICAS
value: "1"
- op: replace
path: /data/OAUTH2_PROXY_REPLICAS
value: "1"
- op: replace
path: /data/VELERO_RETENTION
value: "72h"
# dev-softened Kyverno for cold-start (base defaults Enforce/Fail/3 = prod posture).
# Not credential-driven — bootstrap-safety knobs so a cold cluster comes up before
# admission is proven/HA; flip dev to the strict posture once green.
- op: replace
path: /data/KYVERNO_VALIDATION_ACTION
value: "Audit"
- op: replace
path: /data/KYVERNO_FORCE_FAILURE_POLICY_IGNORE
value: "true"
# 2 (not base-default 1): the webhook is fail-closed, so a single-pod restart 502s every
# admission-gated write cluster-wide. Two replicas give a fail-closed webhook availability
# headroom during a restart/upgrade while staying dev-modest (prod runs 3).
- op: replace
path: /data/KYVERNO_ADMISSION_REPLICAS
value: "2"
Reference in new issue View Git Blame Copy Permalink