jakman85andClaude Opus 5
3f980d2a17
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).
New cluster-vars knobs (base default = dev literal, per the file's existing convention):
K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml
CILIUM_LB_POOL_STOP "
UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml
CRAFTY_L4_VIP platform/crafty/service-l4.yaml
S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
semaphore}-db.yaml, and platform/velero/helmrelease.yaml
Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.
clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.
PROD ADDRESS PLAN (operator, 2026-08-31):
nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
Cilium LB pool .20-.30
CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.
Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.
DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.
Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
platform/velero
Velero — off-cluster backup and DR of Kubernetes resource state + PVC data.
- Governing ADR: ADR-0003 — Longhorn (+ bulk tier + Velero)
- Spec: docs/04 — Storage & State
- Serves: R-STATE-03
- Owner: WS-STORAGE
What this is (LIVE)
Flux HelmRelease (Velero chart 12.x / Velero 1.18.x) wired to the on-Proxmox MinIO S3 target:
- Endpoint:
http://192.168.1.9:9000(plain HTTP, path-style, regionus-east-1placeholder) - BucketStorageLocation
default: provideraws, bucketvelero,s3ForcePathStyle=true. - AWS plugin (
velero/velero-plugin-for-aws) loaded as an init container. - File System Backup (kopia):
deployNodeAgent=true,uploaderType=kopia,snapshotsEnabled=false— storage-agnostic PVC backup, no CSI snapshot / data-mover. - Credentials:
velero-cloud-credentials— a SOPS Secret (velero-cloud.sops.yaml) holding thecloudINI key, the scoped MinIO key limited to thevelero+cnpgbuckets. Kept in sync with the staging source of truthplatform/backup/minio-credentials.sops.yamland the Proxmox host ansible-vault.
Reconciled by its own non-gating Flux Kustomization clusters/base/backup.yaml
(dependsOn: platform-config, wait:false) — a backup controller must never gate the platform graph.
Backup scope (what's included vs excluded)
Two daily Schedules (cadence ${VELERO_BACKUP_SCHEDULE}, retention ${VELERO_RETENTION}):
| Schedule | Namespaces | Volumes (kopia FSB) |
|---|---|---|
daily |
ALL except monitoring, unifi |
all PVC data (defaultVolumesToFsBackup:true, opt-out) |
daily-resources-only |
monitoring, unifi |
none (defaultVolumesToFsBackup:false) — resources only |
Excluded volume data (reconstructable, per the sizing decision):
monitoring— Prometheus TSDB PVC (ephemeral) + Loki 72h-log PVC. Grafana/Alertmanager areemptyDir(no PVC); Grafana dashboards are GitOps-provisioned.unifi— the ~7.4 GiB KubeVirt VM disk (huge; UniFi is protected by its own config export).
Their Kubernetes resources are still captured (by daily-resources-only) to aid DR.
Included (the irreplaceable ~6 GiB): every other namespace's PVC data via kopia — Nextcloud data,
Gitea, Plex/*arr config, PiHole, and the CNPG Postgres PVCs. (CNPG DBs additionally have native Barman
PITR to the cnpg bucket — see the DB manifests.)
Post-deploy (PM)
velero backup-location get→defaultAvailable.velero backup create --from-schedule daily→ objects land unders3://velero/.- DR drill (R-STATE-03, Phase-4 gate):
velero restore create --from-backup <name>into a scratch namespace and verify PVC data restores via the node-agent.