Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
9d00a3fcd3ad1da3cebddc52d4d6391fd0ca548a
home-network-k3s-impl/clusters
T
History
jakman85andClaude Opus 5 161430d915 feat(backup): the overlay picks the credential — BACKUP_CREDENTIALS_SUFFIX 
The previous commit created the two halves. This one wires the choice, so dev renders MinIO
credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the
overlay patch is the entire difference between environments; no base manifest is forked).

    cluster-vars  BACKUP_CREDENTIALS_SUFFIX   base ""  |  dev ""  |  prod "-prod"

    velero HelmRelease  credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
    3x CNPG Cluster     s3Credentials.*.name:      <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}

ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters
of the way, leaving one consumer reading the OTHER environment's key with nothing rendering
wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED
variable, never on a defined-but-stale one. With one suffix the four move together or not at
all. The cost is that the rendered Secret name is not greppable from the consumer, so both
kustomizations spell the two names out in full beside the resources.

THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects
that exist in every render, and that dev is running today. The alternative failure mode is
strictly worse than the shared-Secret state this replaces: velero's pod hangs in
ContainerCreating forever when `existingSecret` names a missing Secret (the volume never
mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which
under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable
default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the
value explicitly anyway, so neither inherits it silently.

Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is
untouched (it stays decryption-only, and velero stays suspended on prod).

VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and
clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform,
platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with
that environment's variables:

  strict envsubst              PASSES for all 8 (env x tier) combinations
  dev   velero existingSecret  velero-cloud-credentials
  dev   keycloak/nextcloud/semaphore-db  <db>-backup-credentials         (both key refs each)
  prod  velero existingSecret  velero-cloud-credentials-prod
  prod  keycloak/nextcloud/semaphore-db  <db>-backup-credentials-prod    (both key refs each)
  both  all EIGHT Secret objects present in the rendered output (dev and prod alike)
  no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell
    escapes in apps/wave-0, which are pre-existing and unrelated.

Read alongside the endpoints the same overlays already set, the two environments are now
isolated on credential AND target:
  dev   velero s3://velero (root) + cnpg s3://cnpg      @ http://192.168.1.9:9000  region us-east-1
  prod  velero k3s-prod-backup/velero + barman prefix   @ https://<acct>.r2.cloudflarestorage.com  region auto

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 01:41:56 -05:00
..
base
feat(backup): the overlay picks the credential — BACKUP_CREDENTIALS_SUFFIX
2026-09-07 01:41:56 -05:00
dev
feat(backup): the overlay picks the credential — BACKUP_CREDENTIALS_SUFFIX
2026-09-07 01:41:56 -05:00
prod
feat(backup): the overlay picks the credential — BACKUP_CREDENTIALS_SUFFIX
2026-09-07 01:41:56 -05:00
README.md
fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply
2026-09-06 22:14:19 -05:00

README.md

clusters/

Flux entrypoints and the environment-overlay model (R-ENV-02, R-GITOPS-01/02/03).

Governing spec: docs/05-gitops-and-delivery.md, docs/02-environments-and-provisioning.md, ADR-0004 (Flux).

Phase 0 scaffolding only. These are structurally-correct stubs. No cluster exists — nothing here reconciles until WS-INFRA delivers a cluster and bootstraps Flux (roadmap Phase 1).

Layout — base + overlay (never fork the base)

clusters/
  base/                 shared Flux objects (one authoring, both environments)
    flux-system.yaml    GitRepository — the reconciliation source (bootstrap → Gitea)
    platform.yaml       Flux Kustomization → ./platform   (reconciles first)
    apps-wave-0.yaml    Flux Kustomization → ./apps/wave-0 (dependsOn platform-config)
    apps-wave-1.yaml    Flux Kustomization → ./apps/wave-1 (dependsOn apps-wave-0)
    apps-wave-2.yaml    Flux Kustomization → ./apps/wave-N (dependsOn apps-wave-N-1)
    cluster-vars.yaml   ConfigMap of overlay knobs (BASE defaults)
    kustomization.yaml
  dev/                  ACTIVE build target — patches cluster-vars with dev values only
    kustomization.yaml
  prod/                 DESIGN-ONLY until roadmap Phase 7 — patches cluster-vars with prod values only
    kustomization.yaml

How the overlay model satisfies R-ENV-02

The base authors the reconciliation graph once. platform.yaml and the app-wave Kustomizations (apps-wave-0/1/2.yaml — the single apps.yaml was retired; apps-vms.yaml deliberately omits it, see below) declare postBuild.substituteFrom the cluster-vars ConfigMap. Platform and app manifests use ${TOKEN} placeholders for every value that legitimately differs between environments (issuer, replica counts, storage sizes, retention, node placement).

clusters/dev and clusters/prod differ ONLY by the values they patch into cluster-vars — they add no resources of their own and never copy/fork a base manifest. Promotion to prod (docs/02) is a change to the prod overlay's variables, nothing else.

LAN addressing is a knob, not a literal (fixed 2026-08-31)

Until 2026-08-31 the base manifests hardcoded the dev LAN: the k3s API host Cilium dials, the Cilium LB IP pool range, the pinned crafty/unifi L4 VIPs and the off-cluster S3 backup endpoint. That broke R-ENV-02 outright — clusters/prod would have rendered dev addresses (192.168.1.x) onto the prod LAN (192.168.2.x). They are now the cluster-vars "LAN addressing" block (K3S_API_HOST, CILIUM_LB_POOL_START/_STOP, UNIFI_L4_VIP, CRAFTY_L4_VIP, S3_BACKUP_ENDPOINT, alongside the pre-existing GATEWAY_VIP), set explicitly by BOTH overlays.

Two rules when touching that block:

  1. Substitution coverage. A ${TOKEN} only resolves in a path whose Flux Kustomization declares postBuild.substituteFrom: cluster-vars; anywhere else the literal string ${TOKEN} ships to the cluster. apps-vms was the one path without it and now has it — which means every shell ${...} in a VM manifest's cloud-init must be escaped $${...} (see clusters/base/apps-vms.yaml for the full envsubst escape contract).
  2. GATEWAY_VIP must be inside CILIUM_LB_POOL_START..STOP, and in practice equal to START: the shared Gateway pins no address, so Cilium LB-IPAM gives it the pool's first free IP while split-horizon CoreDNS answers with ${GATEWAY_VIP}. A VIP outside the pool blackholes in-cluster hairpin traffic while the Gateway still looks healthy.

Values the operator has not decided are TBD-PROD-* placeholders, not guesses: prod inherits the DEV base defaults if an overlay op is simply omitted, so an invalid value that fails loudly is strictly safer than a plausible one that quietly comes up on the wrong network.

How Flux is pointed at an environment

flux bootstrap (run by WS-INFRA in Phase 1) targets one environment's path per cluster:

# dev cluster
flux bootstrap ... --path=clusters/dev
# prod cluster (Phase 7 only — design-only until then)
flux bootstrap ... --path=clusters/prod

Bootstrap materializes the flux-system Kustomization that renders the chosen overlay, which pulls in base/ and applies the env-scoped cluster-vars.

Reconciliation order

GitRepository (flux-system)
        │
        ▼
Kustomization: platform  (./platform, substituteFrom cluster-vars)
        │  healthy
        ▼
Kustomization: apps-wave-0  (./apps/wave-0, dependsOn platform-config, substituteFrom cluster-vars)
        │  healthy
        ▼
Kustomization: apps-wave-1  (./apps/wave-1, dependsOn apps-wave-0)
        │  healthy
        ▼
Kustomization: apps-wave-N  (./apps/wave-N, dependsOn apps-wave-N-1)

Second-order CRDs need a THIRD pass (platform-config-cdi, added 2026-09-05)

The controllers/configs split (platform → platform-config) assumes a CRD is registered by the controller install. CDI breaks that: cdi-operator.yaml registers only cdis.cdi.kubevirt.io, and cdi-operator creates storageprofiles / datavolumes / datasources only after it has reconciled the CDI CR. Since kustomize-controller server-side dry-runs every object in a pass before applying any of them, having the CDI CR and the CDI StorageProfiles in one pass was a deadlock, not a race: unknown kind → pass aborts → CR never applied → CRD never created → platform-config can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs have existed for months); it cost 12h on prod's cold start.

So there is a third tier, platform-config-cdi (./platform-config-cdi, dependsOn: platform-config), and a rule:

A CR belongs in platform-config-cdi, not platform-config, when the CRD it needs is created by a controller at runtime rather than by a manifest in git.

apps-vms dependsOn it as well as platform-config, because CDI reads StorageProfile/longhorn-vm at DataVolume provisioning time — a VM disk provisioned before that profile exists is stamped RWO/Filesystem and is permanently not live-migratable.

Wave 0 dependsOn platform-config (transitively platform) so CNI/storage/TLS/DNS/gateway exist before any workload reconciles. The app tier is split into dependency-ordered WAVES (catalog/render.py --emit-waves): each apps-wave-<N> health-gates (wait:true) ONLY its own wave, so a crashloop in a later wave can no longer stall an earlier wave or block new revisions. See DESIGN-waves.md for the wave table + cutover runbook.

TODOs (later phases — do not action in Phase 0)

  • Source cutover (WS-GITOPS, Phase 6): base/flux-system.yaml currently points at the bootstrap source. After Gitea is onboarded, repoint spec.url to gitea.home.keithjackson.dev and keep an off-cluster mirror (spec docs/05 "Source-of-truth", CONTRIBUTING.md).
  • SOPS decryption (WS-SEC, Phase 1): add spec.decryption.provider: sops (age secret name only, never plaintext) to the platform/apps Kustomizations once the age key is bootstrapped (ADR-0007).
  • Split ordered Flux Kustomizations if intra-platform ordering needs to be explicit (e.g. a separate cilium/longhorn Kustomization others dependsOn).
Reference in new issue View Git Blame Copy Permalink