home-network-k3s-impl/clusters

clusters/

STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below. This document speaks as though no cluster exists and the platform is scaffolding. A three-node k3s cluster has existed for weeks and Flux has been reconciling against it. Gitea is live, the Actions runner incluster-dind is online, and main is a protected branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as describing a live gap rather than an expected stage — most importantly, L2/L3 conformance still does not run in CI (vars.CLUSTER_AVAILABLE is unset), which is a real coverage hole, not a phase.

This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on documentation rather than measurement, and main was unprotected with zero runners registered the whole time. Query the live API before asserting platform state.

Flux entrypoints and the environment-overlay model (R-ENV-02, R-GITOPS-01/02/03).

Governing spec: docs/05-gitops-and-delivery.md, docs/02-environments-and-provisioning.md, ADR-0004 (Flux).

Phase 0 scaffolding only. These are structurally-correct stubs. No cluster exists — nothing here reconciles until WS-INFRA delivers a cluster and bootstraps Flux (roadmap Phase 1).

Layout — base + overlay (never fork the base)

clusters/
  base/                 shared Flux objects (one authoring, both environments)
    flux-system.yaml    GitRepository — the reconciliation source (bootstrap → Gitea)
    platform.yaml       Flux Kustomization → ./platform   (reconciles first)
    apps-wave-0.yaml    Flux Kustomization → ./apps/wave-0 (dependsOn platform-config)
    apps-wave-1.yaml    Flux Kustomization → ./apps/wave-1 (dependsOn apps-wave-0)
    apps-wave-2.yaml    Flux Kustomization → ./apps/wave-N (dependsOn apps-wave-N-1)
    cluster-vars.yaml   ConfigMap of overlay knobs (BASE defaults)
    kustomization.yaml
  dev/                  ACTIVE build target — patches cluster-vars with dev values only
    kustomization.yaml
  prod/                 DESIGN-ONLY until roadmap Phase 7 — patches cluster-vars with prod values only
    kustomization.yaml

How the overlay model satisfies R-ENV-02

The base authors the reconciliation graph once. platform.yaml and the app-wave Kustomizations (apps-wave-0/1/2.yaml — the single apps.yaml was retired; apps-vms.yaml deliberately omits it, see below) declare postBuild.substituteFrom the cluster-vars ConfigMap. Platform and app manifests use ${TOKEN} placeholders for every value that legitimately differs between environments (issuer, replica counts, storage sizes, retention, node placement).

clusters/dev and clusters/prod differ ONLY by the values they patch into cluster-vars — they add no resources of their own and never copy/fork a base manifest. Promotion to prod (docs/02) is a change to the prod overlay's variables, nothing else.

LAN addressing is a knob, not a literal (fixed 2026-08-31)

Until 2026-08-31 the base manifests hardcoded the dev LAN: the k3s API host Cilium dials, the Cilium LB IP pool range, the pinned crafty/unifi L4 VIPs and the off-cluster S3 backup endpoint. That broke R-ENV-02 outright — clusters/prod would have rendered dev addresses (192.168.1.x) onto the prod LAN (192.168.2.x). They are now the cluster-vars "LAN addressing" block (K3S_API_HOST, CILIUM_LB_POOL_START/_STOP, UNIFI_L4_VIP, CRAFTY_L4_VIP, S3_BACKUP_ENDPOINT, alongside the pre-existing GATEWAY_VIP), set explicitly by BOTH overlays.

Two rules when touching that block:

1. Substitution coverage. A ${TOKEN} only resolves in a path whose Flux Kustomization declares postBuild.substituteFrom: cluster-vars; anywhere else the literal string ${TOKEN} ships to the cluster. apps-vms was the one path without it and now has it — which means every shell ${...} in a VM manifest's cloud-init must be escaped $${...} (see clusters/base/apps-vms.yaml for the full envsubst escape contract).
2. GATEWAY_VIP must be inside CILIUM_LB_POOL_START..STOP, and in practice equal to START: the shared Gateway pins no address, so Cilium LB-IPAM gives it the pool's first free IP while split-horizon CoreDNS answers with ${GATEWAY_VIP}. A VIP outside the pool blackholes in-cluster hairpin traffic while the Gateway still looks healthy.

Values the operator has not decided are TBD-PROD-* placeholders, not guesses: prod inherits the DEV base defaults if an overlay op is simply omitted, so an invalid value that fails loudly is strictly safer than a plausible one that quietly comes up on the wrong network.

How Flux is pointed at an environment

flux bootstrap (run by WS-INFRA in Phase 1) targets one environment's path per cluster:

# dev cluster
flux bootstrap ... --path=clusters/dev
# prod cluster (Phase 7 only — design-only until then)
flux bootstrap ... --path=clusters/prod

Bootstrap materializes the flux-system Kustomization that renders the chosen overlay, which pulls in base/ and applies the env-scoped cluster-vars.

Reconciliation order

GitRepository (flux-system)
        │
        ▼
Kustomization: platform  (./platform, substituteFrom cluster-vars)
        │  healthy
        ▼
Kustomization: apps-wave-0  (./apps/wave-0, dependsOn platform-config, substituteFrom cluster-vars)
        │  healthy
        ▼
Kustomization: apps-wave-1  (./apps/wave-1, dependsOn apps-wave-0)
        │  healthy
        ▼
Kustomization: apps-wave-N  (./apps/wave-N, dependsOn apps-wave-N-1)

Second-order CRDs need a THIRD pass (platform-config-cdi, added 2026-09-05)

The controllers/configs split (platform → platform-config) assumes a CRD is registered by the controller install. CDI breaks that: cdi-operator.yaml registers only cdis.cdi.kubevirt.io, and cdi-operator creates storageprofiles / datavolumes / datasources only after it has reconciled the CDI CR. Since kustomize-controller server-side dry-runs every object in a pass before applying any of them, having the CDI CR and the CDI StorageProfiles in one pass was a deadlock, not a race: unknown kind → pass aborts → CR never applied → CRD never created → platform-config can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs have existed for months); it cost 12h on prod's cold start.

So there is a third tier, platform-config-cdi (./platform-config-cdi, dependsOn: platform-config), and a rule:

A CR belongs in platform-config-cdi, not platform-config, when the CRD it needs is created by a controller at runtime rather than by a manifest in git.

apps-vms dependsOn it as well as platform-config, because CDI reads StorageProfile/longhorn-vm at DataVolume provisioning time — a VM disk provisioned before that profile exists is stamped RWO/Filesystem and is permanently not live-migratable.

Wave 0 dependsOn platform-config (transitively platform) so CNI/storage/TLS/DNS/gateway exist before any workload reconciles. The app tier is split into dependency-ordered WAVES (catalog/render.py --emit-waves): each apps-wave-<N> health-gates (wait:true) ONLY its own wave, so a crashloop in a later wave can no longer stall an earlier wave or block new revisions. See DESIGN-waves.md for the wave table + cutover runbook.

TODOs (later phases — do not action in Phase 0)

• Source cutover (WS-GITOPS, Phase 6): base/flux-system.yaml currently points at the bootstrap source. After Gitea is onboarded, repoint spec.url to gitea.home.keithjackson.dev and keep an off-cluster mirror (spec docs/05 "Source-of-truth", CONTRIBUTING.md).
• SOPS decryption (WS-SEC, Phase 1): add spec.decryption.provider: sops (age secret name only, never plaintext) to the platform/apps Kustomizations once the age key is bootstrapped (ADR-0007).
• Split ordered Flux Kustomizations if intra-platform ordering needs to be explicit (e.g. a separate cilium/longhorn Kustomization others dependsOn).