jakman85andClaude Opus 5
f21e89d9d2
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
138 lines
7.4 KiB
Markdown
138 lines
7.4 KiB
Markdown
# clusters/
|
|
|
|
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
|
|
|
|
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
|
|
[`docs/02-environments-and-provisioning.md`](../../home-network-k3s/docs/02-environments-and-provisioning.md),
|
|
[ADR-0004 (Flux)](../../home-network-k3s/decisions/adr-0004-gitops-flux.md).
|
|
|
|
> Phase 0 scaffolding only. These are structurally-correct stubs. **No cluster exists** — nothing
|
|
> here reconciles until WS-INFRA delivers a cluster and bootstraps Flux (roadmap Phase 1).
|
|
|
|
## Layout — base + overlay (never fork the base)
|
|
|
|
```
|
|
clusters/
|
|
base/ shared Flux objects (one authoring, both environments)
|
|
flux-system.yaml GitRepository — the reconciliation source (bootstrap → Gitea)
|
|
platform.yaml Flux Kustomization → ./platform (reconciles first)
|
|
apps-wave-0.yaml Flux Kustomization → ./apps/wave-0 (dependsOn platform-config)
|
|
apps-wave-1.yaml Flux Kustomization → ./apps/wave-1 (dependsOn apps-wave-0)
|
|
apps-wave-2.yaml Flux Kustomization → ./apps/wave-N (dependsOn apps-wave-N-1)
|
|
cluster-vars.yaml ConfigMap of overlay knobs (BASE defaults)
|
|
kustomization.yaml
|
|
dev/ ACTIVE build target — patches cluster-vars with dev values only
|
|
kustomization.yaml
|
|
prod/ DESIGN-ONLY until roadmap Phase 7 — patches cluster-vars with prod values only
|
|
kustomization.yaml
|
|
```
|
|
|
|
## How the overlay model satisfies R-ENV-02
|
|
|
|
The base authors the reconciliation graph **once**. `platform.yaml` and the app-wave Kustomizations
|
|
(`apps-wave-0/1/2.yaml` — the single `apps.yaml` was retired; `apps-vms.yaml` deliberately omits it,
|
|
see below) declare `postBuild.substituteFrom` the `cluster-vars` ConfigMap. Platform and
|
|
app manifests use `${TOKEN}` placeholders for every value that legitimately differs between
|
|
environments (issuer, replica counts, storage sizes, retention, node placement).
|
|
|
|
**`clusters/dev` and `clusters/prod` differ ONLY by the values they patch into `cluster-vars`** —
|
|
they add no resources of their own and never copy/fork a base manifest. Promotion to prod
|
|
([`docs/02`](../../home-network-k3s/docs/02-environments-and-provisioning.md)) is a change to the
|
|
prod overlay's variables, nothing else.
|
|
|
|
### LAN addressing is a knob, not a literal (fixed 2026-08-31)
|
|
|
|
Until 2026-08-31 the base manifests hardcoded the **dev** LAN: the k3s API host Cilium dials, the
|
|
Cilium LB IP pool range, the pinned crafty/unifi L4 VIPs and the off-cluster S3 backup endpoint. That
|
|
broke R-ENV-02 outright — `clusters/prod` would have rendered dev addresses (192.168.1.x) onto the
|
|
prod LAN (192.168.2.x). They are now the `cluster-vars` "LAN addressing" block
|
|
(`K3S_API_HOST`, `CILIUM_LB_POOL_START`/`_STOP`, `UNIFI_L4_VIP`, `CRAFTY_L4_VIP`,
|
|
`S3_BACKUP_ENDPOINT`, alongside the pre-existing `GATEWAY_VIP`), set explicitly by BOTH overlays.
|
|
|
|
Two rules when touching that block:
|
|
|
|
1. **Substitution coverage.** A `${TOKEN}` only resolves in a path whose Flux Kustomization declares
|
|
`postBuild.substituteFrom: cluster-vars`; anywhere else the literal string `${TOKEN}` ships to the
|
|
cluster. `apps-vms` was the one path without it and now has it — which means every shell `${...}`
|
|
in a VM manifest's cloud-init must be escaped `$${...}` (see `clusters/base/apps-vms.yaml` for the
|
|
full envsubst escape contract).
|
|
2. **`GATEWAY_VIP` must be inside `CILIUM_LB_POOL_START..STOP`,** and in practice equal to START: the
|
|
`shared` Gateway pins no address, so Cilium LB-IPAM gives it the pool's first free IP while
|
|
split-horizon CoreDNS answers with `${GATEWAY_VIP}`. A VIP outside the pool blackholes in-cluster
|
|
hairpin traffic while the Gateway still looks healthy.
|
|
|
|
Values the operator has not decided are `TBD-PROD-*` placeholders, not guesses: prod inherits the
|
|
DEV base defaults if an overlay op is simply omitted, so an invalid value that fails loudly is
|
|
strictly safer than a plausible one that quietly comes up on the wrong network.
|
|
|
|
## How Flux is pointed at an environment
|
|
|
|
`flux bootstrap` (run by WS-INFRA in Phase 1) targets one environment's path per cluster:
|
|
|
|
```
|
|
# dev cluster
|
|
flux bootstrap ... --path=clusters/dev
|
|
# prod cluster (Phase 7 only — design-only until then)
|
|
flux bootstrap ... --path=clusters/prod
|
|
```
|
|
|
|
Bootstrap materializes the `flux-system` `Kustomization` that renders the chosen overlay, which pulls
|
|
in `base/` and applies the env-scoped `cluster-vars`.
|
|
|
|
## Reconciliation order
|
|
|
|
```
|
|
GitRepository (flux-system)
|
|
│
|
|
▼
|
|
Kustomization: platform (./platform, substituteFrom cluster-vars)
|
|
│ healthy
|
|
▼
|
|
Kustomization: apps-wave-0 (./apps/wave-0, dependsOn platform-config, substituteFrom cluster-vars)
|
|
│ healthy
|
|
▼
|
|
Kustomization: apps-wave-1 (./apps/wave-1, dependsOn apps-wave-0)
|
|
│ healthy
|
|
▼
|
|
Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1)
|
|
```
|
|
|
|
### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
|
|
|
|
The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
|
|
controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
|
|
and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
|
|
reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
|
|
before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
|
|
**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
|
|
`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
|
|
have existed for months); it cost 12h on prod's cold start.
|
|
|
|
So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
|
|
platform-config`), and a rule:
|
|
|
|
> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
|
|
> **controller at runtime** rather than by a manifest in git.
|
|
|
|
`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
|
|
`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
|
|
profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
|
|
|
|
Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
|
|
exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
|
|
(`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own
|
|
wave, so a crashloop in a later wave can no longer stall an earlier wave or block new revisions.
|
|
See [DESIGN-waves.md](../DESIGN-waves.md) for the wave table + cutover runbook.
|
|
|
|
## TODOs (later phases — do not action in Phase 0)
|
|
|
|
- **Source cutover (WS-GITOPS, Phase 6):** `base/flux-system.yaml` currently points at the bootstrap
|
|
source. After Gitea is onboarded, repoint `spec.url` to `gitea.home.keithjackson.dev` and keep an
|
|
off-cluster mirror (spec `docs/05` "Source-of-truth", `CONTRIBUTING.md`).
|
|
- **SOPS decryption (WS-SEC, Phase 1):** add `spec.decryption.provider: sops` (age secret name only,
|
|
never plaintext) to the platform/apps Kustomizations once the age key is bootstrapped
|
|
([ADR-0007](../../home-network-k3s/decisions/adr-0007-secrets-sops-age.md)).
|
|
- **Split ordered Flux Kustomizations** if intra-platform ordering needs to be explicit (e.g. a
|
|
separate `cilium`/`longhorn` Kustomization others `dependsOn`).
|
|
</content>
|