jakman85andClaude Opus 5
e72050c690
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no runner -- while the tree asserted both. This corrects the documents that carried, and in several cases CAUSED, that belief. CLAUDE.md gains section 2b: never infer platform state from this repository. It carries the specific API and kubectl checks for protection, runners, CI status, workloads and Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea job reports success). This is the rule whose absence produced every other finding. .gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host, this file is dormant". Gitea had been live for weeks. Both were written as plans and never revisited once the plan completed. The Transition section is now marked COMPLETE and says what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI). DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously hit "required status check; runner incluster-dind online" with nothing marking it false until a month later. The correction now sits on the bullet itself. ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an expected phase. That framing is why the gap went unexamined. apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to `sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded that a fresh token alone does not re-register a runner holding a stale /data/.runner. apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder. It has not since the claim -- it holds the spent token, which is inert and deliberately left in place. catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was uncited, upstream documents no expiry, and it was the basis for a since-disproven claim that the pipeline was too slow to fit inside it. Measured: ~2m45s. Both .sops.yaml files were verified to still decrypt after the comment edits (`sops -d > /dev/null`), and render==golden confirms the catalog comment does not propagate into the rendered manifest. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
152 lines
8.3 KiB
Markdown
152 lines
8.3 KiB
Markdown
# clusters/
|
|
|
|
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
|
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
|
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
|
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
|
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
|
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
|
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
|
> coverage hole, not a phase.
|
|
>
|
|
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
|
> documentation rather than measurement, and `main` was unprotected with zero runners
|
|
> registered the whole time. **Query the live API before asserting platform state.**
|
|
|
|
|
|
Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
|
|
|
|
Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
|
|
[`docs/02-environments-and-provisioning.md`](../../home-network-k3s/docs/02-environments-and-provisioning.md),
|
|
[ADR-0004 (Flux)](../../home-network-k3s/decisions/adr-0004-gitops-flux.md).
|
|
|
|
> Phase 0 scaffolding only. These are structurally-correct stubs. **No cluster exists** — nothing
|
|
> here reconciles until WS-INFRA delivers a cluster and bootstraps Flux (roadmap Phase 1).
|
|
|
|
## Layout — base + overlay (never fork the base)
|
|
|
|
```
|
|
clusters/
|
|
base/ shared Flux objects (one authoring, both environments)
|
|
flux-system.yaml GitRepository — the reconciliation source (bootstrap → Gitea)
|
|
platform.yaml Flux Kustomization → ./platform (reconciles first)
|
|
apps-wave-0.yaml Flux Kustomization → ./apps/wave-0 (dependsOn platform-config)
|
|
apps-wave-1.yaml Flux Kustomization → ./apps/wave-1 (dependsOn apps-wave-0)
|
|
apps-wave-2.yaml Flux Kustomization → ./apps/wave-N (dependsOn apps-wave-N-1)
|
|
cluster-vars.yaml ConfigMap of overlay knobs (BASE defaults)
|
|
kustomization.yaml
|
|
dev/ ACTIVE build target — patches cluster-vars with dev values only
|
|
kustomization.yaml
|
|
prod/ DESIGN-ONLY until roadmap Phase 7 — patches cluster-vars with prod values only
|
|
kustomization.yaml
|
|
```
|
|
|
|
## How the overlay model satisfies R-ENV-02
|
|
|
|
The base authors the reconciliation graph **once**. `platform.yaml` and the app-wave Kustomizations
|
|
(`apps-wave-0/1/2.yaml` — the single `apps.yaml` was retired; `apps-vms.yaml` deliberately omits it,
|
|
see below) declare `postBuild.substituteFrom` the `cluster-vars` ConfigMap. Platform and
|
|
app manifests use `${TOKEN}` placeholders for every value that legitimately differs between
|
|
environments (issuer, replica counts, storage sizes, retention, node placement).
|
|
|
|
**`clusters/dev` and `clusters/prod` differ ONLY by the values they patch into `cluster-vars`** —
|
|
they add no resources of their own and never copy/fork a base manifest. Promotion to prod
|
|
([`docs/02`](../../home-network-k3s/docs/02-environments-and-provisioning.md)) is a change to the
|
|
prod overlay's variables, nothing else.
|
|
|
|
### LAN addressing is a knob, not a literal (fixed 2026-08-31)
|
|
|
|
Until 2026-08-31 the base manifests hardcoded the **dev** LAN: the k3s API host Cilium dials, the
|
|
Cilium LB IP pool range, the pinned crafty/unifi L4 VIPs and the off-cluster S3 backup endpoint. That
|
|
broke R-ENV-02 outright — `clusters/prod` would have rendered dev addresses (192.168.1.x) onto the
|
|
prod LAN (192.168.2.x). They are now the `cluster-vars` "LAN addressing" block
|
|
(`K3S_API_HOST`, `CILIUM_LB_POOL_START`/`_STOP`, `UNIFI_L4_VIP`, `CRAFTY_L4_VIP`,
|
|
`S3_BACKUP_ENDPOINT`, alongside the pre-existing `GATEWAY_VIP`), set explicitly by BOTH overlays.
|
|
|
|
Two rules when touching that block:
|
|
|
|
1. **Substitution coverage.** A `${TOKEN}` only resolves in a path whose Flux Kustomization declares
|
|
`postBuild.substituteFrom: cluster-vars`; anywhere else the literal string `${TOKEN}` ships to the
|
|
cluster. `apps-vms` was the one path without it and now has it — which means every shell `${...}`
|
|
in a VM manifest's cloud-init must be escaped `$${...}` (see `clusters/base/apps-vms.yaml` for the
|
|
full envsubst escape contract).
|
|
2. **`GATEWAY_VIP` must be inside `CILIUM_LB_POOL_START..STOP`,** and in practice equal to START: the
|
|
`shared` Gateway pins no address, so Cilium LB-IPAM gives it the pool's first free IP while
|
|
split-horizon CoreDNS answers with `${GATEWAY_VIP}`. A VIP outside the pool blackholes in-cluster
|
|
hairpin traffic while the Gateway still looks healthy.
|
|
|
|
Values the operator has not decided are `TBD-PROD-*` placeholders, not guesses: prod inherits the
|
|
DEV base defaults if an overlay op is simply omitted, so an invalid value that fails loudly is
|
|
strictly safer than a plausible one that quietly comes up on the wrong network.
|
|
|
|
## How Flux is pointed at an environment
|
|
|
|
`flux bootstrap` (run by WS-INFRA in Phase 1) targets one environment's path per cluster:
|
|
|
|
```
|
|
# dev cluster
|
|
flux bootstrap ... --path=clusters/dev
|
|
# prod cluster (Phase 7 only — design-only until then)
|
|
flux bootstrap ... --path=clusters/prod
|
|
```
|
|
|
|
Bootstrap materializes the `flux-system` `Kustomization` that renders the chosen overlay, which pulls
|
|
in `base/` and applies the env-scoped `cluster-vars`.
|
|
|
|
## Reconciliation order
|
|
|
|
```
|
|
GitRepository (flux-system)
|
|
│
|
|
▼
|
|
Kustomization: platform (./platform, substituteFrom cluster-vars)
|
|
│ healthy
|
|
▼
|
|
Kustomization: apps-wave-0 (./apps/wave-0, dependsOn platform-config, substituteFrom cluster-vars)
|
|
│ healthy
|
|
▼
|
|
Kustomization: apps-wave-1 (./apps/wave-1, dependsOn apps-wave-0)
|
|
│ healthy
|
|
▼
|
|
Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1)
|
|
```
|
|
|
|
### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
|
|
|
|
The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
|
|
controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
|
|
and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
|
|
reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
|
|
before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
|
|
**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
|
|
`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
|
|
have existed for months); it cost 12h on prod's cold start.
|
|
|
|
So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
|
|
platform-config`), and a rule:
|
|
|
|
> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
|
|
> **controller at runtime** rather than by a manifest in git.
|
|
|
|
`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
|
|
`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
|
|
profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
|
|
|
|
Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
|
|
exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
|
|
(`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own
|
|
wave, so a crashloop in a later wave can no longer stall an earlier wave or block new revisions.
|
|
See [DESIGN-waves.md](../DESIGN-waves.md) for the wave table + cutover runbook.
|
|
|
|
## TODOs (later phases — do not action in Phase 0)
|
|
|
|
- **Source cutover (WS-GITOPS, Phase 6):** `base/flux-system.yaml` currently points at the bootstrap
|
|
source. After Gitea is onboarded, repoint `spec.url` to `gitea.home.keithjackson.dev` and keep an
|
|
off-cluster mirror (spec `docs/05` "Source-of-truth", `CONTRIBUTING.md`).
|
|
- **SOPS decryption (WS-SEC, Phase 1):** add `spec.decryption.provider: sops` (age secret name only,
|
|
never plaintext) to the platform/apps Kustomizations once the age key is bootstrapped
|
|
([ADR-0007](../../home-network-k3s/decisions/adr-0007-secrets-sops-age.md)).
|
|
- **Split ordered Flux Kustomizations** if intra-platform ordering needs to be explicit (e.g. a
|
|
separate `cilium`/`longhorn` Kustomization others `dependsOn`).
|
|
</content>
|