jakman85andClaude Opus 5
666a60776a
P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix for the incident, not the feature gate. Verified against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION- gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision N is exactly what writes a newly-added cluster-vars knob. Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all 11 substituting Kustomizations chain transitively through it (platform-config -> platform; the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config). The "not chaining through platform" set is EMPTY, so this one edge protects the whole graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/ sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready on apply success without health-gating its children. Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest anywhere under clusters/<env> — including clusters/base — now stalls platform and everything downstream, where today it would not. Fail-closed, per doctrine. P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README). There is no gotk-components.yaml in git, and the live kustomize-controller Deployment carries NO Flux ownership labels — it was installed imperatively by `flux install` from bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT the controllers. Nothing under clusters/** can reach a controller flag. So it uses Flux's documented mechanism for hand-installed clusters: a new bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a container's args is atomic and a merge patch would replace all five install-generated args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the patch, ASSERT the gate string is present so a future Flux bump that breaks the patch target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is vendored, so components can never drift from the pinned FLUX_VERSION. Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial Deployment as a Flux-managed resource (a future prune would delete kustomize-controller). ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive on a git push — the controller Deployment is outside Flux's inventory, so an operator must re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race turns a transient empty render into a transient hard Kustomization failure. ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left platform blocked forever on a flux-system Kustomization that does not exist, and nothing downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev. Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7): prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs the prod age-key decision, a security choice not a copy), and no GitRepository source URL patch (needs a decision on whether prod tracks the same branch as dev, which would make a dev-facing merge reconcile prod). Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS. Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not
open — clusters/prod is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on Proxmox VE (192.168.1.50/.51/.52), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap
(R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only;
the full report is conformance/_artifacts/coverage/coverage.md.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.