jakman85andClaude Opus 5
6b88670129
Two runbook corrections from executing step A of the dev migration. 1. P1 said "only NOW unsuspend prod", gated on D8 (dev's apex vacated). That silently assumed prod would be serving the bare apex. IT IS NOT. Prod runsc8c1d29, where BASE_DOMAIN=prod.home.keithjackson.dev and EXTERNALDNS_TXT_OWNER_ID=prod — verified live in prod's applied cluster-vars. Prod's names are a DISJOINT SUBTREE under a DISTINCT registry owner, so neither §10a incident mode applies: it cannot claim dev's `default`-owned records, and *.prod.home.keithjackson.dev is a DIFFERENT name set from dev's, so it has its own Let's Encrypt duplicate-cert bucket and races no _acme-challenge with dev. Prod can therefore be brought up IN FULL while dev still holds the apex — and should be, because it proves the whole stack before the flip, turning the cutover into a DNS/TLS change on a known-good cluster rather than a bring-up and a cutover simultaneously. D8 gates ONLY the later move of prod's BASE_DOMAIN to the apex. 2. D1 was executed via the REPOINT option, not suspend — repointing removes the dependency that causes the strand instead of merely deferring it. Dev now fetches from http://192.168.2.132:3000/... branch `dev-live`: IP-addressed, plain HTTP, dependent on NEITHER DNS NOR TLS, which is the property a DNS/TLS migration needs. A merge to `main` is now inert for dev. THE TRAP, recorded because it is not obvious: a bare copy of main@4809199 on the bootstrap Gitea would NOT have held. The root renders ./clusters/dev, and clusters/dev/kustomization.yaml CONTAINS the patch that sets GitRepository.spec.url — so Flux would have fetched from the bootstrap Gitea and then applied manifests that set the url straight back, reverting the repoint and re-arming the strand. The branch must declare its own source. dev-live (a4a3c2e) carries exactly one content change on4809199, proven by `kubectl kustomize clusters/dev` differing in EXACTLY TWO of 357 lines. NEW DEPENDENCY, and it is the weak link: 192.168.2.132 is a DHCP lease on the workstation's WIFI (wlp0s20f3, NetworkManager ipv4.method=auto — confirmed by inspection, not assumed). A lease change or the workstation going offline stalls dev's FETCHING; the cluster keeps serving and the fix is a kubectl patch. Hence: a static reservation is worth having, and the window should be kept short — the exit is D4. Pointing Flux at gitea.gitea.svc.cluster.local would drop the DNS, TLS and workstation dependencies at once and is the better long-term shape, but namespace `gitea` has default-deny plus a Cilium policy admitting only the Gateway, so it needs a new authored policy — deliberately, not mid-migration. Post-cutover verification: 13/13 Kustomizations Ready on dev-live@a4a3c2e, cluster-vars byte-identical to the D0 snapshot (BASE_DOMAIN still the apex), 18/18 Longhorn volumes attached+healthy, all 16 HTTPRoute hostnames unchanged, and the Cloudflare record set re-queried and diffed byte-for-byte against D0 — identical, 57 records, 36 TXT all owner=default. Refs R-ENV-02, R-GITOPS-01. Co-Authored-By: Claude Opus 5 <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not
open — clusters/prod is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on Proxmox VE (192.168.1.50/.51/.52), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap
(R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only;
the full report is conformance/_artifacts/coverage/coverage.md.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.