jakman85andClaude Opus 5
380ea98a38
The second requirement ever proven on this project, and the one the operator had attested was working but the suite could not attest to. Test authored 2026-08-31 carrying a `conformance.home.k3s/never-executed` annotation; executed for the first time tonight. test dr-restore run 20260902T001846Z-fdc3265 req R-STATE-03 (safety-critical, DR) junit passed / 0 failed / 0 skipped sentinel DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK proof proven shafdc3265(clean tree) kube v1.36.4+k3s1 WHAT IT ACTUALLY PROVED, in three steps: 1. The backup target is genuinely OFF-CLUSTER. BSL velero/default phase=Available at http://192.168.1.9:9000 — endpoint resolved and asserted to be none of: loopback, any node Internal/ExternalIP, any Service ClusterIP, any Pod IP, inside any node podCIDR, or inside any ServiceCIDR. 2. Backups are SCHEDULED, not merely possible — velero-daily and velero-daily-resources-only, both phase=Enabled, last run 2026-09-01T03:00:27Z. 3. The drill: seed a nanosecond-unique canary onto a longhorn PVC; Velero Backup to Completed with a kopia PodVolumeBackup carrying a real snapshotID (volume DATA reached the remote repository, not just resource JSON); ANNIHILATE the source — delete the namespace, poll until gone, force-delete the PV, FAIL if the original PV survived; Restore into a clean namespace; assert the restored PVC bound a DIFFERENT PV and the canary bytes are byte-identical. The bytes could only have come from outside the cluster. VALIDATES THE PARSER FIX FROM57d4df8. This log contains SEVEN occurrences of DR_RESTORE_SKIPPED_OK — every one of them inside a `=== COMMAND` script echo — and the pass token twice in real `=== STDOUT`. Under the pre-fix scanner this genuine pass would have been recorded as `skipped`, exactly as node-loss was. The section-parsing rule is doing real work, not theoretical work. CLEANUP VERIFIED against a pre-run baseline: Velero backups back to 7 (its own chainsaw-*-dr-backup deleted via DeleteBackupRequest), 0 chainsaw backups, 0 restores, 0 orphan Released/Available PVs, both scratch namespaces gone, 13/13 Kustomizations Ready, 0 pods not Running. It wrote into and cleaned out of the SAME bucket and kopia repository the live nightly schedule uses, and left it as it found it. SEPARATE FINDING, cleaned up: `chainsaw-stable-worm` was still Active from 2026-08-25 — empty residue from an earlier conformance session that never tore its namespace down. Deleted. Worth knowing that Chainsaw namespace leaks can outlive a run by a week unnoticed. NOT proven by this test, stated in its own header rather than implied: off-HOST DR (the dev nodes are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor loss); no direct S3 client read; real service data (Nextcloud/Gitea/CNPG drills remain operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC. Serves: R-STATE-03, R-TEST-03, R-TEST-05. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not
open — clusters/prod is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on Proxmox VE (192.168.1.50/.51/.52), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap
(R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only;
the full report is conformance/_artifacts/coverage/coverage.md.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.