jakman85andClaude Opus 5
59ffbaf11a
Six L2 behavioural tests embedded `home.keithjackson.dev`. They are test INPUTS,
not reconciled manifests, so postBuild substitution never touched them — correct
while dev held the apex, wrong the moment prod came up on
prod.home.keithjackson.dev, and blocking §9's "suite passes green against prod".
Each test now asks the cluster what it is, via a test-level chainsaw binding over
flux-system/cluster-vars — the SAME ConfigMap Flux renders the platform from, so
a test cannot disagree with the cluster it runs on:
bindings:
- name: baseDomain
value: (x_k8s_get($client,'v1','ConfigMap','flux-system','cluster-vars').data.BASE_DOMAIN)
Chosen over a flag or env var because there is nothing to forget at run time, and
test-level scope covers every step (step outputs/bindings are step-scoped).
THE APEX WAS NOT THE ONLY LEAK. dns-exposure and dns-internal-resolution also
hardcoded DEV'S Cilium LB range 192.168.1.53-.60; prod's is 192.168.2.20-.30, so
both were wrong for prod regardless of hostname. Replaced with assertions on the
OBSERVED value — the DNS answer must equal that cluster's GATEWAY_VIP, and the
public record must equal the LB IP the Service was actually assigned — which is
strictly STRONGER than the old range check as well as environment-independent.
sso-forward-auth held the literal only in prose (it talks over Service DNS).
PROVEN, NOT ASSERTED — and the positive control is the important half:
* expressions extracted verbatim from the tracked files resolve to the correct,
DIFFERENT values per cluster (dev home.keithjackson.dev / 192.168.1.53,
prod prod.home.keithjackson.dev / 192.168.2.20);
* the same assert with the HARDCODED apex passes on dev and FAILS on prod
("Invalid value: *.prod.home... Expected: *.home..."), so the harness is
provably not blind — that is exactly the drift removed;
* chainsaw lint VALID on all six; run.sh --lint clean over the whole tree.
END-TO-END L2 RUN AGAINST LIVE DEV: `dns-exposure` PASSED (42.6s) — the binding
produced conformance-dns.home.keithjackson.dev, external-dns published it, and the
probe Job asserted the record against the assigned LB IP. This is the first time
one of these tests has been executed since parameterization, and it exercises the
full path rather than just the binding.
FINDING FROM THAT RUN, worth recording: the test's teardown is ASYNCHRONOUS. It
reports PASS as soon as chainsaw deletes the namespace, but the Cloudflare record
is reaped later by external-dns's own --interval=1m sync. A zone query immediately
after PASS still showed conformance-dns.home.keithjackson.dev plus its owner=default
TXT registry entry. Harmless here, but it means a green dns-exposure does NOT prove
cleanup happened, and if external-dns were suspended the record would persist
silently. Verified separately that it clears.
conformance/incluster/rbac.yaml: its cluster-wide `get configmaps` is now
LOAD-BEARING for these bindings — narrowing it to the test namespace breaks five
tests. Commented so it is not "tidied" later.
l3-fault-injection/dr-restore: its KNOWN-GAPS block said off-host is unproven
because the MinIO shares a Proxmox host with dev's VMs. Still true as run, so the
text now says both — that the R2 move (32d6766) is what closes it, that this test's
pass is against MinIO, and that the ledger row proving R-STATE-03 PREDATES the move,
so off-host DR cannot be claimed until the drill is re-run against R2.
NOT VERIFIED, plainly: no full L2 suite run against prod, so "green against prod"
remains unproven — what is proven is that the tests now derive the CORRECT
expectation for whichever cluster they are pointed at. Whether prod SATISFIES them
(split-horizon, LE wildcard, Grafana's issuer) is untested.
VERIFIED: full CI gate PASS. flux-postbuild-substitution 260 assertions,
render-golden 10/10.
Refs R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-ENV-02, R-DNS-01/02/03.
Co-Authored-By: Claude Opus 5 <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not
open — clusters/prod is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on Proxmox VE (192.168.1.50/.51/.52), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap
(R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only;
the full report is conformance/_artifacts/coverage/coverage.md.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.