jakman85andClaude Opus 5
4f37fc375e
Prod is up and serving. 113 pods, 0 not Running/Completed. wildcard-home Certificate Ready (Let's Encrypt PROD, *.prod.home.keithjackson.dev). Gateway VIP 192.168.2.20, crafty L4 .22, unifi L4 .21. 36 DNS records published under prod.home.keithjackson.dev, every TXT registry entry owner=prod. End-to-end HTTPS with certificate validation (no -k), pinned to prod's gateway: apex 200, keycloak 302 (login), gitea 200, grafana 302 (SSO). DEV UNTOUCHED: all 57 pre-existing records byte-identical to the pre-standup baseline. velero remains suspended — S3_BACKUP_ENDPOINT is still the TBD placeholder, and resuming it would create a BackupStorageLocation that can never reach a bucket: a tier reporting healthy while backups silently do not exist. TWO BUGS IN MY OWN VERIFICATION, both of which would have made the safety check worthless. Recording them because a check you cannot trust is worse than none — it trains you to skip the one alarm that matters. 1. SNAP_DIR defaulted to $TMPDIR. Every `dev.sh run` is a FRESH container, so the DNS baseline was discarded between steps and `verify`'s before/after comparison — the check that detects damage to the other cluster's records — could never run across invocations. Now repo-relative (bind-mounted, so it persists on the host) and gitignored as a run artifact. 2. The first real comparison cried wolf: "RECORDS OUTSIDE prod.home CHANGED", listing pairs of IDENTICAL lines. Cause was locale-dependent sort collation — the host ordered `*.keithjackson.dev` differently from the container, so a diff of two identical sets reported changes. Proved it was noise by re-comparing under LC_ALL=C: all 57 records identical. dns_snapshot now strips CRs and sorts under LC_ALL=C so ordering is reproducible everywhere. ACTION REQUIRED (operator, pfSense): the local override is a wildcard on *.home.keithjackson.dev and now also swallows the new prod.home.* subtree, resolving EVERY prod hostname to 192.168.2.50 (dev). Cloudflare correctly returns 192.168.2.20. Prod is reachable from anywhere using public DNS, but NOT from behind pfSense. Narrow the override, or add one pointing prod.home.* at 192.168.2.20. That override is also the trap that produced a false healthy reading during the 2026-09-04 incident, and it is subtle because it PARTIALLY works: dev's *.home.keithjackson.dev wildcard matches prod.home.keithjackson.dev (one label) so the apex returns 200 over valid TLS, while keycloak.prod.home... (two labels) fails with a certificate-name error. I hit this myself during verification and mistook it for a broken prod cert. Never verify DNS or TLS here through a local resolver. Refs R-ENV-02, R-GITOPS-01, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not
open — clusters/prod is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on Proxmox VE (192.168.1.50/.51/.52), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap
(R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only;
the full report is conformance/_artifacts/coverage/coverage.md.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.