0 Fork 0

Files

T

jakman85andClaude Opus 5 12c40d8c9c docs: reconcile the documentation with the deployed reality

The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.

Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.

README.md
  - Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
  - Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
  - Real conformance number (36/51, 70.6%) and how to regenerate it.
  - Devcontainer workflow; note that the gate was red and why.

infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
  cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
  paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
  the toolchain, off-repo state that must not be lost, and the fact that Flux's
  git source is an app inside the cluster (`git push origin main` IS the deploy).

infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
  rationale are still worth keeping; the libvirt execution steps are not.

infra/README.md — Proxmox, plus a map of the subtree and where the cluster
  actually lives (it is not on this workstation).

infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
  is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
  talks to an HTTP API. Everything runs in the container now. Documents the full
  CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
  and the read-only spec repo especially — without the latter, spec-immutability
  silently skips instead of verifying).

infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
  is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
  against the running nodes), pfSense NAT marked done, and a warning that
  terraform.tfstate is absent on a fresh workstation — an apply with empty state
  would try to recreate the three live nodes.

infra/tofu/variables.tf — section comment disagreed with its own defaults.

infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
  domains that do not exist; behaviour unchanged, but it no longer lies.

apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
  since been dropped. Now: live status, 40Gi, resolved open questions marked
  resolved, and the two constraints the next person will otherwise rediscover —
  KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
  postBuild.substituteFrom (envsubst corrupts the cloud-init).

DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
  migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
  the Flux source cutover, CI going live in-cluster, ten app onboardings and the
  renderer bugs they exposed, the wave split (verified: DBs adopted, not
  recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
  VM, and kubectl SSO. Plus verified live state and the carried-forward open list.

Two corrections to the record while verifying:
  - kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
    [oidc:platform-admins]), so the rolling restart 3472f1d deferred has
    happened.
  - UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
    24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
    and the CNP explicitly permits cluster -> :11443, so it is genuinely not
    listening rather than a policy artifact.

Docs and comments only; no manifest semantics change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-08-27 00:00:05 -05:00

4.8 KiB

Raw Blame History

home-network-k3s-impl

Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts (Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies, Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only specification at ../home-network-k3s.

Rules (from the spec — non-negotiable)

The spec is read-only. Never modify anything under ../home-network-k3s. Implement against it; escalate spec problems, don't edit them (spec AGENTS.md §11, R-TEST-06).
Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a manual demo (spec docs/14, agents/definition-of-done.md).
No work on main. Every change is a feature branch merged into main (spec CONTRIBUTING.md).
No plaintext secrets — SOPS-encrypted only (spec docs/07).
Phase discipline — only build what the active roadmap phase authorizes (spec roadmap/phases.md).

Layout

infra/          node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
                pinned dev container that carries the whole toolchain
bootstrap/      one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/       platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
                longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
                vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
                (ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
                Split out because CRs and the CRDs they use cannot apply in one pass.
apps/           application tier, reconciled as dependency-ordered WAVES:
                  wave-0/ wave-1/ wave-2/   rendered container apps (see DESIGN-waves.md)
                  vms/                      KubeVirt VirtualMachines (non-gating tier)
catalog/        ServiceDefinition entries (the onboarding surface) + render.py
schemas/        ServiceDefinition JSON Schema + renderer contract
clusters/       Flux entrypoints + dev/ and prod/ overlays
conformance/    Chainsaw behavior/conformance suite + harness + coverage report
ci/             the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/         Gitea Actions CI running that gate on an in-cluster runner

Status

Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not open — clusters/prod is a design-only overlay.

Phase	Delivered
0 Foundations	ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI
1 Cluster bring-up	3-node k3s HA on Proxmox VE (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS
2 Platform core	Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS
3 Egress / VPN	ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch
4 Observability	kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled
5 IaaS	KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest
6 Identity + CI	Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate

Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap (R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only; the full report is conformance/_artifacts/coverage/coverage.md.

The running narrative — decisions, root causes, incidents, follow-ups — is DECISIONS-LOG.md. It, not this table, is the authoritative record.

Working on this repo

Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:

infra/devcontainer/dev.sh build            # once
infra/devcontainer/dev.sh run make -C ci gate   # the same gate CI runs
infra/devcontainer/dev.sh                  # interactive shell (sops, flux, kubectl, tofu, ansible)

The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only so the R-TEST-06 spec-immutability check can actually run. See infra/devcontainer/README.md.

The gate is currently RED on main: yamllint commas errors in the hand-authored apps/vms/unifi/ manifests. Every other static stage is green. See the last DECISIONS-LOG.md entry.