jakman85andClaude Opus 5
6d62aca0d2
OPERATOR CORRECTION (2026-09-05). I had been calling the resolver that shadows
prod "a pfSense override" throughout this repo. Both halves of that were wrong:
- It is the WORKSTATION'S OWN dnsmasq, not pfSense. The live resolver is the
one NetworkManager spawns (dns=dnsmasq, --conf-dir=/etc/NetworkManager/
dnsmasq.d); the standalone dnsmasq.service is INACTIVE, so /etc/dnsmasq.d is
dormant. The dev profile's `address=/.home.keithjackson.dev/192.168.2.50`
matches the domain AND EVERY SUBDOMAIN, so it also swallowed the whole new
prod.home.* tree and answered it with dev's address.
- The k3s prod cluster is NOT behind pfSense. It sits on the SOHO router
network 192.168.2.0/24, which is pfSense's WAN; pfSense CONTAINS the dev LAN
192.168.1.0/24. This workstation is on 192.168.2.0/24 too, which is why the
rules below use prod's real VIPs and need no NAT hop, unlike dev's .50.
New infra/workstation/dnsmasq/ — authored, not click-ops:
20-prod.home.keithjackson.dev.conf prod.home.* -> 192.168.2.20 (Gateway),
unifi -> .21, crafty-mc -> .22 (L4 VIPs)
install.sh installs + reloads; refuses non-root
dnsmasq resolves `address=` by LONGEST MATCH, so the prod rules win WITHOUT
editing or reordering the dev profile. PROVEN, not assumed: ran a throwaway
dnsmasq on 127.0.0.1:5354 loaded with both profiles —
prod.home / gitea.prod / keycloak.prod -> 192.168.2.20
unifi.prod -> 192.168.2.21 crafty-mc.prod -> 192.168.2.22
gitea.home / home.keithjackson.dev -> 192.168.2.50 (DEV, UNCHANGED)
`dnsmasq --test` on the new file: syntax OK.
install.sh pins the destination because getting it wrong is SILENT: a file in
/etc/dnsmasq.d exists, looks installed, and changes nothing. It also uses
`systemctl reload NetworkManager` rather than SIGHUP — dnsmasq does not re-read
its config on SIGHUP, so a HUP would look like it worked and not apply anything.
WHY THE MISLABEL MATTERED: the trap PARTIALLY works. Dev serves a
*.home.keithjackson.dev wildcard cert, which matches prod.home.keithjackson.dev
(one label) so the prod apex returns 200 over VALID TLS, while
keycloak.prod.home.keithjackson.dev (two labels) fails with a certificate-name
error. It reads as a broken prod cert; it is the wrong server answering. I made
exactly that misreading during verification.
Also recorded, because it changes what the apex flip IS: the SOHO router NATs
inbound 443 to the pfSense WAN (192.168.2.102) to serve the EXISTING NON-K3S
production environment, which must stay live until the switch is deliberately
made. So §10a sequence P is not just a k3s-side DNS/TLS change — it is the
legacy-prod cutover, and P as written does NOT cover repointing that NAT. The
two must be planned together or the apex resolves to k3s while inbound 443 still
lands on the legacy box. Phase C touches none of it: prod's Gateway is
192.168.2.20:443 (a different address) and its cert is DNS-01, needing no
inbound 443. And the MinIO timeout is now correctly described as expected
topology — crossing INTO pfSense from its WAN — not a misconfiguration.
VERIFIED: full CI gate PASS (spec-immutability, lint, l0, render-golden, l1).
Refs R-DNS-02, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not
open — clusters/prod is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on Proxmox VE (192.168.1.50/.51/.52), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap
(R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only;
the full report is conformance/_artifacts/coverage/coverage.md.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.