jakman85andClaude Opus 5
ff0533106b
1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public
upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS
forwarding on dev cannot recur. Prod deliberately differs from dev, which is
PiHole-first because dev needs the local jackson.lan zone.
SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer *.home.keithjackson.dev and
only one is configured here:
pods -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin
answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster;
PiHole is NOT in this path. Prod is already correct (.20).
LAN -> PiHole. Not a cluster concern.
nodes -> THIS FILE.
The one thing public node DNS breaks: the k3s API SERVER is a node process carrying
oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the
issuer through the NODE resolver, so prod would get the PUBLIC record instead of the
prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node
resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are
recorded in PROD-PROVISIONING.md §4 for the operator to pick.
2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the
first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering
within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD-* : no prod backup target
exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is
not DR for prod.
3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and
proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un'
--become` returns root. The L0 no-plaintext-secrets check asserts every ansible
vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate.
4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so
EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now
sets collections_path = collections, and the collections are installed there from the
existing collections/requirements.yml (gitignored — pinned by requirements, fetched at
setup, vendored bytes are not source).
5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers
are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of
rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power
losses on a consumer drive with NO power-loss protection, in a single failure domain.
etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without
PLP is exactly how a member corrupts.
6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open
item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on
one node is outweighed by the other two and causes node ejection. That holds for the
dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity +
rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates
faithfully to all three replicas, so quorum sees no disagreement — low probability, and
the only mitigation is different hardware. Follow-up noted: Longhorn's
snapshotDataIntegrity is left at the chart default rather than set explicitly, and it
is the mechanism that operationalises the operator's own argument.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not
open — clusters/prod is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on Proxmox VE (192.168.1.50/.51/.52), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap
(R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only;
the full report is conformance/_artifacts/coverage/coverage.md.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.