jakman85andClaude Opus 5
7432012713
Unsuspended prod's `platform` after pre-applying the Kyverno CRDs.
8/9 HelmReleases Ready, 76/76 pods Running (cert-manager, CNPG, Cilium,
Longhorn, kube-prometheus-stack, Loki, Alloy, external-dns). `platform` itself
is NOT Ready and the 10 downstream tiers stay blocked.
THE CRD PRE-APPLY WORKED and is now proven against a live API server: 22 CRDs
Established, a second run a clean no-op (rc=0), reproduced under the PINNED
toolchain (helm 3.16.0) matching the 3.20.0 run byte-for-byte. It also confirmed
the predicted failure mode: Kyverno's Deployments had 55 RESTARTS EACH over 4.5h
and stabilised within minutes of the apply — controllers with no API types.
BLOCKER 1 — Kyverno's Helm release is wedged in `uninstalling`. The install
timed out *because* of that crash loop; install.remediation.retries:3 then fired
an uninstall which ALSO timed out, leaving sh.helm.release.v1.kyverno.v1 at
status=uninstalling. helm-controller cannot advance from there, so `platform`
(wait: true) never goes Ready. The Deployments themselves are healthy at the
right chart version — this is stale Helm bookkeeping, not a broken workload.
Fix needs `helm uninstall` + `flux reconcile --force`; destructive, so NOT run
unattended. Safe here because prod carries no workloads and a cold install
proves the bootstrap path. The pre-applied CRDs live OUTSIDE the release and
survive the uninstall — precisely why the pre-apply is a separate step.
BLOCKER 2 — AND THIS ONE IS ARMED. Prod's external-dns Deployment runs
--policy=sync --registry=txt --domain-filter=keithjackson.dev WITH NO
--txt-owner-id. Absent that flag it defaults to `default` — the owner of ALL 36
of dev's TXT registry entries. Scaling it to 1 would claim dev's records and,
under policy: sync, DELETE THEM: the 2026-09-04 incident, exactly.
The repo is not at fault. helmrelease.yaml sets txtOwnerId from cluster-vars,
prod resolves it to `prod`, and the LIVE HelmRelease spec.values carries
txtOwnerId: "prod". Verified against the real chart: 1.21.1 with txtOwnerId:
prod renders --txt-owner-id=prod, so the key is right and an upgrade would fix
it. The fault is Helm/Flux STATE: `helm get values` shows the installed release
has policy and domainFilters but NO txtOwnerId — it predates c8c1d29 — while
helm-controller reports Ready=True, observedGeneration=3, and a history entry
whose configDigest EQUALS lastAttemptedConfigDigest. It believes it already
deployed the current config, so it will never upgrade on its own.
The only thing that prevented the incident was replicas=0 — manual drift from
the incident recovery, which any Helm upgrade resets to the chart default of 1.
That is not a guard, it is luck.
CONTAINED: `flux suspend helmrelease external-dns`, replicas left at 0. Costs
nothing (prod publishes no hostnames yet; app tiers suspended) and is reversible.
Remediation ordering matters and is documented: the naive resume + reconcile
--force adds the flag and starts the pod in ONE operation, leaving no window to
verify — a race whose failure mode is deleting dev's DNS. Pin replicas at 0,
confirm --txt-owner-id=prod in the live args, THEN scale.
Dev verified unharmed throughout: Cloudflare snapshotted before the unsuspend
and re-queried after — 57 records byte-for-byte identical, 36 TXT still
owner=default. Queried against the Cloudflare API, never a local resolver.
Refs R-GITOPS-01, R-ENV-02, R-SEC-01.
Co-Authored-By: Claude Opus 5 <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not
open — clusters/prod is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on Proxmox VE (192.168.1.50/.51/.52), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap
(R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only;
the full report is conformance/_artifacts/coverage/coverage.md.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.