jakman85andClaude Opus 5
e7c4a95c2d
The four backup-credential SOPS Secrets are SHARED BASE manifests: platform/ and apps/ are
rendered by the same Flux tiers on both clusters, and the ONLY per-environment difference in
this repo is cluster-vars substitution (R-ENV-02). So writing the R2 token into all four
(0a6a862) did not "give prod credentials" — it REPLACED DEV'S. Merging that as-is would have
handed dev's Velero and all three dev CNPG Clusters an R2 key id to sign MinIO requests with:
403 on every WAL segment, and archive_mode=on turns that into a disk-full clock on the pgdata
PVC, not a degraded backup. Dev's PROVEN DR restore (R-STATE-03) is the thing that would have
broken.
This commit splits the material. Each credential is now TWO Secret objects, both applied to
BOTH clusters, differing only in name suffix:
dev -> velero-cloud-credentials keycloak/nextcloud/semaphore-db-backup-credentials
prod -> velero-cloud-credentials-prod ...-db-backup-credentials-prod
The consumer picks one by name; that wiring is the NEXT commit, so this one changes no
rendered reference and is a no-op for both live clusters.
WHY DEV KEEPS THE UNSUFFIXED NAME. Dev is live. Renaming its Secret would change
`s3Credentials.accessKeyId.name` on three running CNPG Clusters — which projects into the
instance pods' environment and therefore ROLLS keycloak-db, nextcloud-db and semaphore-db —
and would roll the velero Deployment, all to rename an object whose contents do not change.
This repo already makes exactly this call for dev's live identifiers (EXTERNALDNS_TXT_OWNER_ID
"default", VELERO_BACKUP_PREFIX ""): the live cluster keeps the name it is running, and the
NEW environment takes the new one. The asymmetry is deliberate and documented at each
kustomization.
WHERE THE MATERIAL CAME FROM. Neither side is invented:
- the four unsuffixed files are restored byte-for-byte from `main` (the MinIO material as
it stood before 0a6a862), so dev's Secrets are IDENTICAL to what dev is running;
- the four `-prod` files carry the R2 token from 0a6a862 (already on this branch as
9739e9b), re-encrypted only because metadata.name changed.
RE-ENCRYPTION, AND WHY A COPY+SED IS NOT ENOUGH. `.sops.yaml` sets
`encrypted_regex: ^(data|stringData)$`, so metadata.name stays cleartext — but SOPS MACs the
WHOLE document (there is no `mac_only_encrypted` here), so editing that cleartext line
invalidates the MAC and `sops -d` then refuses the file. The `-prod` files were therefore
produced inside the dev toolbox (infra/devcontainer, which has sops 3.9.4 and the age key
mounted) by decrypt -> rename -> re-encrypt, with the plaintext never leaving a container
tmpfile. Note for the next person: `sops -e --output NEW OLD` matches creation rules against
the INPUT path, so encrypting a tmpfile into a repo path needs `--filename-override NEW` or
it dies with "no matching creation rules found".
Verified: all eight files `sops -d` cleanly; the four dev files hash to dev's material and the
four prod files to R2's, and the two sets differ. `.sops.yaml` needed no new rule — the
existing `(platform|apps|catalog)/.*(secret|sops)\.ya?ml$` rule already matches
`*-prod.sops.yaml`, so the new files encrypt to the same age recipient.
apps/wave-0/kustomization.yaml is GENERATED, so render.py learns the `-prod` sidecar rather
than the golden being hand-edited: `<db>-backup-credentials-prod.sops.yaml` now places into
the DB's wave alongside its unsuffixed twin (both must exist before the Cluster that names
one), and the selftest asserts it.
Co-Authored-By: Claude Opus 5 <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not
open — clusters/prod is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on Proxmox VE (192.168.1.50/.51/.52), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap
(R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only;
the full report is conformance/_artifacts/coverage/coverage.md.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.