jakman85andClaude Opus 5
8f9b2e4270
*** THE BUG, AND IT WAS MINE. *** The placeholder guards were `case "$s3" in
""|TBD-*)` — a PREFIX glob. Two commits earlier (32d6766) I changed the
placeholder from `TBD-PROD-S3-BACKUP-ENDPOINT` to
`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`, which does NOT start with
`TBD-`. Verified: fnmatch('https://TBD-R2-...', 'TBD-*') is False. All three
guards silently stopped matching, so **step_resume would have resumed velero on
prod straight onto a nonexistent hostname** — velero installs, reports healthy,
and no backup exists. That is precisely the failure the guard was written to
prevent, disarmed by a value change in a DIFFERENT file with nothing tying the
two together. Replaced with substring-matching s3_is_placeholder in preflight,
s3check and resume.
s3check rewritten from a reachability ping into a real proof. Per credential, from
a throwaway in-cluster pod (pod egress is the only egress that matters — the
workstation sits on a different network side and a host-side reading has already
misled this project once):
1. the credential LOADS AND AUTHENTICATES in the exact shape its consumer reads
— Velero's INI mounted from the `cloud` key, CNPG's discrete env keys;
2. the bucket EXISTS and is LISTABLE under this environment's own prefix, via
ListObjectsV2 — the op Velero's BSL sync and Barman's WAL scan actually use.
ListBuckets deliberately avoided: a bucket-scoped R2 token may not grant it;
3. PUT -> GET -> sha256+length compare -> DELETE -> confirm-gone. A HEAD would
pass with a read-only token and with every write being rejected;
4. the R2 checksum trap probed ON PURPOSE by repeating the PUT with checksums
enabled, and named either way.
Buckets and prefixes are DERIVED (velero's from the authored HelmRelease via yq,
cnpg's parsed from CNPG_BACKUP_PATH), not restated.
Fails closed, with no "unknown, carrying on" arm left: no pod, unfinished pod,
empty log, missing/duplicate/malformed verdict, any failed assertion, a claimed
PASS with failures>0, an assertion count below the caller-computed minimum, an
OK-line count disagreeing with the claimed count, or a PASS from a pod that ended
Failed. Failures are classified by cause — no egress, NoSuchBucket,
InvalidAccessKeyId, AccessDenied (token not scoped to BOTH buckets),
InvalidArgument -> the checksum trap.
`amazon/aws-cli` pinned rather than `minio/mc`: mc lacks sed/awk (which already
broke one probe) AND cannot send checksum headers, so it cannot test for the trap.
TIER_ORDER now includes platform-config-cdi, between platform-config and the leaf
tiers. Without it `resume` returned success without ever waiting on the tier that
guarantees the CDI StorageProfiles — and apps-vms depends on them: unifi
provisions on longhorn-vm, where an unapplied profile means CDI infers RWO instead
of the authored RWX, permanently non-live-migratable.
CREDENTIALS: one key, four committed Secrets, two shapes — and NO manifest change
is needed, only encrypted values. platform/backup/set-r2-credentials.sh writes all
four with `sops set`, which encrypts IN PLACE, so no decrypted Secret ever exists
in the working tree to be committed by accident — strictly better than staging a
plaintext template. Prompts with `read -rs`, rejects anything outside the S3
credential charset (a paste that swallowed a newline authenticates nowhere), never
prints a value, and verifies by decrypting back and comparing fingerprints.
`--check` is read-only. The MinIO key is RETAINED — it is what reads the
pre-migration backups.
THE ACCOUNT ID IS ALREADY IN THE REPO: infra/tofu/backend.r2.hcl commits
https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com for the tofu
state bucket. If the backup buckets go in the same account, only the two token
values are needed. The placeholder is KEPT anyway — fail-closed beats convenience,
and the guard above just demonstrated why.
Corroborating the checksum fix independently: infra/tofu/backend.tf already carries
`region = "auto"` and `skip_s3_checksum = true # R2 rejects AWS trailing
(streaming) checksums`. This repo had learned both R2 facts for tofu state already.
VERSION FINDINGS, documented not measured against R2: plugin v1.14.2 documents
checksumAlgorithm defaulting to CRC32, so the authored "" is load-bearing;
Velero's kopia backend has NO checksum knob, so BSL metadata and PVC-data
compatibility are SEPARATE questions; and in the live CNPG 16.4 image,
boto3/botocore 1.35.60 is below the 1.36 line that turned on default request
checksums — so CNPG should be clear today, but barman-cloud exposes no escape
hatch, making a CNPG image bump an R2 re-verification event.
VERIFIED BY ME, independently of the agent: the dead-guard glob (fnmatch False);
the placeholder now refused on prod before any pod is created; a full round trip
against dev's MinIO — 7/7 assertions on BOTH credential paths, byte-identical,
pod Succeeded, nothing left behind, checksum trap correctly reported absent; all
four Secrets and the committed account id present. Full CI gate PASS.
STILL UNPROVEN AND SAID SO IN THE DOC: Velero-to-R2 (metadata or kopia data),
CNPG-barman-to-R2, and any restore from R2. R-STATE-03 is PROVEN MinIO-ONLY and
its ledger row predates this move. infra/BACKUP-R2.md §5 is the re-proof sequence
and it ends in a RESTORE, not an upload.
Refs R-STATE-03, R-IDP-05, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not
open — clusters/prod is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on Proxmox VE (192.168.1.50/.51/.52), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap
(R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only;
the full report is conformance/_artifacts/coverage/coverage.md.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.