Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
531 Commits 139 Branches 0 Tags
6f61c9622daf94076caa656b3cf74aff0d28c34d
Commit Graph
4 Commits
Author SHA1 Message Date
jakman85andClaude Opus 5 8f9b2e4270 feat(backup): s3check proves a real round trip — and revives a guard I had killed 
*** THE BUG, AND IT WAS MINE. *** The placeholder guards were `case "$s3" in
""|TBD-*)` — a PREFIX glob. Two commits earlier (32d6766) I changed the
placeholder from `TBD-PROD-S3-BACKUP-ENDPOINT` to
`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`, which does NOT start with
`TBD-`. Verified: fnmatch('https://TBD-R2-...', 'TBD-*') is False. All three
guards silently stopped matching, so **step_resume would have resumed velero on
prod straight onto a nonexistent hostname** — velero installs, reports healthy,
and no backup exists. That is precisely the failure the guard was written to
prevent, disarmed by a value change in a DIFFERENT file with nothing tying the
two together. Replaced with substring-matching s3_is_placeholder in preflight,
s3check and resume.

s3check rewritten from a reachability ping into a real proof. Per credential, from
a throwaway in-cluster pod (pod egress is the only egress that matters — the
workstation sits on a different network side and a host-side reading has already
misled this project once):
  1. the credential LOADS AND AUTHENTICATES in the exact shape its consumer reads
     — Velero's INI mounted from the `cloud` key, CNPG's discrete env keys;
  2. the bucket EXISTS and is LISTABLE under this environment's own prefix, via
     ListObjectsV2 — the op Velero's BSL sync and Barman's WAL scan actually use.
     ListBuckets deliberately avoided: a bucket-scoped R2 token may not grant it;
  3. PUT -> GET -> sha256+length compare -> DELETE -> confirm-gone. A HEAD would
     pass with a read-only token and with every write being rejected;
  4. the R2 checksum trap probed ON PURPOSE by repeating the PUT with checksums
     enabled, and named either way.
Buckets and prefixes are DERIVED (velero's from the authored HelmRelease via yq,
cnpg's parsed from CNPG_BACKUP_PATH), not restated.

Fails closed, with no "unknown, carrying on" arm left: no pod, unfinished pod,
empty log, missing/duplicate/malformed verdict, any failed assertion, a claimed
PASS with failures>0, an assertion count below the caller-computed minimum, an
OK-line count disagreeing with the claimed count, or a PASS from a pod that ended
Failed. Failures are classified by cause — no egress, NoSuchBucket,
InvalidAccessKeyId, AccessDenied (token not scoped to BOTH buckets),
InvalidArgument -> the checksum trap.

`amazon/aws-cli` pinned rather than `minio/mc`: mc lacks sed/awk (which already
broke one probe) AND cannot send checksum headers, so it cannot test for the trap.

TIER_ORDER now includes platform-config-cdi, between platform-config and the leaf
tiers. Without it `resume` returned success without ever waiting on the tier that
guarantees the CDI StorageProfiles — and apps-vms depends on them: unifi
provisions on longhorn-vm, where an unapplied profile means CDI infers RWO instead
of the authored RWX, permanently non-live-migratable.

CREDENTIALS: one key, four committed Secrets, two shapes — and NO manifest change
is needed, only encrypted values. platform/backup/set-r2-credentials.sh writes all
four with `sops set`, which encrypts IN PLACE, so no decrypted Secret ever exists
in the working tree to be committed by accident — strictly better than staging a
plaintext template. Prompts with `read -rs`, rejects anything outside the S3
credential charset (a paste that swallowed a newline authenticates nowhere), never
prints a value, and verifies by decrypting back and comparing fingerprints.
`--check` is read-only. The MinIO key is RETAINED — it is what reads the
pre-migration backups.

THE ACCOUNT ID IS ALREADY IN THE REPO: infra/tofu/backend.r2.hcl commits
https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com for the tofu
state bucket. If the backup buckets go in the same account, only the two token
values are needed. The placeholder is KEPT anyway — fail-closed beats convenience,
and the guard above just demonstrated why.
Corroborating the checksum fix independently: infra/tofu/backend.tf already carries
`region = "auto"` and `skip_s3_checksum = true # R2 rejects AWS trailing
(streaming) checksums`. This repo had learned both R2 facts for tofu state already.

VERSION FINDINGS, documented not measured against R2: plugin v1.14.2 documents
checksumAlgorithm defaulting to CRC32, so the authored "" is load-bearing;
Velero's kopia backend has NO checksum knob, so BSL metadata and PVC-data
compatibility are SEPARATE questions; and in the live CNPG 16.4 image,
boto3/botocore 1.35.60 is below the 1.36 line that turned on default request
checksums — so CNPG should be clear today, but barman-cloud exposes no escape
hatch, making a CNPG image bump an R2 re-verification event.

VERIFIED BY ME, independently of the agent: the dead-guard glob (fnmatch False);
the placeholder now refused on prod before any pod is created; a full round trip
against dev's MinIO — 7/7 assertions on BOTH credential paths, byte-identical,
pod Succeeded, nothing left behind, checksum trap correctly reported absent; all
four Secrets and the committed account id present. Full CI gate PASS.

STILL UNPROVEN AND SAID SO IN THE DOC: Velero-to-R2 (metadata or kopia data),
CNPG-barman-to-R2, and any restore from R2. R-STATE-03 is PROVEN MinIO-ONLY and
its ledger row predates this move. infra/BACKUP-R2.md §5 is the re-proof sequence
and it ends in a RESTORE, not an upload.

Refs R-STATE-03, R-IDP-05, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
 2026-09-05 13:24:26 -05:00
jakman85andClaude Opus 5 3f980d2a17 feat(env): tokenize hardcoded dev LAN addresses into cluster-vars (R-ENV-02) 
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).

New cluster-vars knobs (base default = dev literal, per the file's existing convention):
  K3S_API_HOST          platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
  CILIUM_LB_POOL_START  platform-config/cilium/l2-ip-pool.yaml
  CILIUM_LB_POOL_STOP   "
  UNIFI_L4_VIP          apps/vms/unifi/50-service-l4.yaml
  CRAFTY_L4_VIP         platform/crafty/service-l4.yaml
  S3_BACKUP_ENDPOINT    platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
                        semaphore}-db.yaml, and platform/velero/helmrelease.yaml

Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.

clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.

PROD ADDRESS PLAN (operator, 2026-08-31):
  nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
  Cilium LB pool .20-.30

CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.

Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.

DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.

Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-08-31 22:43:36 -05:00
jakman85andClaude Opus 4.8 d1248e6778 feat(backup): wire Velero + CNPG Barman to the live MinIO S3 target 
Author the GitOps side of off-cluster backups against the on-Proxmox MinIO
(http://192.168.1.9:9000, path-style; buckets velero + cnpg).

Velero (R-STATE-03):
- HelmRelease (chart 8.x / Velero 1.15) + velero-plugin-for-aws init container +
  node-agent kopia File System Backup (deployNodeAgent, uploaderType=kopia,
  snapshotsEnabled=false). BSL default -> bucket velero, s3ForcePathStyle.
- Two daily Schedules: `daily` backs up the whole cluster's resources+PVC data
  (defaultVolumesToFsBackup opt-out) EXCEPT monitoring+unifi; `daily-resources-only`
  captures monitoring+unifi RESOURCES only (Prometheus TSDB, Loki 72h logs, and the
  ~7.4GiB UniFi VM disk are reconstructable — excluded from volume backup).
- velero-cloud-credentials SOPS Secret (cloud INI key, ns velero).
- Own non-gating Flux tier clusters/base/backup.yaml (dependsOn platform-config,
  wait:false) + clusters/dev SOPS decryption patch — mirrors headlamp.

CNPG native Barman PITR (R-IDP-05 / R-STATE-03):
- keycloak-db, nextcloud-db, semaphore-db gain spec.backup.barmanObjectStore
  (s3://cnpg/<cluster>, endpointURL MinIO, gzip WAL+data, 30d retention) + a daily
  ScheduledBackup (6-field cron, immediate) each.
- Per-namespace scoped-key SOPS Secrets (<db>-backup-credentials), same-ns only.
- render.py: emit-waves now recognizes <svc>-db-backup-credentials.sops.yaml as a
  DB-wave sidecar (+ self-test); wave-0 kustomization regenerated.

Creds mirror platform/backup/minio-credentials.sops.yaml (scoped key k3s-backup),
SOPS round-trip verified with the cluster age key. gate: green.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-08-28 22:37:17 -05:00
jakman85andClaude Opus 4.8 af4f6c07ae T-0.1 Flux GitOps layout + dev/prod overlays + platform component skeletons (R-GITOPS-01/03, R-ENV-02, R-PLAT-01) 
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-08-16 00:08:31 -05:00