The second requirement ever proven on this project, and the one the operator had attested
was working but the suite could not attest to. Test authored 2026-08-31 carrying a
`conformance.home.k3s/never-executed` annotation; executed for the first time tonight.
test dr-restore run 20260902T001846Z-fdc3265
req R-STATE-03 (safety-critical, DR) junit passed / 0 failed / 0 skipped
sentinel DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK proof proven
sha fdc3265 (clean tree) kube v1.36.4+k3s1
WHAT IT ACTUALLY PROVED, in three steps:
1. The backup target is genuinely OFF-CLUSTER. BSL velero/default phase=Available at
http://192.168.1.9:9000 — endpoint resolved and asserted to be none of: loopback, any
node Internal/ExternalIP, any Service ClusterIP, any Pod IP, inside any node podCIDR, or
inside any ServiceCIDR.
2. Backups are SCHEDULED, not merely possible — velero-daily and
velero-daily-resources-only, both phase=Enabled, last run 2026-09-01T03:00:27Z.
3. The drill: seed a nanosecond-unique canary onto a longhorn PVC; Velero Backup to
Completed with a kopia PodVolumeBackup carrying a real snapshotID (volume DATA reached
the remote repository, not just resource JSON); ANNIHILATE the source — delete the
namespace, poll until gone, force-delete the PV, FAIL if the original PV survived;
Restore into a clean namespace; assert the restored PVC bound a DIFFERENT PV and the
canary bytes are byte-identical.
The bytes could only have come from outside the cluster.
VALIDATES THE PARSER FIX FROM 57d4df8. This log contains SEVEN occurrences of
DR_RESTORE_SKIPPED_OK — every one of them inside a `=== COMMAND` script echo — and the pass
token twice in real `=== STDOUT`. Under the pre-fix scanner this genuine pass would have been
recorded as `skipped`, exactly as node-loss was. The section-parsing rule is doing real work,
not theoretical work.
CLEANUP VERIFIED against a pre-run baseline: Velero backups back to 7 (its own
chainsaw-*-dr-backup deleted via DeleteBackupRequest), 0 chainsaw backups, 0 restores, 0
orphan Released/Available PVs, both scratch namespaces gone, 13/13 Kustomizations Ready, 0
pods not Running. It wrote into and cleaned out of the SAME bucket and kopia repository the
live nightly schedule uses, and left it as it found it.
SEPARATE FINDING, cleaned up: `chainsaw-stable-worm` was still Active from 2026-08-25 —
empty residue from an earlier conformance session that never tore its namespace down. Deleted.
Worth knowing that Chainsaw namespace leaks can outlive a run by a week unnoticed.
NOT proven by this test, stated in its own header rather than implied: off-HOST DR (the dev
nodes are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor
loss); no direct S3 client read; real service data (Nextcloud/Gitea/CNPG drills remain
operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC.
Serves: R-STATE-03, R-TEST-03, R-TEST-05.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85andClaude Opus 5
380ea98a38