Closes the LAST scaffold-only safety-critical requirement. The operator has attested DR
works (Velero -> Proxmox versitygw S3, restore demonstrated), but the L3 test carried the
right tag and layer label over a 5-line TODO body — so coverage counted it and reported
"0 safety-critical gaps" while the suite asserted nothing. That was test debt, not
capability debt. This makes the suite assert what the operator already demonstrated.
Three steps:
1. assert-backup-target-is-off-cluster (read-only): BSL must be phase Available (Velero
itself reached the store), s3Url host is not localhost/*.svc/*.cluster.local, and the
resolved IPv4 is none of — loopback, any Node Internal/ExternalIP, any Service
ClusterIP, any Pod IP, inside any node podCIDR, inside any ServiceCIDR. A BSL that
exists but never reaches Available FAILS (a live DR defect), it does not skip.
2. assert-backups-are-scheduled: R-STATE-03 says SCHEDULED off-cluster backups — Velero
+ BSL but zero Schedule in phase Enabled is a fail.
3. The drill: seed a nanosecond-unique canary onto a longhorn PVC in its own scratch
namespace; Backup to Completed; assert a PodVolumeBackup for that volume is Completed
WITH a non-empty status.snapshotID (the kopia snapshot of volume DATA reached the
remote repo, not just resource JSON); ANNIHILATE the source (delete ns, poll until
gone, force-delete a lingering PV, FAIL if the original PV survived — a survivor makes
the restore prove nothing); Restore with namespaceMapping into a clean namespace;
assert the restored PVC bound a DIFFERENT PV and that the canary bytes are identical.
Off-cluster proof = endpoint outside every cluster address + BSL Available + kopia
snapshotID present + every in-cluster copy destroyed + restore landing on a new PV. The
bytes could only have come from outside.
Skips GREEN on: no longhorn SC, Velero API absent, no BSL, Velero/node-agent not ready —
all three steps re-derive the same gates so it can never half-run. An RBAC denial is
deliberately NOT a skip: it fails loudly as a harness defect. Teardown is a step-level
`finally` that ALWAYS runs — deletes the Restore, issues a DeleteBackupRequest so objects
leave the bucket (2h ttl as backstop), deletes both scratch namespaces by name and by
label, releases leftover PVs whose claimRef is one of its own. Everything it destroys is
something it created.
conformance/incluster/rbac.yaml: authored the R-STATE-03 grants (velero.io backups/
restores/deletebackuprequests create+delete+read; BSL/schedules/podvolume* read-only;
persistentvolumes get/list/DELETE — the off-cluster proof requires destroying the PV).
Also two pre-existing gaps other tests already depended on: storageclasses get/list (the
skip gate of five tests) and apps deployments/daemonsets read. No write verbs on platform
controllers.
NOT ASSERTED, stated in the file header rather than implied: off-HOST (on dev the nodes
are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor
loss); no direct S3 client read (that would hand a privileged test identity the backup
credential for weaker evidence); real service data (Nextcloud/Gitea/CNPG drills stay
operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC.
status: scaffold-only 5 -> 4; safety-critical w/o executable L3 1 -> 0 (of 6); executable
35 -> 36. coverage unchanged at 40/51 (correct — coverage only asks whether a tagged test
exists). PROVEN stayed 0 / not-derivable: it did NOT flip to proven just because the body
is non-empty.
AUTHORED, NEVER EXECUTED — carries a conformance.home.k3s/never-executed annotation. No
cluster access from this workstation. To convert authored -> proven the operator must, IN
A MAINTENANCE WINDOW (it deletes namespaces and PVs and writes/deletes bucket objects):
apply the updated rbac.yaml, run `./conformance/run.sh --requirements R-STATE-03 --layer
l3`, confirm the log shows DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK and NOT
DR_RESTORE_SKIPPED_OK (green but proves nothing), confirm no leaked *-dr-src/*-dr-dst
namespaces or orphan PVs or *-dr-backup objects, and record the green run.
Serves: R-STATE-03, R-TEST-05.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85andClaude Opus 5
14be9b68f6