test(conformance): dr-restore actually asserts off-cluster restore (R-STATE-03)

Closes the LAST scaffold-only safety-critical requirement. The operator has attested DR
works (Velero -> Proxmox versitygw S3, restore demonstrated), but the L3 test carried the
right tag and layer label over a 5-line TODO body — so coverage counted it and reported
"0 safety-critical gaps" while the suite asserted nothing. That was test debt, not
capability debt. This makes the suite assert what the operator already demonstrated.

Three steps:
1. assert-backup-target-is-off-cluster (read-only): BSL must be phase Available (Velero
   itself reached the store), s3Url host is not localhost/*.svc/*.cluster.local, and the
   resolved IPv4 is none of — loopback, any Node Internal/ExternalIP, any Service
   ClusterIP, any Pod IP, inside any node podCIDR, inside any ServiceCIDR. A BSL that
   exists but never reaches Available FAILS (a live DR defect), it does not skip.
2. assert-backups-are-scheduled: R-STATE-03 says SCHEDULED off-cluster backups — Velero
   + BSL but zero Schedule in phase Enabled is a fail.
3. The drill: seed a nanosecond-unique canary onto a longhorn PVC in its own scratch
   namespace; Backup to Completed; assert a PodVolumeBackup for that volume is Completed
   WITH a non-empty status.snapshotID (the kopia snapshot of volume DATA reached the
   remote repo, not just resource JSON); ANNIHILATE the source (delete ns, poll until
   gone, force-delete a lingering PV, FAIL if the original PV survived — a survivor makes
   the restore prove nothing); Restore with namespaceMapping into a clean namespace;
   assert the restored PVC bound a DIFFERENT PV and that the canary bytes are identical.

Off-cluster proof = endpoint outside every cluster address + BSL Available + kopia
snapshotID present + every in-cluster copy destroyed + restore landing on a new PV. The
bytes could only have come from outside.

Skips GREEN on: no longhorn SC, Velero API absent, no BSL, Velero/node-agent not ready —
all three steps re-derive the same gates so it can never half-run. An RBAC denial is
deliberately NOT a skip: it fails loudly as a harness defect. Teardown is a step-level
`finally` that ALWAYS runs — deletes the Restore, issues a DeleteBackupRequest so objects
leave the bucket (2h ttl as backstop), deletes both scratch namespaces by name and by
label, releases leftover PVs whose claimRef is one of its own. Everything it destroys is
something it created.

conformance/incluster/rbac.yaml: authored the R-STATE-03 grants (velero.io backups/
restores/deletebackuprequests create+delete+read; BSL/schedules/podvolume* read-only;
persistentvolumes get/list/DELETE — the off-cluster proof requires destroying the PV).
Also two pre-existing gaps other tests already depended on: storageclasses get/list (the
skip gate of five tests) and apps deployments/daemonsets read. No write verbs on platform
controllers.

NOT ASSERTED, stated in the file header rather than implied: off-HOST (on dev the nodes
are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor
loss); no direct S3 client read (that would hand a privileged test identity the backup
credential for weaker evidence); real service data (Nextcloud/Gitea/CNPG drills stay
operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC.

status: scaffold-only 5 -> 4; safety-critical w/o executable L3 1 -> 0 (of 6); executable
35 -> 36. coverage unchanged at 40/51 (correct — coverage only asks whether a tagged test
exists). PROVEN stayed 0 / not-derivable: it did NOT flip to proven just because the body
is non-empty.

AUTHORED, NEVER EXECUTED — carries a conformance.home.k3s/never-executed annotation. No
cluster access from this workstation. To convert authored -> proven the operator must, IN
A MAINTENANCE WINDOW (it deletes namespaces and PVs and writes/deletes bucket objects):
apply the updated rbac.yaml, run `./conformance/run.sh --requirements R-STATE-03 --layer
l3`, confirm the log shows DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK and NOT
DR_RESTORE_SKIPPED_OK (green but proves nothing), confirm no leaked *-dr-src/*-dr-dst
namespaces or orphan PVs or *-dr-backup objects, and record the green run.

Serves: R-STATE-03, R-TEST-05.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

conformance/incluster/rbac.yaml

@@ -64,6 +64,22 @@ rules:
   - apiGroups: [""]
     resources: [pods/log]
     verbs: [get]                            # read fault/probe Job logs on failure
+  # ---- Storage-tier precondition gate (READ-ONLY). Every storage test self-skips on the
+  # absence of the `longhorn` StorageClass rather than failing: data-survival (R-STATE-01),
+  # node-loss (R-STATE-02), dr-restore (R-STATE-03), vm-data-survival / vm-lifecycle /
+  # vm-default-deny (R-PLAT-03). Without this read the gate cannot tell "absent" from
+  # "forbidden" and would skip for the wrong reason.
+  - apiGroups: [storage.k8s.io]
+    resources: [storageclasses]
+    verbs: [get, list]
+  # ---- Controller-readiness preconditions (READ-ONLY). dr-restore (R-STATE-03) skips cleanly
+  # unless the Velero Deployment and its node-agent DaemonSet (kopia File System Backup, the
+  # volume-DATA path) are actually ready — a backup path that is not up is an absent
+  # precondition, not a failure. No write verbs: tests never scale or patch a platform
+  # controller.
+  - apiGroups: [apps]
+    resources: [deployments, daemonsets]
+    verbs: [get, list, watch]
   # ---- Probe / fault Jobs (R-TLS-01/03 tls-probe; R-NET-06 tunnel stop/start)
   # `create` cannot be constrained by resourceNames, and the fault Job names are
   # fixed per-test (tunnel-stop/tunnel-start/tunnel-start-finally); scoped by the
@@ -103,10 +119,29 @@ rules:
   #   resources: [imagepolicies, imagerepositories]
   #   verbs: [get, patch]
   #
-  # ---- R-STATE-03 (DR restore): trigger a Velero restore
-  # - apiGroups: [velero.io]
-  #   resources: [restores, backups]
-  #   verbs: [create, get, list, watch]
+  # ---- R-STATE-03 (DR restore, tests/l3-fault-injection/dr-restore): the drill creates a
+  # Velero Backup of its OWN scratch namespace, destroys that namespace + its PV, creates a
+  # Restore into a fresh namespace, and (in `finally`) issues a DeleteBackupRequest so the
+  # objects it wrote are removed from the off-cluster bucket again. `create` cannot be
+  # constrained by resourceNames; the blast radius is bounded by the fact that every object
+  # the test creates or deletes is named `<its-own-chainsaw-namespace>-dr-*`.
+  - apiGroups: [velero.io]
+    resources: [backups, restores, deletebackuprequests]
+    verbs: [create, delete, get, list, watch]
+  # Read-only evidence: the BSL is the off-cluster target whose Available phase + endpoint the
+  # test asserts; schedules prove backups are SCHEDULED (not one-off); podvolumebackups /
+  # podvolumerestores prove the kopia snapshot of the volume DATA reached/returned from the
+  # remote repository, and are dumped on failure for diagnosis.
+  - apiGroups: [velero.io]
+    resources: [backupstoragelocations, schedules, podvolumebackups, podvolumerestores]
+    verbs: [get, list, watch]
+  # R-STATE-03 needs `delete` on the PV, not merely read: the proof that the restore came from
+  # OFF-cluster is that no in-cluster copy of the data survived. The test asserts its own PV is
+  # gone, and force-deletes it if a Retain policy or a stuck finalizer left it behind. It only
+  # ever names the PV bound to the PVC it created itself.
+  - apiGroups: [""]
+    resources: [persistentvolumes]
+    verbs: [get, list, delete]
   #
   # ---- R-GITOPS-03 (drift): mutate a reconciled resource + observe Flux revert
   # - apiGroups: [kustomize.toolkit.fluxcd.io]