jakman85andClaude Opus 5
14be9b68f6
Closes the LAST scaffold-only safety-critical requirement. The operator has attested DR works (Velero -> Proxmox versitygw S3, restore demonstrated), but the L3 test carried the right tag and layer label over a 5-line TODO body — so coverage counted it and reported "0 safety-critical gaps" while the suite asserted nothing. That was test debt, not capability debt. This makes the suite assert what the operator already demonstrated. Three steps: 1. assert-backup-target-is-off-cluster (read-only): BSL must be phase Available (Velero itself reached the store), s3Url host is not localhost/*.svc/*.cluster.local, and the resolved IPv4 is none of — loopback, any Node Internal/ExternalIP, any Service ClusterIP, any Pod IP, inside any node podCIDR, inside any ServiceCIDR. A BSL that exists but never reaches Available FAILS (a live DR defect), it does not skip. 2. assert-backups-are-scheduled: R-STATE-03 says SCHEDULED off-cluster backups — Velero + BSL but zero Schedule in phase Enabled is a fail. 3. The drill: seed a nanosecond-unique canary onto a longhorn PVC in its own scratch namespace; Backup to Completed; assert a PodVolumeBackup for that volume is Completed WITH a non-empty status.snapshotID (the kopia snapshot of volume DATA reached the remote repo, not just resource JSON); ANNIHILATE the source (delete ns, poll until gone, force-delete a lingering PV, FAIL if the original PV survived — a survivor makes the restore prove nothing); Restore with namespaceMapping into a clean namespace; assert the restored PVC bound a DIFFERENT PV and that the canary bytes are identical. Off-cluster proof = endpoint outside every cluster address + BSL Available + kopia snapshotID present + every in-cluster copy destroyed + restore landing on a new PV. The bytes could only have come from outside. Skips GREEN on: no longhorn SC, Velero API absent, no BSL, Velero/node-agent not ready — all three steps re-derive the same gates so it can never half-run. An RBAC denial is deliberately NOT a skip: it fails loudly as a harness defect. Teardown is a step-level `finally` that ALWAYS runs — deletes the Restore, issues a DeleteBackupRequest so objects leave the bucket (2h ttl as backstop), deletes both scratch namespaces by name and by label, releases leftover PVs whose claimRef is one of its own. Everything it destroys is something it created. conformance/incluster/rbac.yaml: authored the R-STATE-03 grants (velero.io backups/ restores/deletebackuprequests create+delete+read; BSL/schedules/podvolume* read-only; persistentvolumes get/list/DELETE — the off-cluster proof requires destroying the PV). Also two pre-existing gaps other tests already depended on: storageclasses get/list (the skip gate of five tests) and apps deployments/daemonsets read. No write verbs on platform controllers. NOT ASSERTED, stated in the file header rather than implied: off-HOST (on dev the nodes are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor loss); no direct S3 client read (that would hand a privileged test identity the backup credential for weaker evidence); real service data (Nextcloud/Gitea/CNPG drills stay operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC. status: scaffold-only 5 -> 4; safety-critical w/o executable L3 1 -> 0 (of 6); executable 35 -> 36. coverage unchanged at 40/51 (correct — coverage only asks whether a tagged test exists). PROVEN stayed 0 / not-derivable: it did NOT flip to proven just because the body is non-empty. AUTHORED, NEVER EXECUTED — carries a conformance.home.k3s/never-executed annotation. No cluster access from this workstation. To convert authored -> proven the operator must, IN A MAINTENANCE WINDOW (it deletes namespaces and PVs and writes/deletes bucket objects): apply the updated rbac.yaml, run `./conformance/run.sh --requirements R-STATE-03 --layer l3`, confirm the log shows DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK and NOT DR_RESTORE_SKIPPED_OK (green but proves nothing), confirm no leaked *-dr-src/*-dr-dst namespaces or orphan PVs or *-dr-backup objects, and record the green run. Serves: R-STATE-03, R-TEST-05. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
170 lines
8.5 KiB
YAML
170 lines
8.5 KiB
YAML
# =============================================================================
|
|
# Scoped test ServiceAccount + RBAC for the in-cluster Chainsaw conformance Job
|
|
# (ADR-0011; docs/14 "Where each layer runs"; R-SEC-02 least-privilege).
|
|
#
|
|
# SKELETON / PLACEHOLDER — RBAC is deliberately LEFT EMPTY here and authored
|
|
# least-privilege as each fault-injection capability actually lands (T-3.0+). The
|
|
# rule is: grant nothing until a committed test needs it, then grant exactly that
|
|
# verb/resource, with a comment naming the test that requires it. This SA is a
|
|
# PRIVILEGED TEST IDENTITY — it is its own audit item (R-SEC-02): it MUST NOT be
|
|
# reused by workloads, and every widening is reviewed.
|
|
#
|
|
# CI holds NO cluster credentials. CI only triggers the Job (T-6.1b via Gitea
|
|
# Actions); the Job runs under THIS ServiceAccount, so cluster access stays
|
|
# in-cluster and scoped (docs/14, R-SEC-02).
|
|
#
|
|
# Deployment: reconciled by Flux into the `conformance` namespace once the cluster
|
|
# exists (Phase 1+). Not applied in Phase 0 (no cluster).
|
|
# =============================================================================
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: conformance
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
# The test namespace itself is exempt from nothing: the platform default-deny
|
|
# baseline still applies; the Job opens only the egress it declares.
|
|
pod-security.kubernetes.io/enforce: restricted
|
|
---
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: conformance-runner
|
|
namespace: conformance
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
annotations:
|
|
conformance.home.k3s/audit: >-
|
|
PRIVILEGED TEST IDENTITY. Scoped least-privilege for fault-injection only.
|
|
Never reuse for workloads. Widen only alongside the test that needs it.
|
|
---
|
|
# ClusterRole — the union of what BEHAVIOR + FAULT tests need. Every rule is
|
|
# commented with the test/requirement that justifies it. Start empty; uncomment /
|
|
# add rules as the corresponding L2/L3 tests are delivered (do NOT pre-grant).
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRole
|
|
metadata:
|
|
name: conformance-runner
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
rules:
|
|
# ---- Hermetic fixtures (all L2/L3): create/delete throwaway test namespaces + fixtures
|
|
- apiGroups: [""]
|
|
resources: [namespaces]
|
|
# `patch` is required by R-NET-06 (vpn-killswitch): the test relabels its own
|
|
# ephemeral namespace to PodSecurity enforce=privileged so it can launch the
|
|
# node-pinned privileged fault Job. Scoped to the per-test namespace only.
|
|
verbs: [create, delete, get, list, watch, patch]
|
|
- apiGroups: [""]
|
|
resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
|
|
verbs: [create, delete, get, list, watch, patch]
|
|
- apiGroups: [""]
|
|
resources: [pods/exec]
|
|
verbs: [create] # probe exec for behavior assertions
|
|
- apiGroups: [""]
|
|
resources: [pods/log]
|
|
verbs: [get] # read fault/probe Job logs on failure
|
|
# ---- Storage-tier precondition gate (READ-ONLY). Every storage test self-skips on the
|
|
# absence of the `longhorn` StorageClass rather than failing: data-survival (R-STATE-01),
|
|
# node-loss (R-STATE-02), dr-restore (R-STATE-03), vm-data-survival / vm-lifecycle /
|
|
# vm-default-deny (R-PLAT-03). Without this read the gate cannot tell "absent" from
|
|
# "forbidden" and would skip for the wrong reason.
|
|
- apiGroups: [storage.k8s.io]
|
|
resources: [storageclasses]
|
|
verbs: [get, list]
|
|
# ---- Controller-readiness preconditions (READ-ONLY). dr-restore (R-STATE-03) skips cleanly
|
|
# unless the Velero Deployment and its node-agent DaemonSet (kopia File System Backup, the
|
|
# volume-DATA path) are actually ready — a backup path that is not up is an absent
|
|
# precondition, not a failure. No write verbs: tests never scale or patch a platform
|
|
# controller.
|
|
- apiGroups: [apps]
|
|
resources: [deployments, daemonsets]
|
|
verbs: [get, list, watch]
|
|
# ---- Probe / fault Jobs (R-TLS-01/03 tls-probe; R-NET-06 tunnel stop/start)
|
|
# `create` cannot be constrained by resourceNames, and the fault Job names are
|
|
# fixed per-test (tunnel-stop/tunnel-start/tunnel-start-finally); scoped by the
|
|
# in-namespace blast radius of the ephemeral test namespace.
|
|
- apiGroups: [batch]
|
|
resources: [jobs]
|
|
verbs: [create, delete, get, list, watch, patch]
|
|
#
|
|
# ---- R-NET-01/02 (default-deny, deny-undeclared): apply/patch/delete allow policies
|
|
# - apiGroups: [cilium.io]
|
|
# resources: [ciliumnetworkpolicies]
|
|
# verbs: [create, delete, get, list, patch]
|
|
#
|
|
# ---- R-NET-06 (VPN kill-switch, ADR-0015): the fault is a HOST systemd action
|
|
# (stop/start `vpn-egress-tunnel` on the egress-gateway node), NOT scaling an
|
|
# in-cluster gluetun Deployment (superseded design). It is driven by a privileged,
|
|
# hostPID, node-pinned Job (nsenter -t 1 -- systemctl ...). That power comes from
|
|
# PodSecurity (the test namespace relabeled to enforce=privileged) + the pod's
|
|
# securityContext, NOT from RBAC — the fault Job pod needs NO Kubernetes API
|
|
# rights. The runner only needs `jobs` create/delete + `namespaces` patch above.
|
|
# PREREQUISITE: if a cluster-wide Kyverno disallow-privileged / disallow-host-
|
|
# namespaces policy is enforced, the `chainsaw-conf-*` test namespaces MUST be in
|
|
# its exclusion set (documented, audited exception — R-SEC-02).
|
|
#
|
|
# ---- R-STATE-02 (node loss, tests/l3-fault-injection/node-loss): the fault is
|
|
# `kubectl cordon` + label of ONE node, reverted by `kubectl uncordon` + label removal
|
|
# in the test's `finally`. All three (cordon/uncordon/label) are a `patch` on the node
|
|
# object; `get`/`list` locate the victim + the finally cleanup selector. The test evicts
|
|
# only its OWN writer pod via `delete pod` (already granted above) — it deliberately does
|
|
# NOT `kubectl drain`, so no eviction rights and no reach over unrelated pods are needed.
|
|
- apiGroups: [""]
|
|
resources: [nodes]
|
|
verbs: [get, list, patch] # cordon/uncordon/label = patch spec.unschedulable + metadata.labels
|
|
#
|
|
# ---- R-UPD-02 (bad-update auto-rollback): inject a bad image tag via automation
|
|
# - apiGroups: [image.toolkit.fluxcd.io]
|
|
# resources: [imagepolicies, imagerepositories]
|
|
# verbs: [get, patch]
|
|
#
|
|
# ---- R-STATE-03 (DR restore, tests/l3-fault-injection/dr-restore): the drill creates a
|
|
# Velero Backup of its OWN scratch namespace, destroys that namespace + its PV, creates a
|
|
# Restore into a fresh namespace, and (in `finally`) issues a DeleteBackupRequest so the
|
|
# objects it wrote are removed from the off-cluster bucket again. `create` cannot be
|
|
# constrained by resourceNames; the blast radius is bounded by the fact that every object
|
|
# the test creates or deletes is named `<its-own-chainsaw-namespace>-dr-*`.
|
|
- apiGroups: [velero.io]
|
|
resources: [backups, restores, deletebackuprequests]
|
|
verbs: [create, delete, get, list, watch]
|
|
# Read-only evidence: the BSL is the off-cluster target whose Available phase + endpoint the
|
|
# test asserts; schedules prove backups are SCHEDULED (not one-off); podvolumebackups /
|
|
# podvolumerestores prove the kopia snapshot of the volume DATA reached/returned from the
|
|
# remote repository, and are dumped on failure for diagnosis.
|
|
- apiGroups: [velero.io]
|
|
resources: [backupstoragelocations, schedules, podvolumebackups, podvolumerestores]
|
|
verbs: [get, list, watch]
|
|
# R-STATE-03 needs `delete` on the PV, not merely read: the proof that the restore came from
|
|
# OFF-cluster is that no in-cluster copy of the data survived. The test asserts its own PV is
|
|
# gone, and force-deletes it if a Retain policy or a stuck finalizer left it behind. It only
|
|
# ever names the PV bound to the PVC it created itself.
|
|
- apiGroups: [""]
|
|
resources: [persistentvolumes]
|
|
verbs: [get, list, delete]
|
|
#
|
|
# ---- R-GITOPS-03 (drift): mutate a reconciled resource + observe Flux revert
|
|
# - apiGroups: [kustomize.toolkit.fluxcd.io]
|
|
# resources: [kustomizations]
|
|
# verbs: [get, list, watch] # observe only; do NOT grant reconcile suspension
|
|
#
|
|
# ---- R-PLAT-03 (VM lifecycle): create/start/stop/destroy VMs
|
|
# - apiGroups: [kubevirt.io]
|
|
# resources: [virtualmachines, virtualmachineinstances]
|
|
# verbs: [create, delete, get, list, patch]
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRoleBinding
|
|
metadata:
|
|
name: conformance-runner
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: conformance-runner
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: conformance-runner
|
|
namespace: conformance
|