3 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	380ea98a38	feat(conformance): R-STATE-03 PROVEN — DR restore from off-cluster backup, executed ... The second requirement ever proven on this project, and the one the operator had attested was working but the suite could not attest to. Test authored 2026-08-31 carrying a `conformance.home.k3s/never-executed` annotation; executed for the first time tonight. test dr-restore run 20260902T001846Z-fdc3265 req R-STATE-03 (safety-critical, DR) junit passed / 0 failed / 0 skipped sentinel DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK proof proven sha fdc3265 (clean tree) kube v1.36.4+k3s1 WHAT IT ACTUALLY PROVED, in three steps: 1. The backup target is genuinely OFF-CLUSTER. BSL velero/default phase=Available at http://192.168.1.9:9000 — endpoint resolved and asserted to be none of: loopback, any node Internal/ExternalIP, any Service ClusterIP, any Pod IP, inside any node podCIDR, or inside any ServiceCIDR. 2. Backups are SCHEDULED, not merely possible — velero-daily and velero-daily-resources-only, both phase=Enabled, last run 2026-09-01T03:00:27Z. 3. The drill: seed a nanosecond-unique canary onto a longhorn PVC; Velero Backup to Completed with a kopia PodVolumeBackup carrying a real snapshotID (volume DATA reached the remote repository, not just resource JSON); ANNIHILATE the source — delete the namespace, poll until gone, force-delete the PV, FAIL if the original PV survived; Restore into a clean namespace; assert the restored PVC bound a DIFFERENT PV and the canary bytes are byte-identical. The bytes could only have come from outside the cluster. VALIDATES THE PARSER FIX FROM 57d4df8. This log contains SEVEN occurrences of DR_RESTORE_SKIPPED_OK — every one of them inside a `=== COMMAND` script echo — and the pass token twice in real `=== STDOUT`. Under the pre-fix scanner this genuine pass would have been recorded as `skipped`, exactly as node-loss was. The section-parsing rule is doing real work, not theoretical work. CLEANUP VERIFIED against a pre-run baseline: Velero backups back to 7 (its own chainsaw-*-dr-backup deleted via DeleteBackupRequest), 0 chainsaw backups, 0 restores, 0 orphan Released/Available PVs, both scratch namespaces gone, 13/13 Kustomizations Ready, 0 pods not Running. It wrote into and cleaned out of the SAME bucket and kopia repository the live nightly schedule uses, and left it as it found it. SEPARATE FINDING, cleaned up: `chainsaw-stable-worm` was still Active from 2026-08-25 — empty residue from an earlier conformance session that never tore its namespace down. Deleted. Worth knowing that Chainsaw namespace leaks can outlive a run by a week unnoticed. NOT proven by this test, stated in its own header rather than implied: off-HOST DR (the dev nodes are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor loss); no direct S3 client read; real service data (Nextcloud/Gitea/CNPG drills remain operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC. Serves: R-STATE-03, R-TEST-03, R-TEST-05. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 19:23:09 -05:00
jakman85	4cb5e7a8d4	feat(conformance): first PROVEN row — R-STATE-02 node-loss ... Records the first requirement this project has ever PROVEN, in the only sense its own definition of done accepts: a green, requirement-tagged, reproducible behaviour test, executed against a real cluster, with the evidence committed. test node-loss run 20260901T235117Z-e9a4b66 req R-STATE-02 (safety-critical) junit passed sentinel NODE_LOSS_VOLUME_AVAILABLE_OK proof proven sha 57d4df8 (clean tree) kube v1.36.4+k3s1 What actually ran: seed a 3-replica longhorn volume, cordon the node the writer landed on, evict only its own pod, assert the volume stays available and a reader on another node re-attaches and reads the data back. 131s. It deliberately does NOT drain, so kubelet and the etcd member stay up and quorum is never touched — which is what retired the old 'defer node-loss to a maintenance window' reason as unfounded rather than by fiat. Recorded against the clean tree of the preceding commit so git_sha resolves and the freshness check passes. The test was NOT re-run for this: the preserved run.log is the evidence, and re-running would have proved the parser rather than preserving the run. Serves: R-STATE-02, R-TEST-03, R-TEST-05.	2026-09-01 19:07:39 -05:00
jakman85andClaude Opus 5	e9a4b669d4	feat(conformance): committed run ledger — the only thing that can move PROVEN off zero ... status.py has always reported PROVEN: 0 for all 51 requirements, and named the fix in its own header: "a committed, machine-readable run ledger". There was none — _artifacts/ is gitignored, the root .gitignore kills *.junit.xml, and a stray JUnit was surfaced only as a quarantined mtime advisory. Built BEFORE the first real L3 execution so that run is captured rather than reconstructed from memory. THE TRAP THIS EXISTS TO AVOID. These tests exit 0 and Chainsaw records `passed` on a SELF-SKIP exactly as on a real run, and Chainsaw's JUnit carries no stdout — only properties. So committing the JUnit as proof would let DR_RESTORE_SKIPPED_OK be recorded as proof of R-STATE-03: authoritative-looking and wrong, which is worse than no ledger. The sentinel token is the ONLY thing distinguishing proven from skipped, and it lives solely in the run log. The ledger therefore records the LOG alongside the JUnit and derives proof from the sentinel, never from the JUnit result. Sentinel tokens are DERIVED, not hand-listed: record_run.py parses every contains($stdout,'TOKEN') out of each test's own chainsaw-test.yaml, so the list cannot drift from the tests. 23 of 35 tests declare no sentinel at all. decide() is the single place `proven` is emitted, ordered WORST-FIRST: JUnit failure/error -> failed JUnit <skipped/> -> junit-skipped ANY declared skip token in the log -> skipped (beats a green JUnit AND beats a pass token present in the same log) no sentinel declared -> proven ONLY for L0/L1 (whose JUnit is truthful on its own, since run_l1_offline.py maps exit 3 to <skipped/>); written as an explicit exempt-set so an unknown/blank layer fails CLOSED EVERY declared pass token present -> proven otherwise -> unknown-sentinel-absent Log attribution is a state machine over Chainsaw's per-test headers; a log with no recognisable header falls back to whole-log scanning restricted to GLOBALLY-UNIQUE tokens, so DNS_OK (owned by two tests) can never prove either. record_run.py exits 3 if the JUnit or the log is missing — nothing recorded, never green. STALENESS: a proven row is honoured only while `git merge-base --is-ancestor` holds between the requirement's last implementing/test change and the row's git_sha; otherwise it degrades to stale-proof. A -dirty sha, an unknown rev, or an uncommitted test path all fail ancestry the same way. A requirement reads proven only when EVERY verifying test does (worst-of), and the latest row per test wins — not best-ever. REFUSAL POSTURE PRESERVED: with an empty ledger the report is unchanged in intent — every requirement not-derivable, headline "PROVEN: 0 — NOT DERIVABLE". This replaces "not derivable" with real evidence only where evidence exists, never with optimism. --record is explicit, never implicit (a run that silently records itself is how a bad run becomes permanent false proof), is rejected for --status/--coverage-only/--lint, and re-execs under tee returning ${PIPESTATUS[0]} so recording can never soften a gate. It rm -f's a stale _artifacts/report.junit.xml first — a hazard hit live during development, where a stale JUnit got paired with a fresh log. Selftests: record_run.py 23 checks (incl. end-to-end Chainsaw-passed + NODE_LOSS_SKIPPED_OK -> proof "skipped"), status.py +16 against a throwaway git repo (self-skip -> skipped; proven-at-older-sha -> stale-proof; dirty/unresolvable -> stale-proof; worst-of; latest-row-wins; malformed lines dropped). No regressions: --coverage-only, --status, --lint, --strict-coverage and the merge-gate shape all rc=0 and unchanged; nothing in ci/ references status.py. Root .gitignore needed no change — conformance/runs/ is not itself excluded, so the deeper negations override the root's global *.junit.xml. Verified with git check-ignore. Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 18:49:01 -05:00