jakman85andClaude Opus 5
e9a4b669d4
status.py has always reported PROVEN: 0 for all 51 requirements, and named the fix in its
own header: "a committed, machine-readable run ledger". There was none — _artifacts/ is
gitignored, the root .gitignore kills *.junit.xml, and a stray JUnit was surfaced only as
a quarantined mtime advisory. Built BEFORE the first real L3 execution so that run is
captured rather than reconstructed from memory.
THE TRAP THIS EXISTS TO AVOID. These tests exit 0 and Chainsaw records `passed` on a
SELF-SKIP exactly as on a real run, and Chainsaw's JUnit carries no stdout — only
properties. So committing the JUnit as proof would let DR_RESTORE_SKIPPED_OK be recorded
as proof of R-STATE-03: authoritative-looking and wrong, which is worse than no ledger.
The sentinel token is the ONLY thing distinguishing proven from skipped, and it lives
solely in the run log. The ledger therefore records the LOG alongside the JUnit and
derives proof from the sentinel, never from the JUnit result.
Sentinel tokens are DERIVED, not hand-listed: record_run.py parses every
contains($stdout,'TOKEN') out of each test's own chainsaw-test.yaml, so the list cannot
drift from the tests. 23 of 35 tests declare no sentinel at all.
decide() is the single place `proven` is emitted, ordered WORST-FIRST:
JUnit failure/error -> failed
JUnit <skipped/> -> junit-skipped
ANY declared skip token in the log -> skipped (beats a green JUnit AND beats a pass
token present in the same log)
no sentinel declared -> proven ONLY for L0/L1 (whose JUnit is truthful on its own, since
run_l1_offline.py maps exit 3 to <skipped/>); written as an
explicit exempt-set so an unknown/blank layer fails CLOSED
EVERY declared pass token present -> proven
otherwise -> unknown-sentinel-absent
Log attribution is a state machine over Chainsaw's per-test headers; a log with no
recognisable header falls back to whole-log scanning restricted to GLOBALLY-UNIQUE tokens,
so DNS_OK (owned by two tests) can never prove either. record_run.py exits 3 if the JUnit
or the log is missing — nothing recorded, never green.
STALENESS: a proven row is honoured only while `git merge-base --is-ancestor` holds
between the requirement's last implementing/test change and the row's git_sha; otherwise
it degrades to stale-proof. A -dirty sha, an unknown rev, or an uncommitted test path all
fail ancestry the same way. A requirement reads proven only when EVERY verifying test
does (worst-of), and the latest row per test wins — not best-ever.
REFUSAL POSTURE PRESERVED: with an empty ledger the report is unchanged in intent — every
requirement not-derivable, headline "PROVEN: 0 — NOT DERIVABLE". This replaces "not
derivable" with real evidence only where evidence exists, never with optimism.
--record is explicit, never implicit (a run that silently records itself is how a bad run
becomes permanent false proof), is rejected for --status/--coverage-only/--lint, and
re-execs under tee returning ${PIPESTATUS[0]} so recording can never soften a gate. It
rm -f's a stale _artifacts/report.junit.xml first — a hazard hit live during development,
where a stale JUnit got paired with a fresh log.
Selftests: record_run.py 23 checks (incl. end-to-end Chainsaw-passed + NODE_LOSS_SKIPPED_OK
-> proof "skipped"), status.py +16 against a throwaway git repo (self-skip -> skipped;
proven-at-older-sha -> stale-proof; dirty/unresolvable -> stale-proof; worst-of;
latest-row-wins; malformed lines dropped). No regressions: --coverage-only, --status,
--lint, --strict-coverage and the merge-gate shape all rc=0 and unchanged; nothing in ci/
references status.py.
Root .gitignore needed no change — conformance/runs/ is not itself excluded, so the deeper
negations override the root's global *.junit.xml. Verified with git check-ignore.
Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
223 lines
14 KiB
Markdown
223 lines
14 KiB
Markdown
# conformance/ — the home-k3s conformance harness (WS-TEST)
|
|
|
|
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
|
|
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
|
|
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
|
|
verifies (spec [`docs/14`](../../home-network-k3s/docs/14-testing-and-conformance.md),
|
|
[ADR-0010](../../home-network-k3s/decisions/adr-0010-conformance-testing.md),
|
|
[ADR-0011](../../home-network-k3s/decisions/adr-0011-test-tooling-kyverno-chainsaw.md)).
|
|
|
|
Tooling is fixed by ADR-0011: **Kyverno** for L0 (the CI policy gate + in-cluster
|
|
admission enforcer — owned by task T-0.5) and **Chainsaw** for **L1 rendering / L2
|
|
behavioral / L3 fault-injection** (this harness).
|
|
|
|
## The single command
|
|
|
|
```bash
|
|
./run.sh # full suite (skips L1/L2/L3 if no cluster)
|
|
./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
|
./run.sh --layer l2 # one layer: l1 | l2 | l3 | meta | all
|
|
./run.sh --coverage-only # regenerate the coverage report (no cluster)
|
|
./run.sh --status # derived requirement-STATUS report (no cluster)
|
|
./run.sh --lint # structural-validate test defs (no cluster)
|
|
./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
|
./run.sh --require-all # phase-exit gate: every requirement must be covered
|
|
./run.sh --record # run AND capture the evidence (see "Proof of pass")
|
|
```
|
|
|
|
**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
|
|
- `--strict-coverage` — **hygiene** (always-on, WS-TEST owns it): fails only on
|
|
tag/annotation drift or a tag absent from the registry. Uncovered future-phase
|
|
requirements are *visible gaps in the report*, not a harness failure.
|
|
- `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
|
|
those touched requirements are covered (a listed safety-critical one must have an L3 test).
|
|
- `--require-all` — **phase-exit gate**: every registry requirement must be covered.
|
|
- Documented out-of-scope gaps (e.g. `R-ENV-02` dev→prod promote and `R-GITOPS-01`
|
|
state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map)
|
|
are exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
|
|
`DOCUMENTED_GAPS` so the exemption is explicit and reviewable. (`R-STATE-02` node-loss
|
|
used to be here; it now has a real L3 test — `tests/l3-fault-injection/node-loss` — and
|
|
is covered, not exempted.)
|
|
|
|
### Coverage vs status — two different questions
|
|
|
|
`coverage.md` answers **"does a requirement-tagged test EXIST for R-x?"** (R-TEST-04).
|
|
`status.md` (`./run.sh --status`, generator `lib/status.py`) answers the operator's
|
|
question — **"what is the real state of each requirement given what is actually in the
|
|
repo?"** — deriving three orthogonal axes per requirement:
|
|
|
|
1. **implementation** — is there a tracked artifact naming the requirement?
|
|
2. **reconciliation** — is that artifact reachable by walking `resources:` from
|
|
`clusters/<env>/kustomization.yaml` and expanding every Flux `Kustomization`'s
|
|
`spec.path`? This is what separates **authored** from **actually reconciled**: a
|
|
commented-out `resources:` entry (e.g. `platform/image-automation`) is correctly
|
|
reported as in-git-but-not-on-any-cluster.
|
|
3. **verification** — is the tagged test EXECUTABLE, or a TODO scaffold? Coverage counts
|
|
both as "covered"; status does not. TODO markers in a body always outrank an
|
|
`executable: "true"` annotation, so ambiguity never upgrades a test.
|
|
|
|
**It deliberately refuses to derive proof-of-pass.** Nothing tracked in git records a test
|
|
result — `_artifacts/` and `results/` are git-ignored and DECISIONS-LOG.md prose is not a
|
|
signal — so every requirement's proof state is reported as `not-derivable` rather than
|
|
defaulting to green. A status report that overstates completion is worse than none
|
|
(`agents/definition-of-done.md`). Closing that gap needs a **committed, machine-readable
|
|
run ledger**; until then the strongest claim the report makes is *wired and testable*.
|
|
`--strict-coverage` also applies status's hygiene gate (registry↔spec drift,
|
|
annotation/body conflicts, a safety-critical requirement whose only L3 test is a scaffold).
|
|
Exit codes follow `ci/lib.sh`: `0` pass / `1` fail / `3` skipped — a skip is never green.
|
|
|
|
`run.sh` is the **only** entrypoint. Local runs and the in-cluster CI Chainsaw Job call
|
|
the *same* script with the same config, so results are identical.
|
|
|
|
### Artifacts (always emitted, under `_artifacts/`, git-ignored)
|
|
| Path | What | Consumer |
|
|
|---|---|---|
|
|
| `_artifacts/report.junit.xml` | JUnit — Chainsaw results, or a **skipped-scaffold** shim when no cluster | CI gate |
|
|
| `_artifacts/coverage/coverage.md` | Human requirement-coverage report | PR reviewers / logs |
|
|
| `_artifacts/coverage/coverage.json` | Machine coverage: `{requirement: [tests]}`, gaps, safety gaps | CI gate |
|
|
| `_artifacts/status/status.md` | Human requirement-STATUS report derived from repo state | operator / PM |
|
|
| `_artifacts/status/status.json` | Machine status: implementation / reconciliation / verification per requirement | tooling |
|
|
|
|
## Reproducibility contract (R-TEST-02 — what the harness guarantees)
|
|
- **Single command.** `./run.sh` (or a tagged/layer subset) runs everything and emits the
|
|
artifacts above — locally and in CI.
|
|
- **Hermetic.** Each Chainsaw test provisions its own namespace + fixtures/probes and tears
|
|
them down (`config/chainsaw-config.yaml` auto-creates/auto-deletes the namespace; tests
|
|
add `finally` for anything out-of-namespace). It never relies on pre-existing state.
|
|
- **Deterministic.** Tool versions pinned in [`versions.env`](versions.env); eventual
|
|
consistency handled by **bounded polling with explicit timeouts** (config), never sleeps
|
|
or wall-clock/ordering assumptions.
|
|
- **Isolated & parallel-safe.** Unique generated namespaces + per-requirement labels; tests
|
|
run concurrently without cross-talk.
|
|
- **Machine-readable & requirement-tagged.** JUnit + coverage, every result carrying its
|
|
`R-*` tags (convention in [`tests/README.md`](tests/README.md)).
|
|
- **Self-cleaning.** Fixtures removed on pass **or** fail; a leaked namespace/fixture is
|
|
itself a failure (`skipDelete: false`, enforced delete/cleanup timeouts).
|
|
- **Versioned with the code.** Tests live next to what they verify and change only when the
|
|
human-owned spec they encode changes — **never** weakened to go green (R-TEST-06).
|
|
|
|
## Layout
|
|
```
|
|
run.sh single-command runner (the entrypoint)
|
|
versions.env pinned toolchain (Chainsaw/kubectl/Kyverno) — determinism
|
|
config/chainsaw-config.yaml shared execution contract (timeouts, ns isolation, JUnit)
|
|
requirements/registry.yaml mirror of EVERY spec R-* ID (coverage enumerates from here)
|
|
lib/
|
|
common.sh tagging convention + test discovery/selection helpers
|
|
coverage.py requirement-coverage generator (stdlib only)
|
|
status.py requirement-STATUS generator (stdlib only; --selftest)
|
|
record_run.py run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
|
|
emit_skipped_junit.py Phase-0 skipped-JUnit shim (no cluster)
|
|
runs/
|
|
ledger.jsonl TRACKED append-only proof-of-pass ledger (one row per
|
|
testcase per run) — the only source status.py trusts
|
|
<UTC>-<short-sha>/ TRACKED per-run evidence: run.log + report.junit.xml
|
|
tests/
|
|
l1-rendering/ l2-behavioral/ l3-fault-injection/ meta/ (see tests/README.md)
|
|
incluster/
|
|
chainsaw-job.yaml rbac.yaml in-cluster L2/L3 Job + scoped test SA (ADR-0011)
|
|
```
|
|
|
|
## Proof of pass — the run ledger (`runs/`)
|
|
|
|
`--status` used to report **PROVEN: 0** for all 51 requirements, and said so honestly:
|
|
proof-of-pass was not recoverable from repo state. `_artifacts/` is gitignored and the
|
|
root `.gitignore` kills `*.junit.xml` globally, so no run result ever reached git; the
|
|
only record of a green run was DECISIONS-LOG.md prose. `runs/` is the fix `lib/status.py`
|
|
asked for — **a committed, machine-readable run ledger**.
|
|
|
|
```bash
|
|
./run.sh --record # runs normally, and captures the evidence
|
|
# -> conformance/runs/<UTC>-<short-sha>/run.log (tee'd, carries the sentinels)
|
|
# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml (that run's JUnit)
|
|
# -> conformance/runs/ledger.jsonl (one appended row per testcase)
|
|
python3 lib/record_run.py --run-dir runs/<id> [--dry-run] # or record an existing run dir
|
|
python3 lib/record_run.py --selftest # deterministic, no cluster
|
|
```
|
|
|
|
**Recording is explicit, never implicit.** Plain `./run.sh` writes no ledger row: a run
|
|
that silently records itself is how a bad run becomes a permanent false proof. **Commit
|
|
the run dir and the ledger** — an uncommitted ledger proves nothing.
|
|
|
|
### Why a committed JUnit alone would be WORSE than nothing
|
|
|
|
The L2/L3 tests are runnable and **self-skipping**: when a precondition is missing (no
|
|
`longhorn` StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token
|
|
and `exit 0`, and their `check:` accepts either outcome —
|
|
|
|
```yaml
|
|
(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
|
|
|| contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
|
|
```
|
|
|
|
So **a self-skip exits 0 and Chainsaw records `passed` exactly as a real run does**, and
|
|
Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`.
|
|
A committed JUnit on its own would therefore record `DR_RESTORE_SKIPPED_OK` as *proof of
|
|
R-STATE-03*: fake green, and worse than no ledger because it would look authoritative.
|
|
|
|
**The sentinel token in `run.log` is the only thing that separates proven from skipped.**
|
|
`record_run.py` derives each test's tokens from its own `chainsaw-test.yaml` (every
|
|
`contains($stdout, 'TOKEN')`; `*_SKIP*_OK` is a skip token, the last non-skip token is the
|
|
verdict token), so the table below is never hand-maintained:
|
|
|
|
| Test | pass token (proof) | skip token (never proof) |
|
|
|---|---|---|
|
|
| `l3/node-loss` | `NODE_LOSS_VOLUME_AVAILABLE_OK` | `NODE_LOSS_SKIPPED_OK` |
|
|
| `l3/dr-restore` | `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK` (+ `OFF_CLUSTER_TARGET_OK`, `BACKUPS_ARE_SCHEDULED_OK`) | `DR_RESTORE_SKIPPED_OK`, `DR_SCHEDULE_SKIPPED_OK` |
|
|
| `l3/data-survival` | `DATA_SURVIVED_OK` | `DATA_SURVIVAL_SKIPPED_OK` |
|
|
| `l3/vm-data-survival` | `VM_DATA_SURVIVED_OK` | `VM_DATA_SURVIVAL_SKIPPED_OK` |
|
|
| `l3/hands-off-update` | `HANDS_OFF_UPDATE_OK` | `HANDS_OFF_UPDATE_SKIPPED_OK` |
|
|
| `l2/vm-lifecycle` | `VM_LIFECYCLE_OK` | `VM_LIFECYCLE_SKIPPED_OK` |
|
|
| `l2/vm-default-deny` | `VM_DEFAULT_DENY_OK` | `VM_DEFAULT_DENY_SKIPPED_OK` |
|
|
|
|
A row reaches `proof: "proven"` on **one** path: JUnit `passed`, **every** declared pass
|
|
token present in that test's log lines, and **no** skip token anywhere in them. A skip
|
|
token beats a pass token (worst wins). Anything else is `skipped`, `failed`,
|
|
`junit-skipped`, `unknown-no-sentinel-declared` or `unknown-sentinel-absent` — all
|
|
non-green. L0/L1 need no sentinel because `lib/run_l1_offline.py` already renders their
|
|
skips as JUnit `<skipped/>`, never as a green pass.
|
|
|
|
### Staleness — a proof is a claim about a tree, not a badge
|
|
|
|
`--status` honours a `proven` row only while the run's `git_sha` is a **descendant of the
|
|
last commit touching that requirement's implementing manifests or its verifying test**.
|
|
Otherwise the row degrades to **`stale-proof`**, which is not green. A run captured from a
|
|
dirty working tree is recorded as `<sha>-dirty`, which no ancestry check can satisfy — so
|
|
it never counts. Re-run with `--record` to refresh.
|
|
|
|
A requirement reads `proven` only when **every** verifying test does; one skipped, failed
|
|
or unrecorded test drags the whole requirement down. With an empty ledger `--status`
|
|
behaves exactly as before: every requirement `not-derivable`, headline **PROVEN 0**.
|
|
|
|
## How CI invokes it
|
|
CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
|
|
|
|
1. **No-cluster stage (every PR, fast):** `./run.sh --lint` and `./run.sh --coverage-only
|
|
--strict-coverage --requirements <IDs the change touches>`. Validates every test parses
|
|
+ is tagged, regenerates coverage, **fails the PR** on tag/annotation drift (hygiene)
|
|
and on any *touched* requirement lacking a test (merge gate). (L0 Kyverno CLI runs
|
|
alongside — task T-0.5.)
|
|
2. **Cluster stage (L2/L3):** CI does **not** hold a kubeconfig. It **triggers the
|
|
in-cluster Chainsaw Job** ([`incluster/`](incluster/README.md)), which runs `./run.sh`
|
|
under the scoped `conformance-runner` ServiceAccount and writes JUnit + coverage to a
|
|
results volume CI collects.
|
|
- **Merge gate:** the Job runs `--requirements <IDs the change touches>`; that subset
|
|
runs green **and** those requirements are covered to merge (R-TEST-03).
|
|
- **Phase-exit gate:** the Job runs `--layer all --strict-coverage`; add `--require-all`
|
|
only at the milestone where the full requirement set is expected covered. The phase
|
|
suite must be green before the phase's exit criteria are met.
|
|
|
|
## Phase 0 status (no cluster yet)
|
|
No cluster exists, so L1/L2/L3 **cannot execute**. The behavior tests are **valid
|
|
scaffolds** (steps + assertions authored, tagged, `TODO(T-x.y)` where a later phase
|
|
delivers the fixture). In this phase `run.sh`:
|
|
- always generates the **coverage report** (cluster-independent), and
|
|
- emits a **skipped** JUnit listing every scaffold (so the gate is green and the artifact
|
|
shape is real), and
|
|
- the **`meta/harness-selfcheck`** checks (reproducible output, JUnit+coverage emitted,
|
|
everything tagged, zero safety-critical gaps) are what `run.sh --strict-coverage`
|
|
actually enforces today.
|
|
|
|
As later phases deliver capabilities, each scaffold's `TODO`s are filled and the test flips
|
|
from *skipped* to *executing green* — landing coverage without re-architecting the harness.
|