jakman85andClaude Opus 4.8
9d8e03892f
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
112 lines
6.8 KiB
Markdown
112 lines
6.8 KiB
Markdown
# conformance/ — the home-k3s conformance harness (WS-TEST)
|
|
|
|
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
|
|
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
|
|
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
|
|
verifies (spec [`docs/14`](../../home-network-k3s/docs/14-testing-and-conformance.md),
|
|
[ADR-0010](../../home-network-k3s/decisions/adr-0010-conformance-testing.md),
|
|
[ADR-0011](../../home-network-k3s/decisions/adr-0011-test-tooling-kyverno-chainsaw.md)).
|
|
|
|
Tooling is fixed by ADR-0011: **Kyverno** for L0 (the CI policy gate + in-cluster
|
|
admission enforcer — owned by task T-0.5) and **Chainsaw** for **L1 rendering / L2
|
|
behavioral / L3 fault-injection** (this harness).
|
|
|
|
## The single command
|
|
|
|
```bash
|
|
./run.sh # full suite (skips L1/L2/L3 if no cluster)
|
|
./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
|
./run.sh --layer l2 # one layer: l1 | l2 | l3 | meta | all
|
|
./run.sh --coverage-only # regenerate the coverage report (no cluster)
|
|
./run.sh --lint # structural-validate test defs (no cluster)
|
|
./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
|
./run.sh --require-all # phase-exit gate: every requirement must be covered
|
|
```
|
|
|
|
**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
|
|
- `--strict-coverage` — **hygiene** (always-on, WS-TEST owns it): fails only on
|
|
tag/annotation drift or a tag absent from the registry. Uncovered future-phase
|
|
requirements are *visible gaps in the report*, not a harness failure.
|
|
- `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
|
|
those touched requirements are covered (a listed safety-critical one must have an L3 test).
|
|
- `--require-all` — **phase-exit gate**: every registry requirement must be covered.
|
|
- Documented out-of-scope gaps (e.g. `R-STATE-02` node-loss, owned by WS-STORAGE) are
|
|
exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
|
|
`DOCUMENTED_GAPS` so the exemption is explicit and reviewable.
|
|
|
|
`run.sh` is the **only** entrypoint. Local runs and the in-cluster CI Chainsaw Job call
|
|
the *same* script with the same config, so results are identical.
|
|
|
|
### Artifacts (always emitted, under `_artifacts/`, git-ignored)
|
|
| Path | What | Consumer |
|
|
|---|---|---|
|
|
| `_artifacts/report.junit.xml` | JUnit — Chainsaw results, or a **skipped-scaffold** shim when no cluster | CI gate |
|
|
| `_artifacts/coverage/coverage.md` | Human requirement-coverage report | PR reviewers / logs |
|
|
| `_artifacts/coverage/coverage.json` | Machine coverage: `{requirement: [tests]}`, gaps, safety gaps | CI gate |
|
|
|
|
## Reproducibility contract (R-TEST-02 — what the harness guarantees)
|
|
- **Single command.** `./run.sh` (or a tagged/layer subset) runs everything and emits the
|
|
artifacts above — locally and in CI.
|
|
- **Hermetic.** Each Chainsaw test provisions its own namespace + fixtures/probes and tears
|
|
them down (`config/chainsaw-config.yaml` auto-creates/auto-deletes the namespace; tests
|
|
add `finally` for anything out-of-namespace). It never relies on pre-existing state.
|
|
- **Deterministic.** Tool versions pinned in [`versions.env`](versions.env); eventual
|
|
consistency handled by **bounded polling with explicit timeouts** (config), never sleeps
|
|
or wall-clock/ordering assumptions.
|
|
- **Isolated & parallel-safe.** Unique generated namespaces + per-requirement labels; tests
|
|
run concurrently without cross-talk.
|
|
- **Machine-readable & requirement-tagged.** JUnit + coverage, every result carrying its
|
|
`R-*` tags (convention in [`tests/README.md`](tests/README.md)).
|
|
- **Self-cleaning.** Fixtures removed on pass **or** fail; a leaked namespace/fixture is
|
|
itself a failure (`skipDelete: false`, enforced delete/cleanup timeouts).
|
|
- **Versioned with the code.** Tests live next to what they verify and change only when the
|
|
human-owned spec they encode changes — **never** weakened to go green (R-TEST-06).
|
|
|
|
## Layout
|
|
```
|
|
run.sh single-command runner (the entrypoint)
|
|
versions.env pinned toolchain (Chainsaw/kubectl/Kyverno) — determinism
|
|
config/chainsaw-config.yaml shared execution contract (timeouts, ns isolation, JUnit)
|
|
requirements/registry.yaml mirror of EVERY spec R-* ID (coverage enumerates from here)
|
|
lib/
|
|
common.sh tagging convention + test discovery/selection helpers
|
|
coverage.py requirement-coverage generator (stdlib only)
|
|
emit_skipped_junit.py Phase-0 skipped-JUnit shim (no cluster)
|
|
tests/
|
|
l1-rendering/ l2-behavioral/ l3-fault-injection/ meta/ (see tests/README.md)
|
|
incluster/
|
|
chainsaw-job.yaml rbac.yaml in-cluster L2/L3 Job + scoped test SA (ADR-0011)
|
|
```
|
|
|
|
## How CI invokes it
|
|
CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
|
|
|
|
1. **No-cluster stage (every PR, fast):** `./run.sh --lint` and `./run.sh --coverage-only
|
|
--strict-coverage --requirements <IDs the change touches>`. Validates every test parses
|
|
+ is tagged, regenerates coverage, **fails the PR** on tag/annotation drift (hygiene)
|
|
and on any *touched* requirement lacking a test (merge gate). (L0 Kyverno CLI runs
|
|
alongside — task T-0.5.)
|
|
2. **Cluster stage (L2/L3):** CI does **not** hold a kubeconfig. It **triggers the
|
|
in-cluster Chainsaw Job** ([`incluster/`](incluster/README.md)), which runs `./run.sh`
|
|
under the scoped `conformance-runner` ServiceAccount and writes JUnit + coverage to a
|
|
results volume CI collects.
|
|
- **Merge gate:** the Job runs `--requirements <IDs the change touches>`; that subset
|
|
runs green **and** those requirements are covered to merge (R-TEST-03).
|
|
- **Phase-exit gate:** the Job runs `--layer all --strict-coverage`; add `--require-all`
|
|
only at the milestone where the full requirement set is expected covered. The phase
|
|
suite must be green before the phase's exit criteria are met.
|
|
|
|
## Phase 0 status (no cluster yet)
|
|
No cluster exists, so L1/L2/L3 **cannot execute**. The behavior tests are **valid
|
|
scaffolds** (steps + assertions authored, tagged, `TODO(T-x.y)` where a later phase
|
|
delivers the fixture). In this phase `run.sh`:
|
|
- always generates the **coverage report** (cluster-independent), and
|
|
- emits a **skipped** JUnit listing every scaffold (so the gate is green and the artifact
|
|
shape is real), and
|
|
- the **`meta/harness-selfcheck`** checks (reproducible output, JUnit+coverage emitted,
|
|
everything tagged, zero safety-critical gaps) are what `run.sh --strict-coverage`
|
|
actually enforces today.
|
|
|
|
As later phases deliver capabilities, each scaffold's `TODO`s are filled and the test flips
|
|
from *skipped* to *executing green* — landing coverage without re-architecting the harness.
|