jakman85andClaude Opus 5
1252823d80
Operator ruling: "I care about the state of the repository and what the status of each spec is based on that state" — not a hand-maintained task board. This is that report. conformance/lib/status.py (stdlib-only, --selftest) sits beside lib/coverage.py and IMPORTS it rather than forking the registry/test parsers. Driven by a new `conformance/run.sh --status`; artifacts land in _artifacts/status/ alongside coverage and are git-ignored by the existing convention. Exit codes follow ci/lib.sh (0/1/3); --strict-coverage also applies the status hygiene gate. Signals derived per requirement: is there an implementing artifact; is it RECONCILED (walk resources: from clusters/dev, expand each Flux Kustomization's spec.path, recurse — so a commented-out resource correctly does not count); is there a tagged test and is its body executable or a TODO scaffold (TODO markers OUTRANK an executable: "true" annotation — ambiguity never upgrades a row). DELIBERATELY NOT DERIVED, and reported as such rather than defaulting to green: proof-of-pass (nothing tracked in git records a run result — _artifacts/ and results/ are git-ignored) and cluster liveness. "Reconciled" means Flux WOULD apply it, not that it is healthy. PROVEN is reported as "not derivable" for all 51. WHAT IT SHOWS THAT coverage.md DOES NOT. Coverage reports 37/51 covered, 0 safety-critical gaps. Three of those claims do not survive repo state: 1. 5 of the 37 "covered" requirements have only a TODO-body scaffold (R-STATE-03, R-OBS-02, R-OBS-04, R-GITOPS-03, R-GITOPS-02). Coverage counts a tagged file; it never looks inside it. 2. R-STATE-03 is SAFETY-CRITICAL and its L3 fault test asserts nothing. Coverage reports "0 safety-critical gaps" only because dr-restore/chainsaw-test.yaml exists and carries an L3 label. --strict now fails on this. 3. R-UPD-01/02/03 are NOT reconciled: platform/image-automation/ is commented out of platform/kustomization.yaml. Coverage shows them covered with executable L3 tests, which reads as delivered; automated updates are actually 0/3 wired+testable. Headline: 51 requirements | reconciled 20 (+12 partial) | executable test 32 | wired+testable 22 (+8 partial) | scaffold-only 5 | PROVEN 0 (not derivable). Also surfaces 18 requirements with no reconciled manifest citing their R-ID — listed as a traceability gap (R-TEST-04) rather than guessed either way. Annotating those manifests makes the report self-correct, so it acts as a forcing function instead of another hand-maintained map. Serves: R-TEST-01..06. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
146 lines
9.2 KiB
Markdown
146 lines
9.2 KiB
Markdown
# conformance/ — the home-k3s conformance harness (WS-TEST)
|
|
|
|
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
|
|
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
|
|
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
|
|
verifies (spec [`docs/14`](../../home-network-k3s/docs/14-testing-and-conformance.md),
|
|
[ADR-0010](../../home-network-k3s/decisions/adr-0010-conformance-testing.md),
|
|
[ADR-0011](../../home-network-k3s/decisions/adr-0011-test-tooling-kyverno-chainsaw.md)).
|
|
|
|
Tooling is fixed by ADR-0011: **Kyverno** for L0 (the CI policy gate + in-cluster
|
|
admission enforcer — owned by task T-0.5) and **Chainsaw** for **L1 rendering / L2
|
|
behavioral / L3 fault-injection** (this harness).
|
|
|
|
## The single command
|
|
|
|
```bash
|
|
./run.sh # full suite (skips L1/L2/L3 if no cluster)
|
|
./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
|
./run.sh --layer l2 # one layer: l1 | l2 | l3 | meta | all
|
|
./run.sh --coverage-only # regenerate the coverage report (no cluster)
|
|
./run.sh --status # derived requirement-STATUS report (no cluster)
|
|
./run.sh --lint # structural-validate test defs (no cluster)
|
|
./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
|
./run.sh --require-all # phase-exit gate: every requirement must be covered
|
|
```
|
|
|
|
**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
|
|
- `--strict-coverage` — **hygiene** (always-on, WS-TEST owns it): fails only on
|
|
tag/annotation drift or a tag absent from the registry. Uncovered future-phase
|
|
requirements are *visible gaps in the report*, not a harness failure.
|
|
- `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
|
|
those touched requirements are covered (a listed safety-critical one must have an L3 test).
|
|
- `--require-all` — **phase-exit gate**: every registry requirement must be covered.
|
|
- Documented out-of-scope gaps (e.g. `R-ENV-02` dev→prod promote and `R-GITOPS-01`
|
|
state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map)
|
|
are exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
|
|
`DOCUMENTED_GAPS` so the exemption is explicit and reviewable. (`R-STATE-02` node-loss
|
|
used to be here; it now has a real L3 test — `tests/l3-fault-injection/node-loss` — and
|
|
is covered, not exempted.)
|
|
|
|
### Coverage vs status — two different questions
|
|
|
|
`coverage.md` answers **"does a requirement-tagged test EXIST for R-x?"** (R-TEST-04).
|
|
`status.md` (`./run.sh --status`, generator `lib/status.py`) answers the operator's
|
|
question — **"what is the real state of each requirement given what is actually in the
|
|
repo?"** — deriving three orthogonal axes per requirement:
|
|
|
|
1. **implementation** — is there a tracked artifact naming the requirement?
|
|
2. **reconciliation** — is that artifact reachable by walking `resources:` from
|
|
`clusters/<env>/kustomization.yaml` and expanding every Flux `Kustomization`'s
|
|
`spec.path`? This is what separates **authored** from **actually reconciled**: a
|
|
commented-out `resources:` entry (e.g. `platform/image-automation`) is correctly
|
|
reported as in-git-but-not-on-any-cluster.
|
|
3. **verification** — is the tagged test EXECUTABLE, or a TODO scaffold? Coverage counts
|
|
both as "covered"; status does not. TODO markers in a body always outrank an
|
|
`executable: "true"` annotation, so ambiguity never upgrades a test.
|
|
|
|
**It deliberately refuses to derive proof-of-pass.** Nothing tracked in git records a test
|
|
result — `_artifacts/` and `results/` are git-ignored and DECISIONS-LOG.md prose is not a
|
|
signal — so every requirement's proof state is reported as `not-derivable` rather than
|
|
defaulting to green. A status report that overstates completion is worse than none
|
|
(`agents/definition-of-done.md`). Closing that gap needs a **committed, machine-readable
|
|
run ledger**; until then the strongest claim the report makes is *wired and testable*.
|
|
`--strict-coverage` also applies status's hygiene gate (registry↔spec drift,
|
|
annotation/body conflicts, a safety-critical requirement whose only L3 test is a scaffold).
|
|
Exit codes follow `ci/lib.sh`: `0` pass / `1` fail / `3` skipped — a skip is never green.
|
|
|
|
`run.sh` is the **only** entrypoint. Local runs and the in-cluster CI Chainsaw Job call
|
|
the *same* script with the same config, so results are identical.
|
|
|
|
### Artifacts (always emitted, under `_artifacts/`, git-ignored)
|
|
| Path | What | Consumer |
|
|
|---|---|---|
|
|
| `_artifacts/report.junit.xml` | JUnit — Chainsaw results, or a **skipped-scaffold** shim when no cluster | CI gate |
|
|
| `_artifacts/coverage/coverage.md` | Human requirement-coverage report | PR reviewers / logs |
|
|
| `_artifacts/coverage/coverage.json` | Machine coverage: `{requirement: [tests]}`, gaps, safety gaps | CI gate |
|
|
| `_artifacts/status/status.md` | Human requirement-STATUS report derived from repo state | operator / PM |
|
|
| `_artifacts/status/status.json` | Machine status: implementation / reconciliation / verification per requirement | tooling |
|
|
|
|
## Reproducibility contract (R-TEST-02 — what the harness guarantees)
|
|
- **Single command.** `./run.sh` (or a tagged/layer subset) runs everything and emits the
|
|
artifacts above — locally and in CI.
|
|
- **Hermetic.** Each Chainsaw test provisions its own namespace + fixtures/probes and tears
|
|
them down (`config/chainsaw-config.yaml` auto-creates/auto-deletes the namespace; tests
|
|
add `finally` for anything out-of-namespace). It never relies on pre-existing state.
|
|
- **Deterministic.** Tool versions pinned in [`versions.env`](versions.env); eventual
|
|
consistency handled by **bounded polling with explicit timeouts** (config), never sleeps
|
|
or wall-clock/ordering assumptions.
|
|
- **Isolated & parallel-safe.** Unique generated namespaces + per-requirement labels; tests
|
|
run concurrently without cross-talk.
|
|
- **Machine-readable & requirement-tagged.** JUnit + coverage, every result carrying its
|
|
`R-*` tags (convention in [`tests/README.md`](tests/README.md)).
|
|
- **Self-cleaning.** Fixtures removed on pass **or** fail; a leaked namespace/fixture is
|
|
itself a failure (`skipDelete: false`, enforced delete/cleanup timeouts).
|
|
- **Versioned with the code.** Tests live next to what they verify and change only when the
|
|
human-owned spec they encode changes — **never** weakened to go green (R-TEST-06).
|
|
|
|
## Layout
|
|
```
|
|
run.sh single-command runner (the entrypoint)
|
|
versions.env pinned toolchain (Chainsaw/kubectl/Kyverno) — determinism
|
|
config/chainsaw-config.yaml shared execution contract (timeouts, ns isolation, JUnit)
|
|
requirements/registry.yaml mirror of EVERY spec R-* ID (coverage enumerates from here)
|
|
lib/
|
|
common.sh tagging convention + test discovery/selection helpers
|
|
coverage.py requirement-coverage generator (stdlib only)
|
|
status.py requirement-STATUS generator (stdlib only; --selftest)
|
|
emit_skipped_junit.py Phase-0 skipped-JUnit shim (no cluster)
|
|
tests/
|
|
l1-rendering/ l2-behavioral/ l3-fault-injection/ meta/ (see tests/README.md)
|
|
incluster/
|
|
chainsaw-job.yaml rbac.yaml in-cluster L2/L3 Job + scoped test SA (ADR-0011)
|
|
```
|
|
|
|
## How CI invokes it
|
|
CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
|
|
|
|
1. **No-cluster stage (every PR, fast):** `./run.sh --lint` and `./run.sh --coverage-only
|
|
--strict-coverage --requirements <IDs the change touches>`. Validates every test parses
|
|
+ is tagged, regenerates coverage, **fails the PR** on tag/annotation drift (hygiene)
|
|
and on any *touched* requirement lacking a test (merge gate). (L0 Kyverno CLI runs
|
|
alongside — task T-0.5.)
|
|
2. **Cluster stage (L2/L3):** CI does **not** hold a kubeconfig. It **triggers the
|
|
in-cluster Chainsaw Job** ([`incluster/`](incluster/README.md)), which runs `./run.sh`
|
|
under the scoped `conformance-runner` ServiceAccount and writes JUnit + coverage to a
|
|
results volume CI collects.
|
|
- **Merge gate:** the Job runs `--requirements <IDs the change touches>`; that subset
|
|
runs green **and** those requirements are covered to merge (R-TEST-03).
|
|
- **Phase-exit gate:** the Job runs `--layer all --strict-coverage`; add `--require-all`
|
|
only at the milestone where the full requirement set is expected covered. The phase
|
|
suite must be green before the phase's exit criteria are met.
|
|
|
|
## Phase 0 status (no cluster yet)
|
|
No cluster exists, so L1/L2/L3 **cannot execute**. The behavior tests are **valid
|
|
scaffolds** (steps + assertions authored, tagged, `TODO(T-x.y)` where a later phase
|
|
delivers the fixture). In this phase `run.sh`:
|
|
- always generates the **coverage report** (cluster-independent), and
|
|
- emits a **skipped** JUnit listing every scaffold (so the gate is green and the artifact
|
|
shape is real), and
|
|
- the **`meta/harness-selfcheck`** checks (reproducible output, JUnit+coverage emitted,
|
|
everything tagged, zero safety-critical gaps) are what `run.sh --strict-coverage`
|
|
actually enforces today.
|
|
|
|
As later phases deliver capabilities, each scaffold's `TODO`s are filled and the test flips
|
|
from *skipped* to *executing green* — landing coverage without re-architecting the harness.
|