0 Fork 0

Files

T

jakman85 a177d62925 fix(conformance): busybox-compatible wget in oidc-sso-grafana probe (R-IDP-01)

busybox wget has no --max-redirect flag → it printed usage, so the probe never saw the
302 and R-IDP-01 falsely failed. Grafana verified live: /login/generic_oauth 302s to the
Keycloak homelab authorize endpoint. Use 'wget -S' (dumps the 302 Location before follow).

2026-08-24 23:09:06 -05:00

l1-rendering

Add container-workload-render L1 test (R-PLAT-02)

2026-08-24 22:22:04 -05:00

l2-behavioral

fix(conformance): busybox-compatible wget in oidc-sso-grafana probe (R-IDP-01)

2026-08-24 23:09:06 -05:00

l3-fault-injection

test(updates): complete hands-off-update L3 conformance (R-UPD-01/02/03)

2026-08-24 22:15:03 -05:00

meta/harness-selfcheck

fix(conformance): block-style chainsaw checks (flow-map broke on contains comma)

2026-08-17 17:02:05 -05:00

README.md

T-0.4 Chainsaw conformance harness + coverage + behavior scaffolds (R-TEST-01..06)

2026-08-16 00:08:32 -05:00

README.md

Conformance tests — layout & requirement-tagging convention

Every test is a Chainsaw Test in its own directory:

tests/
  l1-rendering/        L1 — ServiceDefinition -> rendered resources match golden
  l2-behavioral/       L2 — real runtime behavior (probes against a cluster)
  l3-fault-injection/  L3 — break something; assert fail-closed / no-data-loss
  meta/                harness self-checks (R-TEST-*), cluster-independent
    <test-name>/chainsaw-test.yaml

The layer a test belongs to is its subtree and its .../layer annotation; the coverage generator cross-checks the two.

Requirement-tagging convention (authoritative)

A test declares the requirement(s) it verifies three ways; the label is the source of truth, the annotations must agree with it (the coverage generator fails on drift):

metadata:
  name: default-deny
  labels:
    # ONE label per verified R-* ID. Machine-selectable + coverage source of truth.
    conformance.home.k3s/R-NET-01: "true"
  annotations:
    # Human-readable canonical list — MUST equal the label set.
    conformance.home.k3s/requirements: "R-NET-01"
    # Declared layer (L1|L2|L3|META) — must match the subtree.
    conformance.home.k3s/layer: "L2"
    # true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
    conformance.home.k3s/safety-critical: "false"
    # Given/When/Then intent, for the coverage report and reviewers.
    conformance.home.k3s/intent: "Given ... When ... Then ..."

Why a label per requirement (not a comma list): Kubernetes label values can't hold a comma-separated list, and chainsaw test --selector selects on label keys/values. One conformance.home.k3s/<R-ID>=true label per requirement makes both subset-selection and coverage extraction exact and robust.

How this drives the tooling

Subset runs (merge gate): run.sh --requirements R-NET-01,R-NET-06 resolves the R-IDs to the set of test dirs carrying those labels (union, deduped) and runs each once.
Coverage (lib/coverage.py): reads these labels from every chainsaw-test.yaml, maps each registry requirement -> its verifying test(s), and flags gaps + any label/annotation mismatch + tags not in the registry.
JUnit: results (real or skipped-scaffold) carry the R-IDs in classname + properties, so the JUnit itself stays traceable (R-TEST-04).

Authoring rules (R-TEST-01/02/06)

Behavior, not existence. Assert what the system does (a probe cannot connect), never that an object merely exists (a policy can exist and not enforce).
Given/When/Then structure with given-*/when-*/then-* step names.
Hermetic & self-cleaning. Rely on Chainsaw's per-test namespace + auto-teardown; put any out-of-namespace fixture cleanup in a finally block. A leaked fixture is a failure.
Bounded, deterministic waits. Use assert/error polling within the config timeouts; never sleep-and-hope.
Safety-critical => fault-injection. For a safety-critical requirement, actually break the thing and assert fail-closed / no-data-loss, with finally restoring state.
Never weaken a test to go green (R-TEST-06). If the spec can't be met, escalate to the PM — do not relax the assertion or the requirement.

Scaffolds in this tree carry TODO(T-x.y) markers where a fixture/assertion lands as a later roadmap phase delivers the capability under test.