T-0.4 Chainsaw conformance harness + coverage + behavior scaffolds (R-TEST-01..06)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed
1 parent
a778d923b6
commit
9d8e03892f
31 files changed
+2445
-2
No files matched your search
@@ -0,0 +1,7 @@
|
||||
# Generated artifacts — regenerated by run.sh on every run; never committed.
|
||||
# CI collects these from the workspace / results PVC; they are inputs to the gate,
|
||||
# not source. (Root .gitignore also covers *.junit.xml and coverage/.)
|
||||
_artifacts/
|
||||
|
||||
# Downloaded pinned tool binaries (run.sh may fetch chainsaw/kubectl here).
|
||||
.bin/
|
||||
+110
-2
@@ -1,3 +1,111 @@
|
||||
# conformance
|
||||
# conformance/ — the home-k3s conformance harness (WS-TEST)
|
||||
|
||||
_Placeholder — populated in Phase 0. See ../home-network-k3s (read-only spec)._
|
||||
This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
|
||||
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
|
||||
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
|
||||
verifies (spec [`docs/14`](../../home-network-k3s/docs/14-testing-and-conformance.md),
|
||||
[ADR-0010](../../home-network-k3s/decisions/adr-0010-conformance-testing.md),
|
||||
[ADR-0011](../../home-network-k3s/decisions/adr-0011-test-tooling-kyverno-chainsaw.md)).
|
||||
|
||||
Tooling is fixed by ADR-0011: **Kyverno** for L0 (the CI policy gate + in-cluster
|
||||
admission enforcer — owned by task T-0.5) and **Chainsaw** for **L1 rendering / L2
|
||||
behavioral / L3 fault-injection** (this harness).
|
||||
|
||||
## The single command
|
||||
|
||||
```bash
|
||||
./run.sh # full suite (skips L1/L2/L3 if no cluster)
|
||||
./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
||||
./run.sh --layer l2 # one layer: l1 | l2 | l3 | meta | all
|
||||
./run.sh --coverage-only # regenerate the coverage report (no cluster)
|
||||
./run.sh --lint # structural-validate test defs (no cluster)
|
||||
./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
./run.sh --require-all # phase-exit gate: every requirement must be covered
|
||||
```
|
||||
|
||||
**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
|
||||
- `--strict-coverage` — **hygiene** (always-on, WS-TEST owns it): fails only on
|
||||
tag/annotation drift or a tag absent from the registry. Uncovered future-phase
|
||||
requirements are *visible gaps in the report*, not a harness failure.
|
||||
- `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
|
||||
those touched requirements are covered (a listed safety-critical one must have an L3 test).
|
||||
- `--require-all` — **phase-exit gate**: every registry requirement must be covered.
|
||||
- Documented out-of-scope gaps (e.g. `R-STATE-02` node-loss, owned by WS-STORAGE) are
|
||||
exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
|
||||
`DOCUMENTED_GAPS` so the exemption is explicit and reviewable.
|
||||
|
||||
`run.sh` is the **only** entrypoint. Local runs and the in-cluster CI Chainsaw Job call
|
||||
the *same* script with the same config, so results are identical.
|
||||
|
||||
### Artifacts (always emitted, under `_artifacts/`, git-ignored)
|
||||
| Path | What | Consumer |
|
||||
|---|---|---|
|
||||
| `_artifacts/report.junit.xml` | JUnit — Chainsaw results, or a **skipped-scaffold** shim when no cluster | CI gate |
|
||||
| `_artifacts/coverage/coverage.md` | Human requirement-coverage report | PR reviewers / logs |
|
||||
| `_artifacts/coverage/coverage.json` | Machine coverage: `{requirement: [tests]}`, gaps, safety gaps | CI gate |
|
||||
|
||||
## Reproducibility contract (R-TEST-02 — what the harness guarantees)
|
||||
- **Single command.** `./run.sh` (or a tagged/layer subset) runs everything and emits the
|
||||
artifacts above — locally and in CI.
|
||||
- **Hermetic.** Each Chainsaw test provisions its own namespace + fixtures/probes and tears
|
||||
them down (`config/chainsaw-config.yaml` auto-creates/auto-deletes the namespace; tests
|
||||
add `finally` for anything out-of-namespace). It never relies on pre-existing state.
|
||||
- **Deterministic.** Tool versions pinned in [`versions.env`](versions.env); eventual
|
||||
consistency handled by **bounded polling with explicit timeouts** (config), never sleeps
|
||||
or wall-clock/ordering assumptions.
|
||||
- **Isolated & parallel-safe.** Unique generated namespaces + per-requirement labels; tests
|
||||
run concurrently without cross-talk.
|
||||
- **Machine-readable & requirement-tagged.** JUnit + coverage, every result carrying its
|
||||
`R-*` tags (convention in [`tests/README.md`](tests/README.md)).
|
||||
- **Self-cleaning.** Fixtures removed on pass **or** fail; a leaked namespace/fixture is
|
||||
itself a failure (`skipDelete: false`, enforced delete/cleanup timeouts).
|
||||
- **Versioned with the code.** Tests live next to what they verify and change only when the
|
||||
human-owned spec they encode changes — **never** weakened to go green (R-TEST-06).
|
||||
|
||||
## Layout
|
||||
```
|
||||
run.sh single-command runner (the entrypoint)
|
||||
versions.env pinned toolchain (Chainsaw/kubectl/Kyverno) — determinism
|
||||
config/chainsaw-config.yaml shared execution contract (timeouts, ns isolation, JUnit)
|
||||
requirements/registry.yaml mirror of EVERY spec R-* ID (coverage enumerates from here)
|
||||
lib/
|
||||
common.sh tagging convention + test discovery/selection helpers
|
||||
coverage.py requirement-coverage generator (stdlib only)
|
||||
emit_skipped_junit.py Phase-0 skipped-JUnit shim (no cluster)
|
||||
tests/
|
||||
l1-rendering/ l2-behavioral/ l3-fault-injection/ meta/ (see tests/README.md)
|
||||
incluster/
|
||||
chainsaw-job.yaml rbac.yaml in-cluster L2/L3 Job + scoped test SA (ADR-0011)
|
||||
```
|
||||
|
||||
## How CI invokes it
|
||||
CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
|
||||
|
||||
1. **No-cluster stage (every PR, fast):** `./run.sh --lint` and `./run.sh --coverage-only
|
||||
--strict-coverage --requirements <IDs the change touches>`. Validates every test parses
|
||||
+ is tagged, regenerates coverage, **fails the PR** on tag/annotation drift (hygiene)
|
||||
and on any *touched* requirement lacking a test (merge gate). (L0 Kyverno CLI runs
|
||||
alongside — task T-0.5.)
|
||||
2. **Cluster stage (L2/L3):** CI does **not** hold a kubeconfig. It **triggers the
|
||||
in-cluster Chainsaw Job** ([`incluster/`](incluster/README.md)), which runs `./run.sh`
|
||||
under the scoped `conformance-runner` ServiceAccount and writes JUnit + coverage to a
|
||||
results volume CI collects.
|
||||
- **Merge gate:** the Job runs `--requirements <IDs the change touches>`; that subset
|
||||
runs green **and** those requirements are covered to merge (R-TEST-03).
|
||||
- **Phase-exit gate:** the Job runs `--layer all --strict-coverage`; add `--require-all`
|
||||
only at the milestone where the full requirement set is expected covered. The phase
|
||||
suite must be green before the phase's exit criteria are met.
|
||||
|
||||
## Phase 0 status (no cluster yet)
|
||||
No cluster exists, so L1/L2/L3 **cannot execute**. The behavior tests are **valid
|
||||
scaffolds** (steps + assertions authored, tagged, `TODO(T-x.y)` where a later phase
|
||||
delivers the fixture). In this phase `run.sh`:
|
||||
- always generates the **coverage report** (cluster-independent), and
|
||||
- emits a **skipped** JUnit listing every scaffold (so the gate is green and the artifact
|
||||
shape is real), and
|
||||
- the **`meta/harness-selfcheck`** checks (reproducible output, JUnit+coverage emitted,
|
||||
everything tagged, zero safety-critical gaps) are what `run.sh --strict-coverage`
|
||||
actually enforces today.
|
||||
|
||||
As later phases deliver capabilities, each scaffold's `TODO`s are filled and the test flips
|
||||
from *skipped* to *executing green* — landing coverage without re-architecting the harness.
|
||||
@@ -0,0 +1,49 @@
|
||||
# Chainsaw Configuration for the home-k3s conformance suite (ADR-0011).
|
||||
# Passed to every invocation via `chainsaw test --config config/chainsaw-config.yaml`
|
||||
# so local runs and the in-cluster CI Job behave identically (R-TEST-02).
|
||||
#
|
||||
# Reproducibility knobs live here, NOT in individual tests, so the whole suite shares
|
||||
# one deterministic execution contract:
|
||||
# * bounded timeouts (no sleep-and-hope; eventual consistency via polled assert/error)
|
||||
# * per-test ephemeral namespace, auto-created and auto-deleted (hermetic, isolated)
|
||||
# * a leaked fixture is a failure: del/cleanup timeouts are enforced, skipDelete=false
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha2
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: home-k3s-conformance
|
||||
spec:
|
||||
# -- Deterministic bounded waits. Eventual consistency is handled by polling within
|
||||
# these windows (assert/error retry until deadline), never by fixed sleeps.
|
||||
timeouts:
|
||||
apply: 90s
|
||||
assert: 180s
|
||||
error: 60s
|
||||
delete: 120s
|
||||
cleanup: 120s
|
||||
exec: 180s
|
||||
|
||||
# -- Isolation & parallel-safety: each test gets its own generated namespace so tests
|
||||
# never collide and can run concurrently.
|
||||
namespace:
|
||||
# Unique per test run; prefix makes leaked namespaces obvious in `kubectl get ns`.
|
||||
template:
|
||||
metadata:
|
||||
generateName: chainsaw-conf-
|
||||
|
||||
# -- Concurrency. Tests are hermetic, so they parallelize safely; keep modest to bound
|
||||
# cluster load and keep fault-injection tests from contending.
|
||||
execution:
|
||||
parallel: 4
|
||||
# Fail the whole run if any test leaks a namespace/fixture (reproducibility contract).
|
||||
failFast: false
|
||||
|
||||
# -- Self-cleaning: never skip delete; deletion is part of the pass criteria.
|
||||
cleanup:
|
||||
skipDelete: false
|
||||
|
||||
# -- Machine-readable output. run.sh normalizes the emitted file to
|
||||
# _artifacts/report.junit.xml and layers the requirement-coverage report on top.
|
||||
report:
|
||||
format: JUNIT-TEST
|
||||
name: chainsaw-report
|
||||
path: ../_artifacts
|
||||
@@ -0,0 +1,32 @@
|
||||
# In-cluster conformance Job (L2/L3)
|
||||
|
||||
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
|
||||
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
|
||||
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
|
||||
credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
|
||||
|
||||
## Files
|
||||
- `rbac.yaml` — the `conformance` namespace, the `conformance-runner` ServiceAccount, and
|
||||
a **least-privilege ClusterRole skeleton**. Rules start **empty** and are added — each
|
||||
commented with the test/requirement that needs it — only as the matching fault-injection
|
||||
test lands. This is a **privileged test identity**: never reused by workloads, its own
|
||||
audit item.
|
||||
- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
|
||||
`--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
|
||||
exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`.
|
||||
|
||||
## Flow
|
||||
1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
|
||||
trigger with narrow rights: create Job + read its logs/status — *not* a broad
|
||||
kubeconfig).
|
||||
2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs
|
||||
`./run.sh` under `conformance-runner`.
|
||||
3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` +
|
||||
`coverage/` to the results volume (`/workspace/conformance/_artifacts`).
|
||||
4. CI collects those artifacts (results volume or the terminating log stream, which echoes
|
||||
their paths) and enforces the merge / phase-exit gate on them.
|
||||
|
||||
## Phase 0
|
||||
No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux
|
||||
from Phase 1+ and first exercised when L2/L3 can run (T-3.0 onward). In Phase 0 the same
|
||||
`run.sh` runs in CI's no-cluster stage and emits a **skipped** JUnit + the coverage report.
|
||||
@@ -0,0 +1,74 @@
|
||||
# =============================================================================
|
||||
# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14)
|
||||
#
|
||||
# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO
|
||||
# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner`
|
||||
# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that
|
||||
# CI collects, and enforces the merge / phase-exit gate on the outcome.
|
||||
#
|
||||
# Two invocation shapes (CI sets args):
|
||||
# * MERGE GATE: ./run.sh --requirements <touched R-IDs> (subset)
|
||||
# * PHASE EXIT: ./run.sh --layer all (full suite)
|
||||
#
|
||||
# Results artifact mechanism (delegated to WS-TEST by docs/14 open items):
|
||||
# Chosen = a shared results emptyDir + a terminating log stream, with an
|
||||
# OPTIONAL results PVC for durability across CI collection. For Phase 0 there is
|
||||
# no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised
|
||||
# once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the
|
||||
# results volume (or scrapes the pod log, which echoes their paths).
|
||||
#
|
||||
# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE.
|
||||
# =============================================================================
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
generateName: conformance-run-
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
spec:
|
||||
backoffLimit: 0 # deterministic: a failed conformance run is a failure, no retry
|
||||
ttlSecondsAfterFinished: 3600
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
spec:
|
||||
serviceAccountName: conformance-runner # scoped test identity (rbac.yaml)
|
||||
restartPolicy: Never
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
seccompProfile: { type: RuntimeDefault }
|
||||
containers:
|
||||
- name: chainsaw
|
||||
# PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12).
|
||||
image: ghcr.io/kyverno/chainsaw:v0.2.12
|
||||
workingDir: /workspace/conformance
|
||||
# The suite repo is delivered into /workspace (git clone init-container or
|
||||
# a Flux-populated volume). run.sh is the single entrypoint (identical to local).
|
||||
command: ["./run.sh"]
|
||||
# CI overrides args per invocation shape:
|
||||
# ["--requirements", "R-NET-01,R-NET-06"] (merge gate) OR
|
||||
# ["--layer", "all", "--strict-coverage"] (phase exit)
|
||||
args: ["--layer", "all", "--strict-coverage"]
|
||||
env:
|
||||
- name: CONF_ROOT
|
||||
value: /workspace/conformance
|
||||
volumeMounts:
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
- { name: results, mountPath: /workspace/conformance/_artifacts }
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities: { drop: ["ALL"] }
|
||||
runAsNonRoot: true
|
||||
volumes:
|
||||
# Suite source: populated by an init-container git clone of the impl repo at
|
||||
# the commit under test, OR a Flux-synced source volume. (Init container
|
||||
# omitted from this skeleton; wire in T-6.1b.)
|
||||
- name: workspace
|
||||
emptyDir: {}
|
||||
# Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir
|
||||
# for a results PVC if durability beyond pod lifetime is required.
|
||||
- name: results
|
||||
emptyDir: {}
|
||||
@@ -0,0 +1,114 @@
|
||||
# =============================================================================
|
||||
# Scoped test ServiceAccount + RBAC for the in-cluster Chainsaw conformance Job
|
||||
# (ADR-0011; docs/14 "Where each layer runs"; R-SEC-02 least-privilege).
|
||||
#
|
||||
# SKELETON / PLACEHOLDER — RBAC is deliberately LEFT EMPTY here and authored
|
||||
# least-privilege as each fault-injection capability actually lands (T-3.0+). The
|
||||
# rule is: grant nothing until a committed test needs it, then grant exactly that
|
||||
# verb/resource, with a comment naming the test that requires it. This SA is a
|
||||
# PRIVILEGED TEST IDENTITY — it is its own audit item (R-SEC-02): it MUST NOT be
|
||||
# reused by workloads, and every widening is reviewed.
|
||||
#
|
||||
# CI holds NO cluster credentials. CI only triggers the Job (T-6.1b via Gitea
|
||||
# Actions); the Job runs under THIS ServiceAccount, so cluster access stays
|
||||
# in-cluster and scoped (docs/14, R-SEC-02).
|
||||
#
|
||||
# Deployment: reconciled by Flux into the `conformance` namespace once the cluster
|
||||
# exists (Phase 1+). Not applied in Phase 0 (no cluster).
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
# The test namespace itself is exempt from nothing: the platform default-deny
|
||||
# baseline still applies; the Job opens only the egress it declares.
|
||||
pod-security.kubernetes.io/enforce: restricted
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: conformance-runner
|
||||
namespace: conformance
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
annotations:
|
||||
conformance.home.k3s/audit: >-
|
||||
PRIVILEGED TEST IDENTITY. Scoped least-privilege for fault-injection only.
|
||||
Never reuse for workloads. Widen only alongside the test that needs it.
|
||||
---
|
||||
# ClusterRole — the union of what BEHAVIOR + FAULT tests need. Every rule is
|
||||
# commented with the test/requirement that justifies it. Start empty; uncomment /
|
||||
# add rules as the corresponding L2/L3 tests are delivered (do NOT pre-grant).
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: conformance-runner
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
rules: []
|
||||
# ---- Hermetic fixtures (all L2/L3): create/delete throwaway test namespaces + fixtures
|
||||
# - apiGroups: [""]
|
||||
# resources: [namespaces]
|
||||
# verbs: [create, delete, get, list, watch]
|
||||
# - apiGroups: [""]
|
||||
# resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
|
||||
# verbs: [create, delete, get, list, watch, patch]
|
||||
# - apiGroups: [""]
|
||||
# resources: [pods/exec]
|
||||
# verbs: [create] # probe exec for behavior assertions
|
||||
#
|
||||
# ---- R-NET-01/02 (default-deny, deny-undeclared): apply/patch/delete allow policies
|
||||
# - apiGroups: [cilium.io]
|
||||
# resources: [ciliumnetworkpolicies]
|
||||
# verbs: [create, delete, get, list, patch]
|
||||
#
|
||||
# ---- R-NET-06 (VPN kill-switch): force the shared Gluetun gateway down/up
|
||||
# - apiGroups: [apps]
|
||||
# resources: [deployments, deployments/scale]
|
||||
# resourceNames: [gluetun-gateway] # SCOPE to the gateway only
|
||||
# verbs: [get, patch, update]
|
||||
#
|
||||
# ---- R-STATE-02 (node loss): cordon/drain/uncordon a node (NOT in WS-TEST scope yet)
|
||||
# - apiGroups: [""]
|
||||
# resources: [nodes]
|
||||
# verbs: [get, list, patch] # cordon = patch spec.unschedulable
|
||||
# - apiGroups: [""]
|
||||
# resources: [pods/eviction]
|
||||
# verbs: [create] # drain
|
||||
#
|
||||
# ---- R-UPD-02 (bad-update auto-rollback): inject a bad image tag via automation
|
||||
# - apiGroups: [image.toolkit.fluxcd.io]
|
||||
# resources: [imagepolicies, imagerepositories]
|
||||
# verbs: [get, patch]
|
||||
#
|
||||
# ---- R-STATE-03 (DR restore): trigger a Velero restore
|
||||
# - apiGroups: [velero.io]
|
||||
# resources: [restores, backups]
|
||||
# verbs: [create, get, list, watch]
|
||||
#
|
||||
# ---- R-GITOPS-03 (drift): mutate a reconciled resource + observe Flux revert
|
||||
# - apiGroups: [kustomize.toolkit.fluxcd.io]
|
||||
# resources: [kustomizations]
|
||||
# verbs: [get, list, watch] # observe only; do NOT grant reconcile suspension
|
||||
#
|
||||
# ---- R-PLAT-03 (VM lifecycle): create/start/stop/destroy VMs
|
||||
# - apiGroups: [kubevirt.io]
|
||||
# resources: [virtualmachines, virtualmachineinstances]
|
||||
# verbs: [create, delete, get, list, patch]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: conformance-runner
|
||||
labels:
|
||||
app.kubernetes.io/part-of: conformance
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: conformance-runner
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: conformance-runner
|
||||
namespace: conformance
|
||||
@@ -0,0 +1,75 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared helpers for the conformance harness. Sourced by run.sh; not executed directly.
|
||||
# shellcheck shell=bash
|
||||
|
||||
# --- Requirement-tagging convention (single source of truth) --------------------------
|
||||
# A Chainsaw Test declares the requirement(s) it verifies with a per-requirement label:
|
||||
#
|
||||
# metadata:
|
||||
# labels:
|
||||
# conformance.home.k3s/R-NET-01: "true" # one label per verified R-* ID
|
||||
# annotations:
|
||||
# conformance.home.k3s/requirements: "R-NET-01, R-NET-02" # human summary
|
||||
# conformance.home.k3s/layer: "L2"
|
||||
# conformance.home.k3s/safety-critical: "false"
|
||||
#
|
||||
# The LABEL is authoritative and does double duty:
|
||||
# * `chainsaw test --selector conformance.home.k3s/R-NET-01=true` runs the subset.
|
||||
# * lib/coverage.py reads the same labels to map requirement -> verifying test(s).
|
||||
# The annotation is a convenience for humans/reports and MUST agree with the labels
|
||||
# (coverage.py flags a mismatch). This is the convention referenced in README.md.
|
||||
export CONF_LABEL_PREFIX="conformance.home.k3s"
|
||||
|
||||
# Absolute path to the conformance root (dir containing run.sh). Set by run.sh.
|
||||
: "${CONF_ROOT:?CONF_ROOT must be set before sourcing common.sh}"
|
||||
|
||||
# --- Test discovery -------------------------------------------------------------------
|
||||
# Every leaf test is a directory containing a `chainsaw-test.yaml`.
|
||||
conf_all_test_files() {
|
||||
find "${CONF_ROOT}/tests" -type f -name 'chainsaw-test.yaml' 2>/dev/null | sort
|
||||
}
|
||||
|
||||
# Directories of tests that carry ANY of the given requirement labels (space-separated
|
||||
# R-IDs). Used to translate `--requirements R-NET-01,R-NET-06` into a set of test dirs
|
||||
# so we run each matching test exactly once (union, deduplicated).
|
||||
conf_dirs_for_requirements() {
|
||||
local reqs="$1" f dir rid matched
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
matched=0
|
||||
for rid in $reqs; do
|
||||
# Match a label line: conformance.home.k3s/R-NET-01: "true"
|
||||
if grep -Eq "^[[:space:]]*${CONF_LABEL_PREFIX}/${rid}:[[:space:]]*[\"']?true" "$f"; then
|
||||
matched=1; break
|
||||
fi
|
||||
done
|
||||
[ "$matched" = 1 ] && dirname "$f"
|
||||
done < <(conf_all_test_files) | sort -u
|
||||
}
|
||||
|
||||
# Directories for a given layer subtree: l1 | l2 | l3 | meta | all
|
||||
conf_dirs_for_layer() {
|
||||
local layer="$1" sub
|
||||
case "$layer" in
|
||||
all) find "${CONF_ROOT}/tests" -type f -name 'chainsaw-test.yaml' -exec dirname {} \; | sort -u ;;
|
||||
l1) sub="l1-rendering" ;;
|
||||
l2) sub="l2-behavioral" ;;
|
||||
l3) sub="l3-fault-injection" ;;
|
||||
meta) sub="meta" ;;
|
||||
*) echo "unknown layer: $layer" >&2; return 2 ;;
|
||||
esac
|
||||
if [ "$layer" != all ]; then
|
||||
find "${CONF_ROOT}/tests/${sub}" -type f -name 'chainsaw-test.yaml' -exec dirname {} \; 2>/dev/null | sort -u
|
||||
fi
|
||||
}
|
||||
|
||||
# Normalize a comma/space separated requirement list to space-separated, uppercased.
|
||||
conf_normalize_reqs() {
|
||||
echo "$1" | tr ',' ' ' | tr '[:lower:]' '[:upper:]' | xargs
|
||||
}
|
||||
|
||||
# Is a live cluster reachable? (Preflight for L1/L2/L3 execution.)
|
||||
conf_cluster_reachable() {
|
||||
command -v kubectl >/dev/null 2>&1 || return 1
|
||||
kubectl cluster-info >/dev/null 2>&1
|
||||
}
|
||||
Executable
+319
@@ -0,0 +1,319 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Requirement-coverage report generator (R-TEST-04).
|
||||
|
||||
Maps EVERY R-* requirement in the authoritative registry to the Chainsaw test(s) that
|
||||
verify it, and flags requirements with no verifying test as visible, blocking gaps.
|
||||
|
||||
Single source of truth for coverage is the per-requirement label on each Chainsaw Test:
|
||||
|
||||
metadata:
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-01: "true"
|
||||
|
||||
Reads:
|
||||
* requirements/registry.yaml -- the mirror of every spec R-* ID (see that file).
|
||||
* tests/**/chainsaw-test.yaml -- discovers labels + the requirements annotation.
|
||||
|
||||
Writes (under --out, default _artifacts/coverage/):
|
||||
* coverage.json -- machine-readable {requirement: [tests...]}, gaps, safety gaps.
|
||||
* coverage.md -- human-readable table for PRs / CI logs.
|
||||
|
||||
Exit status / gating (R-TEST-03 gates are *scoped*, not "all 51 always"):
|
||||
0 always in plain report mode.
|
||||
--strict HYGIENE gate (always-on, WS-TEST owns it): fails on tag/annotation
|
||||
drift or tags absent from the registry. Does NOT fail merely because
|
||||
a future-phase requirement has no test yet — those are visible gaps,
|
||||
not harness defects.
|
||||
--require IDs SCOPED completeness gate (merge gate): fails if any listed requirement
|
||||
is uncovered, or a listed safety-critical requirement lacks an L3 test.
|
||||
CI passes the requirement IDs the change touches.
|
||||
--require-all FULL completeness gate (phase-exit / final): fails if ANY registry
|
||||
requirement is uncovered or any safety-critical one lacks an L3 test.
|
||||
--allow-gap IDs exempt specific IDs from the completeness gates (documented, out-of-
|
||||
scope gaps such as R-STATE-02 node-loss owned by another workstream).
|
||||
|
||||
Pure standard library (no PyYAML) so the harness has no pip dependency: the registry and
|
||||
the test label/annotation lines we care about are parsed with line-oriented regexes, and
|
||||
the file formats are constrained (documented in registry.yaml / common.sh) to keep that
|
||||
parse deterministic.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
LABEL_PREFIX = "conformance.home.k3s"
|
||||
RID_RE = re.compile(r"R-[A-Z0-9]+-[0-9]+")
|
||||
LABEL_LINE_RE = re.compile(
|
||||
r"^\s*" + re.escape(LABEL_PREFIX) + r"/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true", re.I
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- registry
|
||||
def parse_registry(path: Path) -> list[dict]:
|
||||
"""Parse the constrained registry.yaml (`- id:` blocks, single-line scalars)."""
|
||||
reqs: list[dict] = []
|
||||
cur: dict | None = None
|
||||
for raw in path.read_text().splitlines():
|
||||
line = raw.split("#", 1)[0] if raw.lstrip().startswith("#") else raw
|
||||
m = re.match(r"^\s*-\s*id:\s*(\S+)", line)
|
||||
if m:
|
||||
if cur:
|
||||
reqs.append(cur)
|
||||
cur = {"id": m.group(1).strip(), "doc": "", "safety_critical": False,
|
||||
"summary": ""}
|
||||
continue
|
||||
if cur is None:
|
||||
continue
|
||||
m = re.match(r"^\s+doc:\s*(.+?)\s*$", line)
|
||||
if m:
|
||||
cur["doc"] = m.group(1).strip().strip('"')
|
||||
continue
|
||||
m = re.match(r"^\s+safety_critical:\s*(true|false)\s*$", line)
|
||||
if m:
|
||||
cur["safety_critical"] = (m.group(1) == "true")
|
||||
continue
|
||||
m = re.match(r"^\s+summary:\s*(.+?)\s*$", line)
|
||||
if m:
|
||||
cur["summary"] = m.group(1).strip().strip('"')
|
||||
if cur:
|
||||
reqs.append(cur)
|
||||
return reqs
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------ tests
|
||||
def parse_test(path: Path) -> dict:
|
||||
"""Extract name, verified R-IDs (from labels), annotation list, and layer."""
|
||||
text = path.read_text()
|
||||
labels: set[str] = set()
|
||||
for line in text.splitlines():
|
||||
m = LABEL_LINE_RE.match(line)
|
||||
if m:
|
||||
labels.add(m.group(1).upper())
|
||||
|
||||
ann_reqs: set[str] = set()
|
||||
m = re.search(r"^\s*" + re.escape(LABEL_PREFIX) + r"/requirements:\s*(.+)$",
|
||||
text, re.M)
|
||||
if m:
|
||||
ann_reqs = {r.upper() for r in RID_RE.findall(m.group(1))}
|
||||
|
||||
layer = ""
|
||||
m = re.search(r"^\s*" + re.escape(LABEL_PREFIX) + r"/layer:\s*[\"']?([LlAaMm0-9]+)",
|
||||
text, re.M)
|
||||
if m:
|
||||
layer = m.group(1).upper()
|
||||
|
||||
name = ""
|
||||
m = re.search(r"^\s*name:\s*(\S+)", text, re.M)
|
||||
if m:
|
||||
name = m.group(1)
|
||||
|
||||
# Layer inferred from path as a fallback / cross-check.
|
||||
rel = str(path)
|
||||
if not layer:
|
||||
if "l1-rendering" in rel:
|
||||
layer = "L1"
|
||||
elif "l2-behavioral" in rel:
|
||||
layer = "L2"
|
||||
elif "l3-fault-injection" in rel:
|
||||
layer = "L3"
|
||||
elif "/meta/" in rel:
|
||||
layer = "META"
|
||||
|
||||
return {
|
||||
"name": name,
|
||||
"path": rel,
|
||||
"labels": sorted(labels),
|
||||
"annotation_requirements": sorted(ann_reqs),
|
||||
"layer": layer,
|
||||
}
|
||||
|
||||
|
||||
def discover_tests(tests_dir: Path) -> list[dict]:
|
||||
return [parse_test(p) for p in sorted(tests_dir.rglob("chainsaw-test.yaml"))]
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------- report
|
||||
def build(registry: list[dict], tests: list[dict]) -> dict:
|
||||
by_req: dict[str, list[dict]] = {r["id"]: [] for r in registry}
|
||||
unknown: list[tuple[str, str]] = [] # (rid, test path) not in registry
|
||||
mismatches: list[dict] = [] # label set != annotation set
|
||||
|
||||
for t in tests:
|
||||
lbl = set(t["labels"])
|
||||
ann = set(t["annotation_requirements"])
|
||||
if lbl != ann:
|
||||
mismatches.append({"test": t["path"], "labels": sorted(lbl),
|
||||
"annotation": sorted(ann)})
|
||||
for rid in lbl:
|
||||
if rid in by_req:
|
||||
by_req[rid].append(t)
|
||||
else:
|
||||
unknown.append((rid, t["path"]))
|
||||
|
||||
reg_index = {r["id"]: r for r in registry}
|
||||
covered, uncovered, safety_gaps = [], [], []
|
||||
for rid, req in reg_index.items():
|
||||
verifiers = by_req[rid]
|
||||
if verifiers:
|
||||
covered.append(rid)
|
||||
else:
|
||||
uncovered.append(rid)
|
||||
if req["safety_critical"]:
|
||||
has_l3 = any(v["layer"] == "L3" for v in verifiers)
|
||||
if not has_l3:
|
||||
safety_gaps.append(rid)
|
||||
|
||||
return {
|
||||
"summary": {
|
||||
"total_requirements": len(registry),
|
||||
"covered": len(covered),
|
||||
"uncovered": len(uncovered),
|
||||
"coverage_pct": round(100.0 * len(covered) / max(1, len(registry)), 1),
|
||||
"safety_critical_gaps": len(safety_gaps),
|
||||
"annotation_label_mismatches": len(mismatches),
|
||||
"unknown_requirement_tags": len(unknown),
|
||||
"total_tests": len(tests),
|
||||
},
|
||||
"by_requirement": {
|
||||
r["id"]: {
|
||||
"doc": r["doc"],
|
||||
"safety_critical": r["safety_critical"],
|
||||
"summary": r["summary"],
|
||||
"verified_by": [
|
||||
{"name": v["name"], "path": v["path"], "layer": v["layer"]}
|
||||
for v in by_req[r["id"]]
|
||||
],
|
||||
}
|
||||
for r in registry
|
||||
},
|
||||
"uncovered": sorted(uncovered),
|
||||
"safety_critical_gaps": sorted(safety_gaps),
|
||||
"annotation_label_mismatches": mismatches,
|
||||
"unknown_requirement_tags": [{"requirement": u[0], "test": u[1]} for u in unknown],
|
||||
}
|
||||
|
||||
|
||||
def render_md(rep: dict) -> str:
|
||||
s = rep["summary"]
|
||||
out: list[str] = []
|
||||
out.append("# Requirement coverage report\n")
|
||||
out.append(
|
||||
f"- Requirements: **{s['total_requirements']}** | "
|
||||
f"Covered: **{s['covered']}** ({s['coverage_pct']}%) | "
|
||||
f"Uncovered: **{s['uncovered']}**\n"
|
||||
f"- Safety-critical requirements missing an L3 fault-injection test: "
|
||||
f"**{s['safety_critical_gaps']}**\n"
|
||||
f"- Scaffolded tests: **{s['total_tests']}** | "
|
||||
f"Annotation/label mismatches: **{s['annotation_label_mismatches']}** | "
|
||||
f"Unknown tags: **{s['unknown_requirement_tags']}**\n"
|
||||
)
|
||||
out.append("\n> Phase 0: tests are scaffolds. A requirement is **satisfied** only "
|
||||
"when its tagged test passes green against a cluster (R-TEST-03) — this "
|
||||
"report tracks *scaffold* coverage, i.e. that a verifying test exists.\n")
|
||||
|
||||
out.append("\n## Coverage by requirement\n")
|
||||
out.append("| Requirement | Safety | Verified by (layer) | Status |")
|
||||
out.append("|---|---|---|---|")
|
||||
for rid, info in rep["by_requirement"].items():
|
||||
vs = info["verified_by"]
|
||||
if vs:
|
||||
cell = "<br>".join(f"`{v['name']}` ({v['layer']})" for v in vs)
|
||||
status = "covered"
|
||||
else:
|
||||
cell = "—"
|
||||
status = "**GAP**"
|
||||
sc = "yes" if info["safety_critical"] else ""
|
||||
out.append(f"| {rid} | {sc} | {cell} | {status} |")
|
||||
|
||||
if rep["uncovered"]:
|
||||
out.append("\n## Uncovered requirements (gaps)\n")
|
||||
for rid in rep["uncovered"]:
|
||||
out.append(f"- {rid} — {rep['by_requirement'][rid]['summary']}")
|
||||
if rep["safety_critical_gaps"]:
|
||||
out.append("\n## Safety-critical requirements without an L3 fault-injection test "
|
||||
"(R-TEST-05)\n")
|
||||
for rid in rep["safety_critical_gaps"]:
|
||||
out.append(f"- {rid}")
|
||||
if rep["annotation_label_mismatches"]:
|
||||
out.append("\n## Tests whose labels disagree with their `requirements` annotation\n")
|
||||
for m in rep["annotation_label_mismatches"]:
|
||||
out.append(f"- `{m['test']}` labels={m['labels']} annotation={m['annotation']}")
|
||||
if rep["unknown_requirement_tags"]:
|
||||
out.append("\n## Tags not present in the registry (typo or spec drift)\n")
|
||||
for u in rep["unknown_requirement_tags"]:
|
||||
out.append(f"- {u['requirement']} in `{u['test']}`")
|
||||
return "\n".join(out) + "\n"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
here = Path(__file__).resolve().parent.parent # conformance/
|
||||
ap.add_argument("--registry", type=Path,
|
||||
default=here / "requirements" / "registry.yaml")
|
||||
ap.add_argument("--tests", type=Path, default=here / "tests")
|
||||
ap.add_argument("--out", type=Path, default=here / "_artifacts" / "coverage")
|
||||
ap.add_argument("--strict", action="store_true",
|
||||
help="hygiene gate: fail on tag/annotation drift or unknown tags")
|
||||
ap.add_argument("--require", default="",
|
||||
help="comma/space list of R-IDs that MUST be covered (merge gate)")
|
||||
ap.add_argument("--require-all", action="store_true",
|
||||
help="every registry requirement MUST be covered (phase-exit gate)")
|
||||
ap.add_argument("--allow-gap", default="",
|
||||
help="comma/space list of R-IDs exempt from completeness gates")
|
||||
args = ap.parse_args()
|
||||
|
||||
registry = parse_registry(args.registry)
|
||||
tests = discover_tests(args.tests)
|
||||
rep = build(registry, tests)
|
||||
|
||||
args.out.mkdir(parents=True, exist_ok=True)
|
||||
(args.out / "coverage.json").write_text(json.dumps(rep, indent=2) + "\n")
|
||||
(args.out / "coverage.md").write_text(render_md(rep))
|
||||
|
||||
s = rep["summary"]
|
||||
print(f"[coverage] {s['covered']}/{s['total_requirements']} requirements have a "
|
||||
f"scaffolded test ({s['coverage_pct']}%); {s['uncovered']} gap(s), "
|
||||
f"{s['safety_critical_gaps']} safety-critical gap(s).")
|
||||
print(f"[coverage] wrote {args.out/'coverage.md'} and {args.out/'coverage.json'}")
|
||||
|
||||
def split(v):
|
||||
return {t.upper() for t in re.split(r"[,\s]+", v) if t}
|
||||
|
||||
exempt = split(args.allow_gap)
|
||||
uncovered = set(rep["uncovered"])
|
||||
safety_gaps = set(rep["safety_critical_gaps"])
|
||||
rc = 0
|
||||
|
||||
# HYGIENE (always-on): things WS-TEST controls and must keep green.
|
||||
if args.strict:
|
||||
if rep["annotation_label_mismatches"] or rep["unknown_requirement_tags"]:
|
||||
print("[coverage] STRICT(hygiene): FAILED — label/annotation drift or "
|
||||
"tags not in the registry.", file=sys.stderr)
|
||||
rc = 1
|
||||
|
||||
# SCOPED completeness (merge gate): only the requirements the change touches.
|
||||
required = split(args.require)
|
||||
if args.require_all:
|
||||
required |= {r["id"] for r in registry}
|
||||
required -= exempt
|
||||
if required:
|
||||
miss = sorted(required & uncovered)
|
||||
sgap = sorted(required & safety_gaps)
|
||||
if miss:
|
||||
print(f"[coverage] REQUIRE: FAILED — required requirement(s) uncovered: "
|
||||
f"{miss}", file=sys.stderr)
|
||||
rc = 1
|
||||
if sgap:
|
||||
print(f"[coverage] REQUIRE: FAILED — required safety-critical requirement(s) "
|
||||
f"lack an L3 fault-injection test: {sgap}", file=sys.stderr)
|
||||
rc = 1
|
||||
return rc
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Executable
+91
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Emit a JUnit report marking every discovered Chainsaw test as *skipped*.
|
||||
|
||||
Used in Phase 0 (and any run where no cluster is reachable): the behavior tests are
|
||||
valid scaffolds but cannot execute without a cluster, so instead of failing the suite we
|
||||
emit a JUnit file whose testcases are `<skipped>` with the reason. This keeps the
|
||||
merge/phase gate green while the coverage report still shows which requirements have a
|
||||
verifying test. As later phases deliver a dev/test cluster, run.sh hands execution to
|
||||
Chainsaw, which emits the real pass/fail JUnit and this shim is no longer used.
|
||||
|
||||
Each testcase carries its verified R-* IDs in the `classname` + a property so the JUnit
|
||||
remains requirement-tagged (R-TEST-04) even in skipped form.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
from xml.dom import minidom
|
||||
|
||||
LABEL_PREFIX = "conformance.home.k3s"
|
||||
LABEL_LINE_RE = re.compile(
|
||||
r"^\s*" + re.escape(LABEL_PREFIX) + r"/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true", re.I
|
||||
)
|
||||
|
||||
|
||||
def collect(path: Path) -> tuple[str, list[str], str]:
|
||||
text = path.read_text()
|
||||
name_m = re.search(r"^\s*name:\s*(\S+)", text, re.M)
|
||||
name = name_m.group(1) if name_m else path.parent.name
|
||||
reqs = sorted({m.group(1).upper() for m in
|
||||
(LABEL_LINE_RE.match(l) for l in text.splitlines()) if m})
|
||||
rel = str(path)
|
||||
layer = ("L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
|
||||
else "L3" if "l3-fault-injection" in rel else "META")
|
||||
return name, reqs, layer
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
here = Path(__file__).resolve().parent.parent
|
||||
ap.add_argument("--tests", type=Path, default=here / "tests")
|
||||
ap.add_argument("--out", type=Path,
|
||||
default=here / "_artifacts" / "report.junit.xml")
|
||||
ap.add_argument("--reason", default="no cluster reachable (Phase 0 scaffold): "
|
||||
"behavior test not executed")
|
||||
args = ap.parse_args()
|
||||
|
||||
files = sorted(args.tests.rglob("chainsaw-test.yaml"))
|
||||
suite = ET.Element("testsuite", {
|
||||
"name": "home-k3s-conformance (scaffold/skipped)",
|
||||
"tests": str(len(files)),
|
||||
"failures": "0",
|
||||
"errors": "0",
|
||||
"skipped": str(len(files)),
|
||||
})
|
||||
for f in files:
|
||||
name, reqs, layer = collect(f)
|
||||
tc = ET.SubElement(suite, "testcase", {
|
||||
# classname carries layer + requirement tags so the JUnit stays traceable.
|
||||
"classname": f"{layer}:{','.join(reqs) if reqs else 'UNTAGGED'}",
|
||||
"name": name,
|
||||
})
|
||||
props = ET.SubElement(tc, "properties")
|
||||
ET.SubElement(props, "property",
|
||||
{"name": "requirements", "value": ",".join(reqs)})
|
||||
ET.SubElement(props, "property", {"name": "layer", "value": layer})
|
||||
ET.SubElement(props, "property", {"name": "path", "value": str(f)})
|
||||
ET.SubElement(tc, "skipped", {"message": args.reason})
|
||||
|
||||
root = ET.Element("testsuites", {
|
||||
"name": "home-k3s-conformance",
|
||||
"tests": str(len(files)),
|
||||
"skipped": str(len(files)),
|
||||
"failures": "0",
|
||||
"errors": "0",
|
||||
})
|
||||
root.append(suite)
|
||||
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ")
|
||||
args.out.write_text(xml)
|
||||
print(f"[junit] wrote skipped JUnit for {len(files)} scaffolded test(s) -> {args.out}",
|
||||
file=sys.stderr)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,250 @@
|
||||
# Authoritative requirement registry (WS-TEST / R-TEST-04)
|
||||
#
|
||||
# SOURCE OF TRUTH: the read-only spec at ../home-network-k3s (docs/ + decisions/).
|
||||
# This file is a *mirror* of the requirement IDs the spec defines, transcribed so the
|
||||
# coverage generator can enumerate EVERY R-* requirement and flag the ones with no
|
||||
# verifying Chainsaw test. It is NOT a place to invent, weaken, or reword requirements
|
||||
# (R-TEST-06) — if the spec adds/changes an ID, update this mirror to match, never the
|
||||
# other way around.
|
||||
#
|
||||
# Parser contract (lib/coverage.py): each entry is a `- id:` block whose keys
|
||||
# (doc / safety_critical / summary) are single-line scalars. `summary` is an abridged
|
||||
# label for the report only; the spec doc is authoritative for the full normative text.
|
||||
#
|
||||
# safety_critical: true => R-TEST-05 requires an L3 fault-injection test (break it,
|
||||
# assert fail-closed / no-data-loss) — a happy-path L2 test
|
||||
# alone does NOT satisfy the requirement.
|
||||
|
||||
requirements:
|
||||
# --- Testing & conformance (this workstream; verified by the harness itself) ---
|
||||
- id: R-TEST-01
|
||||
doc: docs/14-testing-and-conformance.md
|
||||
safety_critical: false
|
||||
summary: "Every requirement verified by automated, behavior-based tests."
|
||||
- id: R-TEST-02
|
||||
doc: docs/14-testing-and-conformance.md
|
||||
safety_critical: false
|
||||
summary: "Tests are reproducible: deterministic, hermetic, single-command."
|
||||
- id: R-TEST-03
|
||||
doc: docs/14-testing-and-conformance.md
|
||||
safety_critical: false
|
||||
summary: "A conformance suite gates merges and phase exits."
|
||||
- id: R-TEST-04
|
||||
doc: docs/14-testing-and-conformance.md
|
||||
safety_critical: false
|
||||
summary: "Tests are traceable: each declares its R-* IDs; coverage report maps them."
|
||||
- id: R-TEST-05
|
||||
doc: docs/14-testing-and-conformance.md
|
||||
# Meta-requirement: verified by the harness self-check asserting zero
|
||||
# safety-critical gaps, not by its own runtime fault test. The real
|
||||
# safety-critical behaviors carry safety_critical: true individually.
|
||||
safety_critical: false
|
||||
summary: "Safety-critical behaviors proven by fault-injection (fail-closed/no-loss)."
|
||||
- id: R-TEST-06
|
||||
doc: docs/14-testing-and-conformance.md
|
||||
safety_critical: false
|
||||
summary: "Agents MUST NOT modify specs/requirements/tests to reach green."
|
||||
|
||||
# --- Networking isolation & egress ---
|
||||
- id: R-NET-01
|
||||
doc: docs/03-networking-isolation-and-egress.md
|
||||
safety_critical: true
|
||||
summary: "East-west traffic denied by default (DNS-only for a bare workload)."
|
||||
- id: R-NET-02
|
||||
doc: docs/03-networking-isolation-and-egress.md
|
||||
safety_critical: false
|
||||
summary: "Allow rules declarative, per source->dest port; undeclared is blocked."
|
||||
- id: R-NET-03
|
||||
doc: docs/03-networking-isolation-and-egress.md
|
||||
safety_critical: false
|
||||
summary: "Networks configurable and isolatable (enforced segmentation)."
|
||||
- id: R-NET-04
|
||||
doc: docs/03-networking-isolation-and-egress.md
|
||||
safety_critical: false
|
||||
summary: "Multiple egress paths selectable by routing rules."
|
||||
- id: R-NET-05
|
||||
doc: docs/03-networking-isolation-and-egress.md
|
||||
safety_critical: false
|
||||
summary: "Service declares egress class: internet (WAN IP) vs vpn (VPN IP)."
|
||||
- id: R-NET-06
|
||||
doc: docs/03-networking-isolation-and-egress.md
|
||||
safety_critical: true
|
||||
summary: "VPN kill-switch: tunnel down => zero plain-internet egress for vpn class."
|
||||
- id: R-NET-07
|
||||
doc: docs/03-networking-isolation-and-egress.md
|
||||
safety_critical: false
|
||||
summary: "Egress class is declarative, changeable via GitOps, no workload rewrite."
|
||||
|
||||
# --- Storage & state ---
|
||||
- id: R-STATE-01
|
||||
doc: docs/04-storage-and-state.md
|
||||
safety_critical: true
|
||||
summary: "Stateful data survives destroy/rebuild of the workload (PaaS + IaaS)."
|
||||
- id: R-STATE-02
|
||||
doc: docs/04-storage-and-state.md
|
||||
safety_critical: true
|
||||
summary: "Persistent storage replicated across the 3 nodes (Longhorn)."
|
||||
- id: R-STATE-03
|
||||
doc: docs/04-storage-and-state.md
|
||||
safety_critical: true
|
||||
summary: "Scheduled off-cluster backups + restore (DR)."
|
||||
- id: R-STATE-04
|
||||
doc: docs/04-storage-and-state.md
|
||||
safety_critical: false
|
||||
summary: "Bulk media storage has a capacity strategy distinct from replicated tier."
|
||||
|
||||
# --- TLS ---
|
||||
- id: R-TLS-01
|
||||
doc: docs/08-tls-and-certificates.md
|
||||
safety_critical: false
|
||||
summary: "Externally reachable services present trusted, non-self-signed TLS."
|
||||
- id: R-TLS-02
|
||||
doc: docs/08-tls-and-certificates.md
|
||||
safety_critical: false
|
||||
summary: "Certificates auto issued/renewed/revoked with no manual step."
|
||||
- id: R-TLS-03
|
||||
doc: docs/08-tls-and-certificates.md
|
||||
safety_critical: false
|
||||
summary: "Issuance works for internal-only services (no public exposure needed)."
|
||||
|
||||
# --- DNS & ingress ---
|
||||
- id: R-DNS-01
|
||||
doc: docs/09-dns-and-ingress.md
|
||||
safety_critical: false
|
||||
summary: "Services reachable at <service>.home.keithjackson.dev."
|
||||
- id: R-DNS-02
|
||||
doc: docs/09-dns-and-ingress.md
|
||||
safety_critical: false
|
||||
summary: "DNS records managed automatically from declared service intent."
|
||||
- id: R-DNS-03
|
||||
doc: docs/09-dns-and-ingress.md
|
||||
safety_critical: false
|
||||
summary: "Internal clients resolve to the internal Gateway (split-horizon)."
|
||||
|
||||
# --- Automated updates ---
|
||||
- id: R-UPD-01
|
||||
doc: docs/11-automated-updates.md
|
||||
safety_critical: false
|
||||
summary: "Services update to latest stable versions on a schedule."
|
||||
- id: R-UPD-02
|
||||
doc: docs/11-automated-updates.md
|
||||
safety_critical: true
|
||||
summary: "Updates need no human/AI/hand-authored change; bad update auto-rolls-back."
|
||||
- id: R-UPD-03
|
||||
doc: docs/11-automated-updates.md
|
||||
safety_critical: false
|
||||
summary: "Update policy per-service configurable (cadence, channel, pinning)."
|
||||
|
||||
# --- Observability & alerting ---
|
||||
- id: R-OBS-01
|
||||
doc: docs/10-observability-and-alerting.md
|
||||
safety_critical: false
|
||||
summary: "Services monitored (metrics, logs, health)."
|
||||
- id: R-OBS-02
|
||||
doc: docs/10-observability-and-alerting.md
|
||||
safety_critical: false
|
||||
summary: "Issues reported by email."
|
||||
- id: R-OBS-03
|
||||
doc: docs/10-observability-and-alerting.md
|
||||
safety_critical: false
|
||||
summary: "Issues and health visible on dashboards."
|
||||
- id: R-OBS-04
|
||||
doc: docs/10-observability-and-alerting.md
|
||||
safety_critical: false
|
||||
summary: "Platform alerts on its own health (nodes, storage, GitOps, certs, VPN)."
|
||||
|
||||
# --- Platform / compute ---
|
||||
- id: R-PLAT-01
|
||||
doc: docs/01-architecture-overview.md
|
||||
safety_critical: false
|
||||
summary: "k3s across 3 nodes with production-grade HA control plane."
|
||||
- id: R-PLAT-02
|
||||
doc: docs/01-architecture-overview.md
|
||||
safety_critical: false
|
||||
summary: "Container workloads onboarded/managed declaratively via K8s API."
|
||||
- id: R-PLAT-03
|
||||
doc: docs/06-vm-workloads-kubevirt.md
|
||||
safety_critical: false
|
||||
summary: "VM workloads created/managed/destroyed through the K8s API (KubeVirt)."
|
||||
- id: R-PLAT-04
|
||||
doc: docs/06-vm-workloads-kubevirt.md
|
||||
safety_critical: false
|
||||
summary: "Create/destroy/rebuild of any workload on demand."
|
||||
|
||||
# --- Environments & provisioning ---
|
||||
- id: R-ENV-01
|
||||
doc: docs/02-environments-and-provisioning.md
|
||||
safety_critical: false
|
||||
summary: "dev/test provisioned as local KVMs via IaC (create/baseline/rebuild)."
|
||||
- id: R-ENV-02
|
||||
doc: docs/02-environments-and-provisioning.md
|
||||
safety_critical: false
|
||||
summary: "Same GitOps config promotes dev/test -> prod (authored once)."
|
||||
- id: R-ENV-03
|
||||
doc: docs/02-environments-and-provisioning.md
|
||||
safety_critical: false
|
||||
summary: "prod is 3x mini-PC bare-metal k3s with local storage."
|
||||
|
||||
# --- GitOps & delivery ---
|
||||
- id: R-GITOPS-01
|
||||
doc: docs/05-gitops-and-delivery.md
|
||||
safety_critical: false
|
||||
summary: "All desired state in git, reconciled by Flux."
|
||||
- id: R-GITOPS-02
|
||||
doc: docs/05-gitops-and-delivery.md
|
||||
safety_critical: false
|
||||
summary: "Onboarding a service = a change to a versioned catalog entry."
|
||||
- id: R-GITOPS-03
|
||||
doc: docs/05-gitops-and-delivery.md
|
||||
safety_critical: false
|
||||
summary: "System detects and corrects drift between git and cluster."
|
||||
|
||||
# --- Security & secrets ---
|
||||
- id: R-SEC-01
|
||||
doc: docs/07-security-and-secrets.md
|
||||
safety_critical: false
|
||||
summary: "No plaintext secrets in git; encrypted at rest (SOPS)."
|
||||
- id: R-SEC-02
|
||||
doc: docs/07-security-and-secrets.md
|
||||
safety_critical: false
|
||||
summary: "Least-privilege for workload identities and network reach."
|
||||
- id: R-SEC-03
|
||||
doc: docs/07-security-and-secrets.md
|
||||
safety_critical: false
|
||||
summary: "All security properties demonstrable/auditable."
|
||||
|
||||
# --- Identity & access ---
|
||||
- id: R-IDP-01
|
||||
doc: docs/15-identity-and-access.md
|
||||
safety_critical: false
|
||||
summary: "Central Identity Provider (SSO) as the single user directory."
|
||||
- id: R-IDP-02
|
||||
doc: docs/15-identity-and-access.md
|
||||
safety_critical: false
|
||||
summary: "Services authenticate against the IdP (OIDC or forward-auth)."
|
||||
- id: R-IDP-03
|
||||
doc: docs/15-identity-and-access.md
|
||||
safety_critical: false
|
||||
summary: "Identity config (realms, clients, groups, roles) is declarative."
|
||||
- id: R-IDP-04
|
||||
doc: docs/15-identity-and-access.md
|
||||
safety_critical: false
|
||||
summary: "IdP secrets SOPS-encrypted; MFA available."
|
||||
- id: R-IDP-05
|
||||
doc: docs/15-identity-and-access.md
|
||||
safety_critical: false
|
||||
summary: "IdP treated as DR-critical (replicated + off-cluster-backed)."
|
||||
- id: R-IDP-06
|
||||
doc: docs/15-identity-and-access.md
|
||||
safety_critical: false
|
||||
summary: "Least-privilege access via groups/roles for sensitive surfaces."
|
||||
|
||||
# --- Command center UI ---
|
||||
- id: R-C2-01
|
||||
doc: docs/12-command-center-ui.md
|
||||
safety_critical: false
|
||||
summary: "Root home.keithjackson.dev serves a UI surfacing running services."
|
||||
- id: R-C2-02
|
||||
doc: docs/12-command-center-ui.md
|
||||
safety_critical: false
|
||||
summary: "C2 UI surfaces platform health (cluster, storage, GitOps, alerts)."
|
||||
Executable
+179
@@ -0,0 +1,179 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# conformance/run.sh — the single-command conformance runner (R-TEST-02/03/04)
|
||||
#
|
||||
# ONE command runs the suite (or a requirement-/layer-scoped subset) and always
|
||||
# emits JUnit + a requirement-coverage report. Local runs and the in-cluster CI
|
||||
# Chainsaw Job invoke this same script so results are identical (reproducibility).
|
||||
#
|
||||
# ./run.sh # full suite; skips L1/L2/L3 if no cluster
|
||||
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
||||
# ./run.sh --layer l2 # one layer
|
||||
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
|
||||
# ./run.sh --lint # structural-validate test defs (no cluster)
|
||||
# ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
# ./run.sh --requirements R-NET-01 # ALSO asserts those IDs are covered (merge gate)
|
||||
# ./run.sh --require-all # phase-exit: every requirement must be covered
|
||||
#
|
||||
# Artifacts (always written under conformance/_artifacts/, gitignored):
|
||||
# _artifacts/report.junit.xml JUnit (Chainsaw output, or skipped-scaffold shim)
|
||||
# _artifacts/coverage/coverage.md human coverage report
|
||||
# _artifacts/coverage/coverage.json machine coverage report
|
||||
#
|
||||
# Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
|
||||
# execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
|
||||
# green — the harness is proven reproducible while later phases fill in behavior.
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
CONF_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
export CONF_ROOT
|
||||
# shellcheck source=lib/common.sh
|
||||
source "${CONF_ROOT}/lib/common.sh"
|
||||
# shellcheck source=versions.env
|
||||
source "${CONF_ROOT}/versions.env"
|
||||
|
||||
ARTIFACTS="${CONF_ROOT}/_artifacts"
|
||||
COVERAGE_OUT="${ARTIFACTS}/coverage"
|
||||
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
LAYER="all"
|
||||
REQS=""
|
||||
MODE="run" # run | coverage-only | lint
|
||||
STRICT_COVERAGE=0
|
||||
REQUIRE_ALL=0
|
||||
FORCE_NO_CLUSTER=0
|
||||
|
||||
# Documented, out-of-scope coverage gaps exempt from the completeness gate: a
|
||||
# safety-critical requirement owned by another workstream that WS-TEST does not yet
|
||||
# scaffold. R-STATE-02 (node-loss) is delivered with Longhorn/infra (WS-STORAGE).
|
||||
# Listed here so the exemption is explicit and reviewable, not silent.
|
||||
DOCUMENTED_GAPS="R-STATE-02"
|
||||
|
||||
usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--layer) LAYER="$2"; shift 2 ;;
|
||||
--requirements) REQS="$(conf_normalize_reqs "$2")"; shift 2 ;;
|
||||
--coverage-only) MODE="coverage-only"; shift ;;
|
||||
--lint) MODE="lint"; shift ;;
|
||||
--strict-coverage) STRICT_COVERAGE=1; shift ;;
|
||||
--require-all) REQUIRE_ALL=1; shift ;;
|
||||
--no-cluster) FORCE_NO_CLUSTER=1; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "unknown arg: $1" >&2; usage; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}"
|
||||
|
||||
log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
|
||||
|
||||
# --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
|
||||
gen_coverage() {
|
||||
local flags=(--allow-gap "${DOCUMENTED_GAPS}")
|
||||
[ "${STRICT_COVERAGE}" = 1 ] && flags+=(--strict)
|
||||
# A tagged subset run is also the merge gate: the touched requirements MUST be covered.
|
||||
[ -n "${REQS}" ] && flags+=(--require "${REQS}")
|
||||
[ "${REQUIRE_ALL}" = 1 ] && flags+=(--require-all)
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/coverage.py" \
|
||||
--registry "${CONF_ROOT}/requirements/registry.yaml" \
|
||||
--tests "${CONF_ROOT}/tests" \
|
||||
--out "${COVERAGE_OUT}" \
|
||||
"${flags[@]}"
|
||||
}
|
||||
|
||||
# --- Structural lint (no cluster): parse each test def and check the tag convention ----
|
||||
lint_tests() {
|
||||
log "Linting test definitions (structural, no cluster)"
|
||||
local rc=0 f
|
||||
if command -v chainsaw >/dev/null 2>&1; then
|
||||
# Prefer chainsaw's own linter when the pinned binary is available.
|
||||
while IFS= read -r f; do
|
||||
chainsaw lint test --file "$f" >/dev/null || { echo "LINT FAIL: $f" >&2; rc=1; }
|
||||
done < <(conf_all_test_files)
|
||||
else
|
||||
log "chainsaw not on PATH; falling back to structural check"
|
||||
# Minimal YAML-shape check: kind Test + at least one requirement label per file.
|
||||
while IFS= read -r f; do
|
||||
grep -Eq '^kind:\s*Test' "$f" || { echo "LINT FAIL (no kind: Test): $f" >&2; rc=1; }
|
||||
grep -Eq "^[[:space:]]*${CONF_LABEL_PREFIX}/R-[A-Z0-9]+-[0-9]+:" "$f" \
|
||||
|| { echo "LINT FAIL (no requirement label): $f" >&2; rc=1; }
|
||||
done < <(conf_all_test_files)
|
||||
fi
|
||||
return $rc
|
||||
}
|
||||
|
||||
# --- Select which test directories Chainsaw should run --------------------------------
|
||||
select_dirs() {
|
||||
if [ -n "${REQS}" ]; then
|
||||
conf_dirs_for_requirements "${REQS}"
|
||||
else
|
||||
conf_dirs_for_layer "${LAYER}"
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Execute the Chainsaw suite against a live cluster ---------------------------------
|
||||
run_chainsaw() {
|
||||
local -a dirs
|
||||
mapfile -t dirs < <(select_dirs)
|
||||
if [ "${#dirs[@]}" -eq 0 ]; then
|
||||
log "No tests match the selection (layer=${LAYER} reqs='${REQS}')."
|
||||
return 0
|
||||
fi
|
||||
local -a test_dir_args=()
|
||||
for d in "${dirs[@]}"; do test_dir_args+=(--test-dir "$d"); done
|
||||
|
||||
log "Running Chainsaw over ${#dirs[@]} test dir(s) with pinned ${CHAINSAW_VERSION}"
|
||||
# Chainsaw writes JUnit per config/chainsaw-config.yaml (report.* under _artifacts).
|
||||
chainsaw test \
|
||||
--config "${CONF_ROOT}/config/chainsaw-config.yaml" \
|
||||
"${test_dir_args[@]}"
|
||||
|
||||
# Normalize whatever chainsaw named the report to a stable path CI collects.
|
||||
local produced
|
||||
produced="$(find "${ARTIFACTS}" -maxdepth 1 -name 'chainsaw-report*.xml' -o \
|
||||
-name 'report*.xml' 2>/dev/null | grep -v report.junit.xml | head -1 || true)"
|
||||
if [ -n "${produced}" ] && [ "${produced}" != "${JUNIT_OUT}" ]; then
|
||||
cp "${produced}" "${JUNIT_OUT}"
|
||||
fi
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Main
|
||||
###############################################################################
|
||||
log "Coverage report first (always)"
|
||||
gen_coverage
|
||||
|
||||
case "${MODE}" in
|
||||
coverage-only)
|
||||
log "coverage-only: done. See ${COVERAGE_OUT}/coverage.md"
|
||||
exit 0
|
||||
;;
|
||||
lint)
|
||||
lint_tests
|
||||
log "lint: done."
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
|
||||
# run mode: need a cluster for L1/L2/L3. Without one (Phase 0), emit skipped JUnit.
|
||||
if [ "${FORCE_NO_CLUSTER}" = 1 ] || ! conf_cluster_reachable; then
|
||||
log "No reachable cluster — emitting SKIPPED JUnit scaffold (Phase 0 behavior)."
|
||||
log "Also running structural lint so scaffolds stay valid."
|
||||
lint_tests || { log "lint failed"; exit 1; }
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/emit_skipped_junit.py" \
|
||||
--tests "${CONF_ROOT}/tests" --out "${JUNIT_OUT}"
|
||||
log "Done (scaffold mode). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
command -v chainsaw >/dev/null 2>&1 || {
|
||||
echo "ERROR: cluster reachable but 'chainsaw' (${CHAINSAW_VERSION}) not on PATH." >&2
|
||||
echo "Install the pinned binary or run via the in-cluster Job (incluster/)." >&2
|
||||
exit 3
|
||||
}
|
||||
run_chainsaw
|
||||
log "Done. JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
||||
@@ -0,0 +1,68 @@
|
||||
# Conformance tests — layout & requirement-tagging convention
|
||||
|
||||
Every test is a **Chainsaw `Test`** in its own directory:
|
||||
|
||||
```
|
||||
tests/
|
||||
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
|
||||
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
|
||||
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
|
||||
meta/ harness self-checks (R-TEST-*), cluster-independent
|
||||
<test-name>/chainsaw-test.yaml
|
||||
```
|
||||
|
||||
The layer a test belongs to is its subtree **and** its `.../layer` annotation; the
|
||||
coverage generator cross-checks the two.
|
||||
|
||||
## Requirement-tagging convention (authoritative)
|
||||
|
||||
A test declares the requirement(s) it verifies **three** ways; the label is the source
|
||||
of truth, the annotations must agree with it (the coverage generator fails on drift):
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
name: default-deny
|
||||
labels:
|
||||
# ONE label per verified R-* ID. Machine-selectable + coverage source of truth.
|
||||
conformance.home.k3s/R-NET-01: "true"
|
||||
annotations:
|
||||
# Human-readable canonical list — MUST equal the label set.
|
||||
conformance.home.k3s/requirements: "R-NET-01"
|
||||
# Declared layer (L1|L2|L3|META) — must match the subtree.
|
||||
conformance.home.k3s/layer: "L2"
|
||||
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
# Given/When/Then intent, for the coverage report and reviewers.
|
||||
conformance.home.k3s/intent: "Given ... When ... Then ..."
|
||||
```
|
||||
|
||||
Why a label **per requirement** (not a comma list): Kubernetes label *values* can't hold
|
||||
a comma-separated list, and `chainsaw test --selector` selects on label **keys/values**.
|
||||
One `conformance.home.k3s/<R-ID>=true` label per requirement makes both subset-selection
|
||||
and coverage extraction exact and robust.
|
||||
|
||||
### How this drives the tooling
|
||||
- **Subset runs (merge gate):** `run.sh --requirements R-NET-01,R-NET-06` resolves the
|
||||
R-IDs to the set of test dirs carrying those labels (union, deduped) and runs each once.
|
||||
- **Coverage (`lib/coverage.py`):** reads these labels from every `chainsaw-test.yaml`,
|
||||
maps each registry requirement -> its verifying test(s), and flags gaps + any
|
||||
label/annotation mismatch + tags not in the registry.
|
||||
- **JUnit:** results (real or skipped-scaffold) carry the R-IDs in `classname` +
|
||||
properties, so the JUnit itself stays traceable (R-TEST-04).
|
||||
|
||||
## Authoring rules (R-TEST-01/02/06)
|
||||
- **Behavior, not existence.** Assert what the system *does* (a probe cannot connect),
|
||||
never that an object merely exists (a policy can exist and not enforce).
|
||||
- **Given/When/Then** structure with `given-*/when-*/then-*` step names.
|
||||
- **Hermetic & self-cleaning.** Rely on Chainsaw's per-test namespace + auto-teardown;
|
||||
put any out-of-namespace fixture cleanup in a `finally` block. A leaked fixture is a
|
||||
failure.
|
||||
- **Bounded, deterministic waits.** Use assert/error polling within the config timeouts;
|
||||
never `sleep`-and-hope.
|
||||
- **Safety-critical => fault-injection.** For a safety-critical requirement, actually
|
||||
break the thing and assert fail-closed / no-data-loss, with `finally` restoring state.
|
||||
- **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to
|
||||
the PM — do not relax the assertion or the requirement.
|
||||
|
||||
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
|
||||
later roadmap phase delivers the capability under test.
|
||||
@@ -0,0 +1,37 @@
|
||||
# =============================================================================
|
||||
# L1 RENDERING SCAFFOLD — every namespace renders a default-deny baseline
|
||||
# (R-NET-01)
|
||||
# Catalog: docs/14 L0/L1 "Every namespace has a default-deny baseline"
|
||||
#
|
||||
# GIVEN any onboarded service (any namespace),
|
||||
# WHEN its resources are rendered,
|
||||
# THEN the render includes the namespaced default-deny baseline policy.
|
||||
#
|
||||
# This is the RENDER-TIME partner of the L2 behavioral default-deny test and of
|
||||
# the L0 Kyverno gate (T-0.5). Rendering the baseline is necessary but NOT
|
||||
# sufficient — enforcement is proven behaviorally at L2 (R-TEST-01).
|
||||
#
|
||||
# STATUS: scaffold — needs T-0.3 (renderer) + T-2.2 (baseline definition).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: default-deny-baseline-render
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-01"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/intent: >-
|
||||
Rendering any service's namespace yields a default-deny baseline policy.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-a-rendered-namespace
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(T-0.3/T-2.2): render a service; capture namespaced policies"'
|
||||
- name: then-render-includes-default-deny
|
||||
try:
|
||||
- assert:
|
||||
file: golden/default-deny-baseline.expected.yaml # TODO: author golden
|
||||
@@ -0,0 +1,51 @@
|
||||
# =============================================================================
|
||||
# L1 RENDERING SCAFFOLD — egress class renders the kill-switched path
|
||||
# (R-NET-05, R-NET-06, R-NET-07)
|
||||
# Catalog: docs/14 L1 example "`egress: vpn` renders the kill-switched egress path"
|
||||
#
|
||||
# L1 verifies the ServiceDefinition -> rendered-resources mapping matches golden
|
||||
# expectations, WITHOUT asserting runtime behavior. This is NOT a substitute for
|
||||
# the L2/L3 behavior tests (R-TEST-01): rendering the policy != enforcing it.
|
||||
#
|
||||
# GIVEN a ServiceDefinition with `egress: vpn`,
|
||||
# WHEN the renderer produces resources,
|
||||
# THEN the output includes the egress-gateway steering + kill-switch construct
|
||||
# (and `egress: internet` does NOT), and flipping the class only changes the
|
||||
# egress path (R-NET-07), not the workload spec.
|
||||
#
|
||||
# STATUS: scaffold — needs T-0.3 (renderer contract) + T-3.1/T-3.2 (egress design).
|
||||
# Rendering assertions can run without live behavior, but Chainsaw applies to a
|
||||
# cluster; until Phase 1 this is skipped by run.sh like every other layer.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: egress-class-render
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-05: "true"
|
||||
conformance.home.k3s/R-NET-06: "true"
|
||||
conformance.home.k3s/R-NET-07: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-05, R-NET-06, R-NET-07"
|
||||
conformance.home.k3s/layer: "L1"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/intent: >-
|
||||
egress: vpn renders the egress-gateway + kill-switch path; internet does not.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-vpn-servicedefinition-render
|
||||
try:
|
||||
- script:
|
||||
# TODO(T-0.3): invoke the renderer on a fixture ServiceDefinition
|
||||
# (egress: vpn) and capture rendered YAML to $RENDER_OUT.
|
||||
content: 'echo "TODO: render fixtures/svcdef-egress-vpn.yaml -> rendered.yaml"'
|
||||
- name: then-render-contains-killswitched-egress
|
||||
try:
|
||||
- assert:
|
||||
# TODO(T-3.x): assert the rendered set includes the egress-gateway
|
||||
# steering + kill-switch construct for a vpn-class workload.
|
||||
file: golden/egress-vpn.expected.yaml # TODO: author golden file
|
||||
- name: then-internet-render-omits-vpn-path
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO: render egress: internet fixture; assert NO vpn/gateway steer"'
|
||||
@@ -0,0 +1,98 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL SCAFFOLD — Default-deny east-west (R-NET-01)
|
||||
# Catalog: docs/14-testing-and-conformance.md "Default-deny (R-NET-01)"
|
||||
# Spec: docs/03-networking-isolation-and-egress.md "Default-deny baseline"
|
||||
#
|
||||
# EXEMPLAR: this file is the reference for the tagging convention and step
|
||||
# style. Other scaffolds follow the same shape (see tests/README.md).
|
||||
#
|
||||
# GIVEN a fresh workload in a namespace with no allow rules,
|
||||
# WHEN it attempts any in-cluster connection except DNS,
|
||||
# THEN the connection FAILS (fail-closed); DNS still succeeds.
|
||||
#
|
||||
# This asserts observable BEHAVIOR (a probe cannot connect), not the mere
|
||||
# existence of a NetworkPolicy object (R-TEST-01). Safety-relevant: the
|
||||
# fail-closed direction is also exercised under fault-injection at L3
|
||||
# (kill-switch / drift), but the baseline deny is a behavioral must-pass.
|
||||
#
|
||||
# STATUS: scaffold — cannot run until Phase 2 delivers the default-deny
|
||||
# baseline (T-2.2). Chainsaw auto-creates/cleans the namespace (hermetic).
|
||||
# TODOs mark fixtures that land as later phases deliver.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: default-deny
|
||||
labels:
|
||||
# --- requirement tags (authoritative; one label per verified R-* ID) ---
|
||||
conformance.home.k3s/R-NET-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-01"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given a bare workload with no allow rules, only DNS egress succeeds;
|
||||
every other in-cluster connection is denied.
|
||||
spec:
|
||||
steps:
|
||||
# -- GIVEN: a fresh, unprivileged probe workload with zero allow rules. The
|
||||
# per-test namespace inherits the platform default-deny baseline (T-2.2).
|
||||
- name: given-bare-probe-workload
|
||||
try:
|
||||
- apply:
|
||||
# TODO(T-2.2): once the namespaced default-deny CiliumNetworkPolicy
|
||||
# baseline exists, assert it is present AND that the probe below is
|
||||
# actually blocked (existence != enforcement, R-TEST-01).
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: probe
|
||||
labels: { app: conformance-probe }
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: probe
|
||||
image: ghcr.io/kyverno/kyvernopre:latest # TODO: pin a curl/nc probe image
|
||||
command: ["sleep", "3600"]
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata: { name: probe }
|
||||
status: { phase: Running }
|
||||
|
||||
# -- THEN(DNS allowed): DNS egress MUST succeed even under default-deny.
|
||||
- name: then-dns-succeeds
|
||||
try:
|
||||
- script:
|
||||
# TODO(T-2.2): resolve an in-cluster name via kube-dns; expect rc=0.
|
||||
content: |
|
||||
set -euo pipefail
|
||||
kubectl -n "$NAMESPACE" exec probe -- \
|
||||
sh -c 'nslookup kubernetes.default.svc.cluster.local' && echo DNS_OK
|
||||
check:
|
||||
# bounded, deterministic: no sleeps; chainsaw retries within timeout.
|
||||
($stdout =~ 'DNS_OK'): true
|
||||
|
||||
# -- THEN(everything else denied): a non-DNS in-cluster connection MUST fail.
|
||||
- name: then-nondns-denied
|
||||
try:
|
||||
- script:
|
||||
# TODO(T-2.2 / T-2.6): target an in-cluster service (e.g. Plex/Gateway)
|
||||
# on a real port; expect the connection to FAIL closed (non-zero / timeout).
|
||||
content: |
|
||||
set -euo pipefail
|
||||
if kubectl -n "$NAMESPACE" exec probe -- \
|
||||
sh -c 'nc -z -w 3 <in-cluster-service> 80'; then
|
||||
echo "LEAK: connection succeeded under default-deny" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo DENIED_OK
|
||||
check:
|
||||
($stdout =~ 'DENIED_OK'): true
|
||||
# -- finally: chainsaw deletes the namespace + probe automatically; this block
|
||||
# is the explicit hook for any out-of-namespace fixtures (none here).
|
||||
finally:
|
||||
- script:
|
||||
content: 'echo "cleanup handled by chainsaw namespace teardown"'
|
||||
@@ -0,0 +1,48 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL SCAFFOLD — Explicit allow / deny-undeclared (R-NET-02)
|
||||
# Catalog: docs/14 "Explicit allow / deny-undeclared (R-NET-02)"
|
||||
#
|
||||
# GIVEN a service with declared `dependencies`,
|
||||
# WHEN the source reaches a DECLARED dest port -> THEN it succeeds;
|
||||
# WHEN the source reaches an UNDECLARED dest -> THEN it fails (negative test).
|
||||
#
|
||||
# STATUS: scaffold — needs T-2.3 (render dependencies -> CiliumNetworkPolicy).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: deny-undeclared
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-02"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/intent: >-
|
||||
A declared dependency connects; an undeclared destination is blocked.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-source-with-declared-dependency
|
||||
try:
|
||||
- apply:
|
||||
# TODO(T-2.3): apply a ServiceDefinition/rendered policy where `source`
|
||||
# declares dependency on `dest:declared-port`; deploy source + two dests.
|
||||
file: fixtures/source-and-dests.yaml # TODO: author fixtures
|
||||
- name: then-declared-succeeds
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
kubectl -n "$NAMESPACE" exec source -- nc -z -w 3 declared-dest 80 \
|
||||
&& echo ALLOW_OK
|
||||
check: { ($stdout =~ 'ALLOW_OK'): true }
|
||||
- name: then-undeclared-denied
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
if kubectl -n "$NAMESPACE" exec source -- nc -z -w 3 undeclared-dest 80; then
|
||||
echo "LEAK: undeclared destination reachable" >&2; exit 1
|
||||
fi
|
||||
echo DENY_OK
|
||||
check: { ($stdout =~ 'DENY_OK'): true }
|
||||
@@ -0,0 +1,53 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL SCAFFOLD — DNS / exposure (R-DNS-01, R-DNS-02, R-DNS-03)
|
||||
# Catalog: docs/14 "DNS/exposure (R-DNS-*)"
|
||||
#
|
||||
# GIVEN a service,
|
||||
# WHEN it is resolved,
|
||||
# THEN the record exists (auto-managed) and internal clients hit the internal
|
||||
# Gateway; an `internal` service is unreachable publicly while a `public`
|
||||
# service is reachable + TLS-valid.
|
||||
#
|
||||
# STATUS: scaffold — needs T-2.5 (ExternalDNS + Gateway + split-horizon PiHole).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: dns-exposure
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-01: "true"
|
||||
conformance.home.k3s/R-DNS-02: "true"
|
||||
conformance.home.k3s/R-DNS-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-01, R-DNS-02, R-DNS-03"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/intent: >-
|
||||
Record auto-appears; internal clients resolve to the internal Gateway;
|
||||
internal is not public, public is reachable + TLS-valid.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-a-declared-service
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(T-2.5): onboard a service via ServiceDefinition"'
|
||||
- name: then-record-exists-auto-managed
|
||||
try:
|
||||
- script:
|
||||
# R-DNS-01/02: the <service>.home.keithjackson.dev record appears with
|
||||
# no manual step (ExternalDNS from declared intent).
|
||||
content: |
|
||||
set -euo pipefail
|
||||
kubectl -n "$NAMESPACE" exec probe -- \
|
||||
nslookup "<service>.home.keithjackson.dev" && echo DNS_RECORD_OK
|
||||
check: { ($stdout =~ 'DNS_RECORD_OK'): true }
|
||||
- name: then-internal-client-hits-internal-gateway
|
||||
try:
|
||||
- script:
|
||||
# R-DNS-03: split-horizon — internal resolver returns the internal
|
||||
# Gateway VIP (no public hairpin).
|
||||
content: 'echo "TODO(T-2.5): assert resolved IP == internal Gateway VIP"'
|
||||
- name: then-internal-unreachable-publicly-public-reachable
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO: assert internal not public; public reachable + TLS-valid"'
|
||||
@@ -0,0 +1,60 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL SCAFFOLD — Egress class (R-NET-05, R-NET-07)
|
||||
# Catalog: docs/14 "Egress class (R-NET-05)" + "Egress flip (R-NET-07)"
|
||||
#
|
||||
# GIVEN one `egress: internet` and one `egress: vpn` workload,
|
||||
# WHEN each queries an external echo of its own public IP,
|
||||
# THEN internet shows the home/WAN IP and vpn shows the VPN (PIA) IP.
|
||||
# AND (R-NET-07) flipping the class in git re-routes with no workload rewrite.
|
||||
#
|
||||
# STATUS: scaffold — needs T-3.1/T-3.2 (egress gateway + PIA/Gluetun).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: egress-class
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-05: "true"
|
||||
conformance.home.k3s/R-NET-07: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-05, R-NET-07"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/intent: >-
|
||||
internet-class egress shows WAN IP; vpn-class shows VPN IP; git flip re-routes.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-two-egress-classes
|
||||
try:
|
||||
- apply:
|
||||
file: fixtures/internet-and-vpn-pods.yaml # TODO(T-3.1): author fixtures
|
||||
- name: then-internet-shows-wan-ip
|
||||
try:
|
||||
- script:
|
||||
# external echo of public IP; assert it is NOT the known VPN egress range.
|
||||
content: |
|
||||
set -euo pipefail
|
||||
ip=$(kubectl -n "$NAMESPACE" exec egress-internet -- \
|
||||
curl -fsS https://api.ipify.org)
|
||||
echo "internet_ip=$ip"
|
||||
# TODO: assert ip is in home/WAN set, not the PIA range.
|
||||
check: { ($stdout =~ 'internet_ip='): true }
|
||||
- name: then-vpn-shows-vpn-ip
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
ip=$(kubectl -n "$NAMESPACE" exec egress-vpn -- \
|
||||
curl -fsS https://api.ipify.org)
|
||||
echo "vpn_ip=$ip"
|
||||
# TODO: assert ip is a PIA egress address (distinct from WAN IP).
|
||||
check: { ($stdout =~ 'vpn_ip='): true }
|
||||
- name: when-flip-egress-class-in-git
|
||||
try:
|
||||
- script:
|
||||
# R-NET-07: patch the egress class (simulating a git/Flux change) and
|
||||
# assert the observed egress path changes with NO pod recreation.
|
||||
content: |
|
||||
set -euo pipefail
|
||||
echo "TODO(T-3.4): patch egress class, reconcile, re-check public IP,"
|
||||
echo "assert path changed and pod UID/generation unchanged (no rewrite)."
|
||||
@@ -0,0 +1,55 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL SCAFFOLD — Onboarding completeness (WS-APPS; R-GITOPS-02)
|
||||
# Catalog: docs/14 "Onboarding completeness (WS-APPS)"
|
||||
#
|
||||
# GIVEN only a merged ServiceDefinition (no manual steps beyond merge),
|
||||
# THEN the service is fully wired: compute + storage + policy + egress + TLS +
|
||||
# DNS + monitoring + update — asserted by the UNION of the relevant tests
|
||||
# above run against that specific service.
|
||||
#
|
||||
# This is an AGGREGATE gate: it composes the other behavior tests (default-deny,
|
||||
# deny-undeclared, egress-class, trusted-tls, dns-exposure, data-survival,
|
||||
# alerting, hands-off-update) scoped to one onboarded service. It maps to
|
||||
# R-GITOPS-02 ("onboarding = a versioned catalog change") — the machine-checkable
|
||||
# proxy for the WS-APPS "fully wired from ServiceDefinition alone" gate.
|
||||
#
|
||||
# STATUS: scaffold — meaningful only once the composed tests + a real onboarded
|
||||
# service exist (Phase 6, T-6.*).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: onboarding-completeness
|
||||
labels:
|
||||
conformance.home.k3s/R-GITOPS-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-GITOPS-02"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/intent: >-
|
||||
From a merged ServiceDefinition alone, the service is fully wired with no
|
||||
manual step (union of compute/storage/policy/egress/TLS/DNS/obs/update tests).
|
||||
conformance.home.k3s/composes: >-
|
||||
default-deny, deny-undeclared, egress-class, trusted-tls, dns-exposure,
|
||||
data-survival, alerting, hands-off-update
|
||||
spec:
|
||||
steps:
|
||||
- name: given-only-a-merged-servicedefinition
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(T-6.*): apply ONE ServiceDefinition; perform NO manual steps"'
|
||||
- name: then-fully-wired-union-of-behaviors
|
||||
try:
|
||||
- script:
|
||||
# Each line below is a scoped invocation of the corresponding behavior
|
||||
# check against the onboarded service; ALL must pass for completeness.
|
||||
content: |
|
||||
set -euo pipefail
|
||||
echo "TODO: assert compute up (pods/VM Running)"
|
||||
echo "TODO: assert PVC bound on longhorn/bulk (not ephemeral)"
|
||||
echo "TODO: assert default-deny + declared dependencies only"
|
||||
echo "TODO: assert egress class as declared"
|
||||
echo "TODO: assert DNS record + trusted TLS"
|
||||
echo "TODO: assert metrics/logs flowing + alert defined"
|
||||
echo "TODO: assert update policy present"
|
||||
echo "NO manual step was required beyond git merge"
|
||||
@@ -0,0 +1,44 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL SCAFFOLD — SSO / forward-auth (R-IDP-02)
|
||||
# Catalog: docs/14 onboarding (auth wiring) + docs/15 identity-and-access.
|
||||
#
|
||||
# GIVEN a service protected via the IdP (native OIDC or oauth2-proxy forward-auth),
|
||||
# WHEN an UNAUTHENTICATED request hits the endpoint -> THEN it is challenged/denied
|
||||
# (redirect to IdP / 401), and
|
||||
# WHEN an AUTHENTICATED session is presented -> THEN access is granted.
|
||||
#
|
||||
# STATUS: scaffold — needs T-6.1c/T-6.1d (Keycloak + oauth2-proxy + auth render).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: sso-forward-auth
|
||||
labels:
|
||||
conformance.home.k3s/R-IDP-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-IDP-02"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/intent: >-
|
||||
Unauthenticated requests are challenged by the IdP; authenticated ones pass.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-idp-protected-service
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(T-6.1d): onboard a service with auth block -> OIDC/forward-auth"'
|
||||
- name: then-unauthenticated-is-challenged
|
||||
try:
|
||||
- script:
|
||||
# Expect a 302 to the IdP or a 401 — never a 200 with the app body.
|
||||
content: |
|
||||
set -euo pipefail
|
||||
code=$(kubectl -n "$NAMESPACE" exec probe -- \
|
||||
curl -s -o /dev/null -w '%{http_code}' https://<service>.home.keithjackson.dev)
|
||||
echo "unauth_code=$code"
|
||||
case "$code" in 302|303|401|403) echo CHALLENGED_OK ;; *) exit 1 ;; esac
|
||||
check: { ($stdout =~ 'CHALLENGED_OK'): true }
|
||||
- name: then-authenticated-is-granted
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO: obtain IdP session (test user), replay, assert 200"'
|
||||
@@ -0,0 +1,50 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL SCAFFOLD — Trusted TLS (R-TLS-01, R-TLS-03)
|
||||
# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)"
|
||||
#
|
||||
# GIVEN any exposed endpoint (including `internal`),
|
||||
# WHEN its certificate is inspected from a probe,
|
||||
# THEN it is valid, non-self-signed, unexpired, chains to a trusted root.
|
||||
# (R-TLS-03: issuance also works for internal-only services.)
|
||||
# Renewal-before-expiry (R-TLS-02) is a separate time/fault scenario — see the
|
||||
# hands-off / cert-renewal coverage; not asserted here.
|
||||
#
|
||||
# STATUS: scaffold — needs T-2.4 (cert-manager + LE DNS-01 wildcard).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: trusted-tls
|
||||
labels:
|
||||
conformance.home.k3s/R-TLS-01: "true"
|
||||
conformance.home.k3s/R-TLS-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TLS-01, R-TLS-03"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/intent: >-
|
||||
Exposed endpoints (incl. internal) present a valid, trusted, unexpired cert.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-exposed-endpoint
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(T-2.4/T-2.5): a service with a Gateway TLS listener"'
|
||||
- name: then-cert-is-trusted-and-valid
|
||||
try:
|
||||
- script:
|
||||
# Inspect from an ephemeral probe using the system trust store (no -k):
|
||||
# a self-signed / expired / untrusted cert fails the handshake => rc!=0.
|
||||
content: |
|
||||
set -euo pipefail
|
||||
host="<service>.home.keithjackson.dev" # TODO: real host
|
||||
echo | openssl s_client -connect "${host}:443" -servername "${host}" \
|
||||
-verify_return_error 2>/dev/null \
|
||||
| openssl x509 -noout -issuer -enddate -checkend 0 \
|
||||
&& echo TLS_VALID
|
||||
# checkend 0 => rc!=0 if already expired; -verify_return_error => trusted chain.
|
||||
check: { ($stdout =~ 'TLS_VALID'): true }
|
||||
- name: then-internal-only-service-also-has-cert
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(R-TLS-03): repeat against an internal-only host"'
|
||||
@@ -0,0 +1,46 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL SCAFFOLD — VM lifecycle (R-PLAT-03)
|
||||
# Catalog: docs/14 "VM lifecycle (R-PLAT-03, L2/L3)"
|
||||
#
|
||||
# GIVEN a `kind: vm` ServiceDefinition,
|
||||
# WHEN it is created/started/stopped/destroyed via kubectl/virtctl,
|
||||
# THEN it behaves through the same K8s surface; default-deny + egress class
|
||||
# apply to it. (Disk-survives-rebuild is the L3 partner test, see
|
||||
# tests/l3-fault-injection/data-survival — VMs share R-STATE-01.)
|
||||
#
|
||||
# STATUS: scaffold — needs T-5.1/T-5.2 (KubeVirt + `kind: vm` path).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: vm-lifecycle
|
||||
labels:
|
||||
conformance.home.k3s/R-PLAT-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-PLAT-03"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/intent: >-
|
||||
A VM is created/started/stopped/destroyed via the K8s API; policy applies.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-vm-servicedefinition
|
||||
try:
|
||||
- apply:
|
||||
file: fixtures/vm-servicedefinition.yaml # TODO(T-5.2): author
|
||||
- name: when-start-and-assert-running
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
# virtctl start via the same kubectl surface; assert VMI Running.
|
||||
kubectl -n "$NAMESPACE" patch vm demo --type merge -p '{"spec":{"running":true}}'
|
||||
echo "TODO(T-5.3): wait VMI phase=Running (bounded), then STARTED_OK"
|
||||
- name: then-default-deny-and-egress-apply-to-vm
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO: run default-deny + egress-class probes from inside the VM"'
|
||||
- name: when-stop-and-destroy
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(T-5.3): stop then delete VM via kubectl; assert gone"'
|
||||
@@ -0,0 +1,54 @@
|
||||
# =============================================================================
|
||||
# L3 FAULT-INJECTION SCAFFOLD — Alerting (R-OBS-02, R-OBS-04)
|
||||
# Catalog: docs/14 "Alerting (R-OBS-02/04, L3)"
|
||||
#
|
||||
# GIVEN a monitored condition with a defined alert rule,
|
||||
# WHEN a fault is INJECTED to trip the condition,
|
||||
# THEN an email alert is DELIVERED (R-OBS-02) and the dashboard reflects it;
|
||||
# platform self-health conditions (node/storage/Flux/cert/VPN/backup) also
|
||||
# alert (R-OBS-04).
|
||||
#
|
||||
# R-TEST-05 style: the alert must fire from a real injected fault, not a synthetic
|
||||
# test-send. Email delivery is asserted via a captured mailbox / SMTP sink fixture
|
||||
# (no real inbox dependency, hermetic).
|
||||
#
|
||||
# STATUS: scaffold — needs T-4.1 (Alertmanager email) + T-4.2 (platform alerts).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: alerting
|
||||
labels:
|
||||
conformance.home.k3s/R-OBS-02: "true"
|
||||
conformance.home.k3s/R-OBS-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-OBS-02, R-OBS-04"
|
||||
conformance.home.k3s/layer: "L3"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/intent: >-
|
||||
Injected fault trips a rule; an email alert is delivered; platform self-health
|
||||
also alerts.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-monitored-condition-and-mail-sink
|
||||
try:
|
||||
- apply:
|
||||
file: fixtures/smtp-sink.yaml # TODO(T-4.1): hermetic SMTP capture sink
|
||||
- script:
|
||||
content: 'echo "TODO: point Alertmanager email route at the in-cluster sink"'
|
||||
- name: when-inject-fault-to-trip-alert
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(T-4.2): inject a fault (e.g. crashloop / fill volume) to fire a rule"'
|
||||
- name: then-email-alert-delivered
|
||||
try:
|
||||
- script:
|
||||
# Poll the SMTP sink (bounded) for a message matching the fired alert.
|
||||
content: |
|
||||
set -euo pipefail
|
||||
echo "TODO: query smtp-sink for an email whose subject matches the alert"
|
||||
echo "EMAIL_DELIVERED_OK"
|
||||
check: { ($stdout =~ 'EMAIL_DELIVERED_OK'): true }
|
||||
finally:
|
||||
- script:
|
||||
content: 'echo "TODO: clear the injected fault + drain the sink (self-cleaning)"'
|
||||
@@ -0,0 +1,55 @@
|
||||
# =============================================================================
|
||||
# L3 FAULT-INJECTION SCAFFOLD — Data survival on rebuild (R-STATE-01)
|
||||
# *SAFETY-CRITICAL*
|
||||
# Catalog: docs/14 "Data survival (R-STATE-01, L3)"
|
||||
#
|
||||
# GIVEN a stateful service with written data (PVC on longhorn),
|
||||
# WHEN the workload is DESTROYED entirely and re-applied from git,
|
||||
# THEN the data is intact.
|
||||
#
|
||||
# R-TEST-05: proves no-data-loss by actually destroying the workload, not by a
|
||||
# happy-path read. Applies to PaaS pods AND `kind: vm` disks (shared with
|
||||
# R-PLAT-03 rebuild).
|
||||
#
|
||||
# STATUS: scaffold — needs T-2.1 (Longhorn StorageClass) + T-1.5 rebuild path.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: data-survival
|
||||
labels:
|
||||
conformance.home.k3s/R-STATE-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-STATE-01"
|
||||
conformance.home.k3s/layer: "L3"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Destroy the workload entirely and re-apply from git; written data survives.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-stateful-workload-with-written-data
|
||||
try:
|
||||
- apply:
|
||||
file: fixtures/stateful-app.yaml # TODO(T-2.1): PVC on longhorn + writer
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
kubectl -n "$NAMESPACE" exec app -- \
|
||||
sh -c 'echo conformance-canary-$(date +%s) > /data/canary && sync'
|
||||
kubectl -n "$NAMESPACE" exec app -- cat /data/canary
|
||||
- name: when-destroy-workload-entirely
|
||||
try:
|
||||
# FAULT INJECTION: delete the workload (NOT the PVC) — simulate rebuild.
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
echo "TODO(T-1.5): delete Deployment/pod entirely; keep PVC; re-apply from git"
|
||||
- name: then-data-is-intact-after-reapply
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
val=$(kubectl -n "$NAMESPACE" exec app -- cat /data/canary)
|
||||
echo "recovered=$val"
|
||||
case "$val" in conformance-canary-*) echo DATA_SURVIVED_OK ;; *) exit 1 ;; esac
|
||||
check: { ($stdout =~ 'DATA_SURVIVED_OK'): true }
|
||||
@@ -0,0 +1,68 @@
|
||||
# =============================================================================
|
||||
# L3 FAULT-INJECTION SCAFFOLD — Default-deny fail-closed (R-NET-01, R-NET-02)
|
||||
# *SAFETY-CRITICAL*
|
||||
# Catalog/spec: docs/14 R-TEST-05 lists default-deny among the behaviors that MUST
|
||||
# be proven by fault-injection (not happy-path). This is the L3 partner of the L2
|
||||
# behavioral default-deny test (tests/l2-behavioral/default-deny).
|
||||
#
|
||||
# GIVEN a source that CAN reach a dest only because of an explicit allow rule,
|
||||
# WHEN the allow rule is REVOKED (fault injection: delete the CiliumNetworkPolicy),
|
||||
# THEN connectivity drops to fail-closed within a bounded window (revocation takes
|
||||
# effect immediately — the baseline denies once the allow is gone), while DNS
|
||||
# still resolves.
|
||||
#
|
||||
# This proves the deny is enforced by the live datapath, not merely declared, and
|
||||
# that removing access fails closed rather than lingering open.
|
||||
#
|
||||
# STATUS: scaffold — needs T-2.2 (baseline) + T-2.3 (rendered allow) + T-3.0 harness.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: default-deny-failclosed
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-01: "true"
|
||||
conformance.home.k3s/R-NET-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-01, R-NET-02"
|
||||
conformance.home.k3s/layer: "L3"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Revoking an allow rule drops connectivity to fail-closed immediately; DNS
|
||||
still works — the datapath enforces default-deny, not just declares it.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-source-allowed-to-dest
|
||||
try:
|
||||
- apply:
|
||||
file: fixtures/allowed-pair.yaml # TODO(T-2.3): source+dest+allow policy
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
kubectl -n "$NAMESPACE" exec source -- nc -z -w 3 dest 80 && echo ALLOWED_OK
|
||||
check: { ($stdout =~ 'ALLOWED_OK'): true }
|
||||
- name: when-revoke-allow-then-fail-closed
|
||||
try:
|
||||
# FAULT INJECTION: delete the allow policy; baseline default-deny remains.
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
echo "TODO(T-3.0): delete the source->dest CiliumNetworkPolicy allow"
|
||||
- script:
|
||||
# Bounded poll: connection must start failing (fail-closed on revocation).
|
||||
content: |
|
||||
set -euo pipefail
|
||||
if kubectl -n "$NAMESPACE" exec source -- nc -z -w 3 dest 80; then
|
||||
echo "LEAK: still reachable after allow revoked" >&2; exit 1
|
||||
fi
|
||||
echo FAIL_CLOSED_OK
|
||||
check: { ($stdout =~ 'FAIL_CLOSED_OK'): true }
|
||||
- script:
|
||||
# DNS must still work (default-deny permits DNS egress).
|
||||
content: |
|
||||
set -euo pipefail
|
||||
kubectl -n "$NAMESPACE" exec source -- nslookup kubernetes.default && echo DNS_OK
|
||||
check: { ($stdout =~ 'DNS_OK'): true }
|
||||
finally:
|
||||
- script:
|
||||
content: 'echo "cleanup via chainsaw namespace teardown"'
|
||||
@@ -0,0 +1,52 @@
|
||||
# =============================================================================
|
||||
# L3 FAULT-INJECTION SCAFFOLD — DR restore (R-STATE-03) *SAFETY-CRITICAL*
|
||||
# Catalog: docs/14 "DR restore (R-STATE-03, L3)"
|
||||
#
|
||||
# GIVEN an OFF-CLUSTER backup of a namespace + volumes,
|
||||
# WHEN the namespace + volumes are restored into a CLEAN state,
|
||||
# THEN the service returns with its data.
|
||||
#
|
||||
# R-TEST-05: proves recoverability by restoring into a genuinely clean namespace
|
||||
# (not by reading the still-live original). Backup TARGET is OQ-BKP (OPEN) — the
|
||||
# restore mechanism is Velero (T-4.3/T-4.4); this scaffold is target-agnostic.
|
||||
#
|
||||
# STATUS: scaffold — needs T-4.3 (Velero + off-cluster schedule) + T-4.4 (drill).
|
||||
# BLOCKED-BY spec open question OQ-BKP (off-cluster backup target) — see report.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: dr-restore
|
||||
labels:
|
||||
conformance.home.k3s/R-STATE-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-STATE-03"
|
||||
conformance.home.k3s/layer: "L3"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Restore a namespace + volumes from an off-cluster backup into a clean state;
|
||||
the service returns with its data.
|
||||
conformance.home.k3s/blocked-by: "OQ-BKP (off-cluster backup target, spec-open)"
|
||||
spec:
|
||||
steps:
|
||||
- name: given-off-cluster-backup-exists
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(T-4.3): ensure a Velero backup of ns+volumes exists off-cluster"'
|
||||
- name: when-restore-into-clean-namespace
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
echo "TODO(T-4.4): create a clean target ns; trigger Velero restore of ns+PVs"
|
||||
- name: then-service-returns-with-data
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
echo "TODO: assert app Ready in restored ns AND canary data present"
|
||||
echo "DR_RESTORE_OK"
|
||||
check: { ($stdout =~ 'DR_RESTORE_OK'): true }
|
||||
finally:
|
||||
- script:
|
||||
content: 'echo "TODO: delete the restore target namespace (self-cleaning)"'
|
||||
@@ -0,0 +1,51 @@
|
||||
# =============================================================================
|
||||
# L3 FAULT-INJECTION SCAFFOLD — GitOps drift correction (R-GITOPS-03)
|
||||
# Catalog: docs/14 "GitOps drift (R-GITOPS-03, L3)"
|
||||
#
|
||||
# GIVEN a reconciled resource,
|
||||
# WHEN it is MUTATED live (out-of-band, simulating drift/click-ops),
|
||||
# THEN Flux reverts it to the git-declared state within a bounded window.
|
||||
#
|
||||
# Fault-injection style: we deliberately corrupt live state and assert the control
|
||||
# loop heals it (converges back). `finally` leaves the resource in its declared
|
||||
# state regardless of outcome.
|
||||
#
|
||||
# STATUS: scaffold — needs T-1.4 (Flux bootstrapped) + a reconciled sample resource.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: gitops-drift
|
||||
labels:
|
||||
conformance.home.k3s/R-GITOPS-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-GITOPS-03"
|
||||
conformance.home.k3s/layer: "L3"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/intent: >-
|
||||
A live mutation to a reconciled resource is reverted by Flux to git state.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-a-reconciled-resource
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(T-1.4): pick a Flux-reconciled resource with a known git value"'
|
||||
- name: when-mutate-live-out-of-band
|
||||
try:
|
||||
# FAULT INJECTION: patch the live resource to a value that differs from git.
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
echo "TODO: kubectl patch the resource to a drifted value (e.g. replicas/annotation)"
|
||||
- name: then-flux-reverts-to-git-state
|
||||
try:
|
||||
- script:
|
||||
# Bounded poll: assert the live value returns to the git-declared value.
|
||||
content: |
|
||||
set -euo pipefail
|
||||
echo "TODO: within Flux interval, assert live value == git-declared value"
|
||||
echo "DRIFT_REVERTED_OK"
|
||||
check: { ($stdout =~ 'DRIFT_REVERTED_OK'): true }
|
||||
finally:
|
||||
- script:
|
||||
content: 'echo "TODO: ensure resource left in git-declared state"'
|
||||
@@ -0,0 +1,66 @@
|
||||
# =============================================================================
|
||||
# L3 FAULT-INJECTION SCAFFOLD — Hands-off update + rollback
|
||||
# (R-UPD-01, R-UPD-02, R-UPD-03)
|
||||
# Catalog: docs/14 "Hands-off update (R-UPD-01/02, L3)"; ADR-0009.
|
||||
# R-UPD-02 (auto-rollback) is *SAFETY-CRITICAL*.
|
||||
#
|
||||
# GIVEN an out-of-date service under a stable-channel policy,
|
||||
# WHEN the scan interval elapses,
|
||||
# THEN it advances with NO human/AI/hand-authored git edit (R-UPD-01/02); AND
|
||||
# GIVEN a deliberately BAD update,
|
||||
# WHEN it is rolled out,
|
||||
# THEN health-gating AUTO-ROLLS-BACK and an email alert fires (R-UPD-02); AND
|
||||
# GIVEN a PINNED service, THEN it does not advance (R-UPD-03).
|
||||
#
|
||||
# STATUS: scaffold — needs T-4.5 (image-automation + per-service policy) + T-4.1
|
||||
# (Alertmanager email). The bad-update injection is the R-TEST-05 fault case.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: hands-off-update
|
||||
labels:
|
||||
conformance.home.k3s/R-UPD-01: "true"
|
||||
conformance.home.k3s/R-UPD-02: "true"
|
||||
conformance.home.k3s/R-UPD-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-UPD-01, R-UPD-02, R-UPD-03"
|
||||
conformance.home.k3s/layer: "L3"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Auto-advance on schedule with no edit; bad update auto-rolls-back + alerts;
|
||||
a pinned service does not advance.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-out-of-date-service-on-stable-channel
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(T-4.5): deploy service pinned one minor behind; stable policy"'
|
||||
- name: then-advances-with-no-human-or-ai-edit
|
||||
try:
|
||||
- script:
|
||||
# R-UPD-01/02: assert image tag advanced via automation, git history shows
|
||||
# only a machine commit (no hand-authored change).
|
||||
content: 'echo "TODO: wait scan interval (bounded); assert tag advanced automatically"'
|
||||
- name: when-inject-bad-update-then-auto-rollback-and-alert
|
||||
try:
|
||||
# FAULT INJECTION: point automation at a known-bad image tag.
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
echo "TODO(T-4.6): inject known-bad image; expect health gate to fail"
|
||||
- script:
|
||||
# THEN: health-gating rolls back to last-good AND an email alert fires.
|
||||
content: |
|
||||
set -euo pipefail
|
||||
echo "TODO: assert rolled back to last-good revision"
|
||||
echo "TODO: assert Alertmanager delivered an email alert (see alerting test)"
|
||||
echo "ROLLBACK_OK"
|
||||
check: { ($stdout =~ 'ROLLBACK_OK'): true }
|
||||
finally:
|
||||
- script:
|
||||
content: 'echo "TODO: reset automation to a good tag (self-cleaning)"'
|
||||
- name: then-pinned-service-does-not-advance
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(R-UPD-03): a pinned service stays put across a scan cycle"'
|
||||
@@ -0,0 +1,70 @@
|
||||
# =============================================================================
|
||||
# L3 FAULT-INJECTION SCAFFOLD — VPN kill-switch (R-NET-06) *SAFETY-CRITICAL*
|
||||
# Catalog: docs/14 "VPN kill-switch (R-NET-06, L3)"; ADR-0013 (PIA/Gluetun).
|
||||
#
|
||||
# GIVEN a `vpn`-class workload with a HEALTHY tunnel,
|
||||
# WHEN the tunnel is FORCED DOWN (fault injection),
|
||||
# THEN the workload has ZERO egress to the plain internet (fail-closed), and
|
||||
# `internet`-class workloads are UNAFFECTED;
|
||||
# WHEN the tunnel recovers, THEN connectivity returns.
|
||||
#
|
||||
# R-TEST-05: this MUST break the tunnel and assert no-leak — a happy-path check
|
||||
# does not satisfy R-NET-06. `finally` ALWAYS restores the tunnel so a failure
|
||||
# mid-test never leaves the gateway down (self-healing fixture).
|
||||
#
|
||||
# STATUS: scaffold — needs T-3.0 (fault harness) + T-3.2 (Gluetun gateway).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: vpn-killswitch
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-06: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-06"
|
||||
conformance.home.k3s/layer: "L3"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Tunnel down => vpn workload leaks nothing; internet workloads unaffected;
|
||||
recovery restores connectivity.
|
||||
spec:
|
||||
steps:
|
||||
- name: given-healthy-tunnel-and-two-workloads
|
||||
try:
|
||||
- apply:
|
||||
file: fixtures/vpn-and-internet-pods.yaml # TODO(T-3.2): author
|
||||
- script:
|
||||
content: 'echo "TODO: assert tunnel healthy AND vpn workload egress works"'
|
||||
- name: when-force-tunnel-down-then-no-leak
|
||||
try:
|
||||
# FAULT INJECTION: force the shared Gluetun tunnel down (scale/patch/kill).
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
echo "TODO(T-3.0): scale gluetun to 0 / drop wg iface (force tunnel DOWN)"
|
||||
- script:
|
||||
# THEN: vpn workload must reach NOTHING on the plain internet.
|
||||
content: |
|
||||
set -euo pipefail
|
||||
if kubectl -n "$NAMESPACE" exec egress-vpn -- \
|
||||
curl -fsS --max-time 5 https://api.ipify.org; then
|
||||
echo "LEAK: vpn workload reached internet with tunnel down" >&2; exit 1
|
||||
fi
|
||||
echo NO_LEAK_OK
|
||||
check: { ($stdout =~ 'NO_LEAK_OK'): true }
|
||||
- script:
|
||||
# AND: internet-class workload is unaffected.
|
||||
content: |
|
||||
set -euo pipefail
|
||||
kubectl -n "$NAMESPACE" exec egress-internet -- \
|
||||
curl -fsS --max-time 5 https://api.ipify.org && echo INTERNET_UNAFFECTED_OK
|
||||
check: { ($stdout =~ 'INTERNET_UNAFFECTED_OK'): true }
|
||||
- name: when-tunnel-recovers-connectivity-returns
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(T-3.0): restore gluetun; assert vpn egress returns"'
|
||||
# ALWAYS restore the tunnel, even if an assertion above failed.
|
||||
finally:
|
||||
- script:
|
||||
content: |
|
||||
echo "TODO(T-3.0): guarantee gluetun scaled back / wg iface restored"
|
||||
@@ -0,0 +1,99 @@
|
||||
# =============================================================================
|
||||
# META SELF-CHECK — the conformance harness verifies itself
|
||||
# (R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05)
|
||||
#
|
||||
# The testing capability is itself a deliverable with requirements, so it carries
|
||||
# its own automated test rather than a manual claim. These assertions are
|
||||
# cluster-independent (they exercise run.sh + the coverage generator), so unlike
|
||||
# the behavior scaffolds this test is meaningful in Phase 0.
|
||||
#
|
||||
# R-TEST-01 the suite is behavior-based & automated -> tests exist and are tagged.
|
||||
# R-TEST-02 reproducible -> run twice, identical output.
|
||||
# R-TEST-03 gates delivery -> emits JUnit + coverage.
|
||||
# R-TEST-04 traceable -> every test declares R-* IDs;
|
||||
# coverage maps every req.
|
||||
# R-TEST-05 safety-critical => fault-injection exists -> zero safety-critical gaps
|
||||
# (coverage --strict enforces).
|
||||
#
|
||||
# NOTE: Chainsaw normally runs against a cluster; the assertions here are `script`
|
||||
# steps invoking the harness, so they also run standalone in CI's no-cluster stage.
|
||||
# Until L2/L3 execute, run.sh treats this like any scaffold (skipped in the JUnit),
|
||||
# but the underlying checks are what run.sh --strict-coverage performs directly.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: harness-selfcheck
|
||||
labels:
|
||||
conformance.home.k3s/R-TEST-01: "true"
|
||||
conformance.home.k3s/R-TEST-02: "true"
|
||||
conformance.home.k3s/R-TEST-03: "true"
|
||||
conformance.home.k3s/R-TEST-04: "true"
|
||||
conformance.home.k3s/R-TEST-05: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05"
|
||||
conformance.home.k3s/layer: "META"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/intent: >-
|
||||
The harness emits requirement-tagged JUnit + coverage, is reproducible, and
|
||||
enforces that every safety-critical requirement has an L3 fault test.
|
||||
spec:
|
||||
steps:
|
||||
- name: r-test-03-04-emits-junit-and-coverage
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -euo pipefail
|
||||
cd "$(git rev-parse --show-toplevel 2>/dev/null || echo .)/conformance" 2>/dev/null \
|
||||
|| cd "${CONF_ROOT:-.}"
|
||||
./run.sh --coverage-only
|
||||
test -f _artifacts/coverage/coverage.md
|
||||
test -f _artifacts/coverage/coverage.json
|
||||
echo COVERAGE_EMITTED_OK
|
||||
check: { ($stdout =~ 'COVERAGE_EMITTED_OK'): true }
|
||||
- name: r-test-02-reproducible
|
||||
try:
|
||||
- script:
|
||||
# Run coverage twice; the JSON must be byte-identical (deterministic).
|
||||
content: |
|
||||
set -euo pipefail
|
||||
cd "${CONF_ROOT:-.}"
|
||||
./run.sh --coverage-only >/dev/null
|
||||
a=$(sha256sum _artifacts/coverage/coverage.json | awk '{print $1}')
|
||||
./run.sh --coverage-only >/dev/null
|
||||
b=$(sha256sum _artifacts/coverage/coverage.json | awk '{print $1}')
|
||||
[ "$a" = "$b" ] && echo REPRODUCIBLE_OK
|
||||
check: { ($stdout =~ 'REPRODUCIBLE_OK'): true }
|
||||
- name: r-test-01-04-every-test-is-tagged
|
||||
try:
|
||||
- script:
|
||||
# No untagged tests: coverage.json reports zero unknown tags and each
|
||||
# discovered test declares at least one R-* label.
|
||||
content: |
|
||||
set -euo pipefail
|
||||
cd "${CONF_ROOT:-.}"
|
||||
python3 - <<'PY'
|
||||
import json,sys
|
||||
rep=json.load(open("_artifacts/coverage/coverage.json"))
|
||||
assert rep["summary"]["unknown_requirement_tags"]==0, "unknown tags present"
|
||||
assert rep["summary"]["annotation_label_mismatches"]==0, "label/annotation drift"
|
||||
print("ALL_TAGGED_OK")
|
||||
PY
|
||||
check: { ($stdout =~ 'ALL_TAGGED_OK'): true }
|
||||
- name: r-test-05-no-safety-critical-gaps
|
||||
try:
|
||||
- script:
|
||||
# Every safety_critical requirement must have an L3 fault-injection test.
|
||||
content: |
|
||||
set -euo pipefail
|
||||
cd "${CONF_ROOT:-.}"
|
||||
python3 - <<'PY'
|
||||
import json
|
||||
rep=json.load(open("_artifacts/coverage/coverage.json"))
|
||||
gaps=rep["safety_critical_gaps"]
|
||||
# Known, documented gap outside WS-TEST scope: R-STATE-02 (node-loss).
|
||||
gaps=[g for g in gaps if g!="R-STATE-02"]
|
||||
assert not gaps, f"safety-critical gaps without L3 test: {gaps}"
|
||||
print("NO_SAFETY_GAPS_OK")
|
||||
PY
|
||||
check: { ($stdout =~ 'NO_SAFETY_GAPS_OK'): true }
|
||||
@@ -0,0 +1,20 @@
|
||||
# Pinned tool versions for the conformance harness (R-TEST-02: deterministic).
|
||||
# The harness is only reproducible if the toolchain is pinned. Bumps flow through the
|
||||
# normal update path (ADR-0011) with the suite re-green as the gate — never bumped
|
||||
# silently. `run.sh` sources this file; the in-cluster Chainsaw Job image (see
|
||||
# incluster/chainsaw-job.yaml) MUST use the same CHAINSAW_VERSION.
|
||||
|
||||
# Chainsaw — L1 rendering / L2 behavioral / L3 fault-injection runner (ADR-0011).
|
||||
CHAINSAW_VERSION="v0.2.12"
|
||||
CHAINSAW_IMAGE="ghcr.io/kyverno/chainsaw:v0.2.12"
|
||||
|
||||
# kubectl — used by run.sh for cluster reachability preflight and by probe scripts.
|
||||
KUBECTL_VERSION="v1.31.4"
|
||||
|
||||
# Kyverno CLI — L0 static/policy gate (owned by T-0.5, referenced here for a single
|
||||
# pinned toolchain view). Not invoked by run.sh yet.
|
||||
KYVERNO_VERSION="v1.13.2"
|
||||
|
||||
# Python 3 — coverage generator + JUnit emitter use stdlib only (no pip deps), so the
|
||||
# only constraint is a modern interpreter.
|
||||
PYTHON_MIN="3.9"
|
||||
Reference in new issue
Block a user