diff --git a/conformance/.gitignore b/conformance/.gitignore new file mode 100644 index 0000000..6cba6bd --- /dev/null +++ b/conformance/.gitignore @@ -0,0 +1,7 @@ +# Generated artifacts — regenerated by run.sh on every run; never committed. +# CI collects these from the workspace / results PVC; they are inputs to the gate, +# not source. (Root .gitignore also covers *.junit.xml and coverage/.) +_artifacts/ + +# Downloaded pinned tool binaries (run.sh may fetch chainsaw/kubectl here). +.bin/ diff --git a/conformance/README.md b/conformance/README.md index 2251d00..28f6e5c 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -1,3 +1,111 @@ -# conformance +# conformance/ — the home-k3s conformance harness (WS-TEST) -_Placeholder — populated in Phase 0. See ../home-network-k3s (read-only spec)._ +This subtree is the **definition of "done"** for the whole platform. Nothing is accepted +on a demo, a screenshot, or "it reconciled": every requirement is proven by an +**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it +verifies (spec [`docs/14`](../../home-network-k3s/docs/14-testing-and-conformance.md), +[ADR-0010](../../home-network-k3s/decisions/adr-0010-conformance-testing.md), +[ADR-0011](../../home-network-k3s/decisions/adr-0011-test-tooling-kyverno-chainsaw.md)). + +Tooling is fixed by ADR-0011: **Kyverno** for L0 (the CI policy gate + in-cluster +admission enforcer — owned by task T-0.5) and **Chainsaw** for **L1 rendering / L2 +behavioral / L3 fault-injection** (this harness). + +## The single command + +```bash +./run.sh # full suite (skips L1/L2/L3 if no cluster) +./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged) +./run.sh --layer l2 # one layer: l1 | l2 | l3 | meta | all +./run.sh --coverage-only # regenerate the coverage report (no cluster) +./run.sh --lint # structural-validate test defs (no cluster) +./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift +./run.sh --require-all # phase-exit gate: every requirement must be covered +``` + +**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks: +- `--strict-coverage` — **hygiene** (always-on, WS-TEST owns it): fails only on + tag/annotation drift or a tag absent from the registry. Uncovered future-phase + requirements are *visible gaps in the report*, not a harness failure. +- `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts + those touched requirements are covered (a listed safety-critical one must have an L3 test). +- `--require-all` — **phase-exit gate**: every registry requirement must be covered. +- Documented out-of-scope gaps (e.g. `R-STATE-02` node-loss, owned by WS-STORAGE) are + exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as + `DOCUMENTED_GAPS` so the exemption is explicit and reviewable. + +`run.sh` is the **only** entrypoint. Local runs and the in-cluster CI Chainsaw Job call +the *same* script with the same config, so results are identical. + +### Artifacts (always emitted, under `_artifacts/`, git-ignored) +| Path | What | Consumer | +|---|---|---| +| `_artifacts/report.junit.xml` | JUnit — Chainsaw results, or a **skipped-scaffold** shim when no cluster | CI gate | +| `_artifacts/coverage/coverage.md` | Human requirement-coverage report | PR reviewers / logs | +| `_artifacts/coverage/coverage.json` | Machine coverage: `{requirement: [tests]}`, gaps, safety gaps | CI gate | + +## Reproducibility contract (R-TEST-02 — what the harness guarantees) +- **Single command.** `./run.sh` (or a tagged/layer subset) runs everything and emits the + artifacts above — locally and in CI. +- **Hermetic.** Each Chainsaw test provisions its own namespace + fixtures/probes and tears + them down (`config/chainsaw-config.yaml` auto-creates/auto-deletes the namespace; tests + add `finally` for anything out-of-namespace). It never relies on pre-existing state. +- **Deterministic.** Tool versions pinned in [`versions.env`](versions.env); eventual + consistency handled by **bounded polling with explicit timeouts** (config), never sleeps + or wall-clock/ordering assumptions. +- **Isolated & parallel-safe.** Unique generated namespaces + per-requirement labels; tests + run concurrently without cross-talk. +- **Machine-readable & requirement-tagged.** JUnit + coverage, every result carrying its + `R-*` tags (convention in [`tests/README.md`](tests/README.md)). +- **Self-cleaning.** Fixtures removed on pass **or** fail; a leaked namespace/fixture is + itself a failure (`skipDelete: false`, enforced delete/cleanup timeouts). +- **Versioned with the code.** Tests live next to what they verify and change only when the + human-owned spec they encode changes — **never** weakened to go green (R-TEST-06). + +## Layout +``` +run.sh single-command runner (the entrypoint) +versions.env pinned toolchain (Chainsaw/kubectl/Kyverno) — determinism +config/chainsaw-config.yaml shared execution contract (timeouts, ns isolation, JUnit) +requirements/registry.yaml mirror of EVERY spec R-* ID (coverage enumerates from here) +lib/ + common.sh tagging convention + test discovery/selection helpers + coverage.py requirement-coverage generator (stdlib only) + emit_skipped_junit.py Phase-0 skipped-JUnit shim (no cluster) +tests/ + l1-rendering/ l2-behavioral/ l3-fault-injection/ meta/ (see tests/README.md) +incluster/ + chainsaw-job.yaml rbac.yaml in-cluster L2/L3 Job + scoped test SA (ADR-0011) +``` + +## How CI invokes it +CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**: + +1. **No-cluster stage (every PR, fast):** `./run.sh --lint` and `./run.sh --coverage-only + --strict-coverage --requirements `. Validates every test parses + + is tagged, regenerates coverage, **fails the PR** on tag/annotation drift (hygiene) + and on any *touched* requirement lacking a test (merge gate). (L0 Kyverno CLI runs + alongside — task T-0.5.) +2. **Cluster stage (L2/L3):** CI does **not** hold a kubeconfig. It **triggers the + in-cluster Chainsaw Job** ([`incluster/`](incluster/README.md)), which runs `./run.sh` + under the scoped `conformance-runner` ServiceAccount and writes JUnit + coverage to a + results volume CI collects. + - **Merge gate:** the Job runs `--requirements `; that subset + runs green **and** those requirements are covered to merge (R-TEST-03). + - **Phase-exit gate:** the Job runs `--layer all --strict-coverage`; add `--require-all` + only at the milestone where the full requirement set is expected covered. The phase + suite must be green before the phase's exit criteria are met. + +## Phase 0 status (no cluster yet) +No cluster exists, so L1/L2/L3 **cannot execute**. The behavior tests are **valid +scaffolds** (steps + assertions authored, tagged, `TODO(T-x.y)` where a later phase +delivers the fixture). In this phase `run.sh`: +- always generates the **coverage report** (cluster-independent), and +- emits a **skipped** JUnit listing every scaffold (so the gate is green and the artifact + shape is real), and +- the **`meta/harness-selfcheck`** checks (reproducible output, JUnit+coverage emitted, + everything tagged, zero safety-critical gaps) are what `run.sh --strict-coverage` + actually enforces today. + +As later phases deliver capabilities, each scaffold's `TODO`s are filled and the test flips +from *skipped* to *executing green* — landing coverage without re-architecting the harness. diff --git a/conformance/config/chainsaw-config.yaml b/conformance/config/chainsaw-config.yaml new file mode 100644 index 0000000..6fc133d --- /dev/null +++ b/conformance/config/chainsaw-config.yaml @@ -0,0 +1,49 @@ +# Chainsaw Configuration for the home-k3s conformance suite (ADR-0011). +# Passed to every invocation via `chainsaw test --config config/chainsaw-config.yaml` +# so local runs and the in-cluster CI Job behave identically (R-TEST-02). +# +# Reproducibility knobs live here, NOT in individual tests, so the whole suite shares +# one deterministic execution contract: +# * bounded timeouts (no sleep-and-hope; eventual consistency via polled assert/error) +# * per-test ephemeral namespace, auto-created and auto-deleted (hermetic, isolated) +# * a leaked fixture is a failure: del/cleanup timeouts are enforced, skipDelete=false +apiVersion: chainsaw.kyverno.io/v1alpha2 +kind: Configuration +metadata: + name: home-k3s-conformance +spec: + # -- Deterministic bounded waits. Eventual consistency is handled by polling within + # these windows (assert/error retry until deadline), never by fixed sleeps. + timeouts: + apply: 90s + assert: 180s + error: 60s + delete: 120s + cleanup: 120s + exec: 180s + + # -- Isolation & parallel-safety: each test gets its own generated namespace so tests + # never collide and can run concurrently. + namespace: + # Unique per test run; prefix makes leaked namespaces obvious in `kubectl get ns`. + template: + metadata: + generateName: chainsaw-conf- + + # -- Concurrency. Tests are hermetic, so they parallelize safely; keep modest to bound + # cluster load and keep fault-injection tests from contending. + execution: + parallel: 4 + # Fail the whole run if any test leaks a namespace/fixture (reproducibility contract). + failFast: false + + # -- Self-cleaning: never skip delete; deletion is part of the pass criteria. + cleanup: + skipDelete: false + + # -- Machine-readable output. run.sh normalizes the emitted file to + # _artifacts/report.junit.xml and layers the requirement-coverage report on top. + report: + format: JUNIT-TEST + name: chainsaw-report + path: ../_artifacts diff --git a/conformance/incluster/README.md b/conformance/incluster/README.md new file mode 100644 index 0000000..6710a32 --- /dev/null +++ b/conformance/incluster/README.md @@ -0,0 +1,32 @@ +# In-cluster conformance Job (L2/L3) + +Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a +cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** +on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster +credentials — the Job's own scoped ServiceAccount does (R-SEC-02). + +## Files +- `rbac.yaml` — the `conformance` namespace, the `conformance-runner` ServiceAccount, and + a **least-privilege ClusterRole skeleton**. Rules start **empty** and are added — each + commented with the test/requirement that needs it — only as the matching fault-injection + test lands. This is a **privileged test identity**: never reused by workloads, its own + audit item. +- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to + `--requirements ` (merge gate) or `--layer all --strict-coverage` (phase + exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`. + +## Flow +1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a + trigger with narrow rights: create Job + read its logs/status — *not* a broad + kubeconfig). +2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs + `./run.sh` under `conformance-runner`. +3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` + + `coverage/` to the results volume (`/workspace/conformance/_artifacts`). +4. CI collects those artifacts (results volume or the terminating log stream, which echoes + their paths) and enforces the merge / phase-exit gate on them. + +## Phase 0 +No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux +from Phase 1+ and first exercised when L2/L3 can run (T-3.0 onward). In Phase 0 the same +`run.sh` runs in CI's no-cluster stage and emits a **skipped** JUnit + the coverage report. diff --git a/conformance/incluster/chainsaw-job.yaml b/conformance/incluster/chainsaw-job.yaml new file mode 100644 index 0000000..5821d30 --- /dev/null +++ b/conformance/incluster/chainsaw-job.yaml @@ -0,0 +1,74 @@ +# ============================================================================= +# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14) +# +# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO +# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner` +# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that +# CI collects, and enforces the merge / phase-exit gate on the outcome. +# +# Two invocation shapes (CI sets args): +# * MERGE GATE: ./run.sh --requirements (subset) +# * PHASE EXIT: ./run.sh --layer all (full suite) +# +# Results artifact mechanism (delegated to WS-TEST by docs/14 open items): +# Chosen = a shared results emptyDir + a terminating log stream, with an +# OPTIONAL results PVC for durability across CI collection. For Phase 0 there is +# no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised +# once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the +# results volume (or scrapes the pod log, which echoes their paths). +# +# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE. +# ============================================================================= +apiVersion: batch/v1 +kind: Job +metadata: + generateName: conformance-run- + namespace: conformance + labels: + app.kubernetes.io/part-of: conformance +spec: + backoffLimit: 0 # deterministic: a failed conformance run is a failure, no retry + ttlSecondsAfterFinished: 3600 + template: + metadata: + labels: + app.kubernetes.io/part-of: conformance + spec: + serviceAccountName: conformance-runner # scoped test identity (rbac.yaml) + restartPolicy: Never + securityContext: + runAsNonRoot: true + seccompProfile: { type: RuntimeDefault } + containers: + - name: chainsaw + # PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12). + image: ghcr.io/kyverno/chainsaw:v0.2.12 + workingDir: /workspace/conformance + # The suite repo is delivered into /workspace (git clone init-container or + # a Flux-populated volume). run.sh is the single entrypoint (identical to local). + command: ["./run.sh"] + # CI overrides args per invocation shape: + # ["--requirements", "R-NET-01,R-NET-06"] (merge gate) OR + # ["--layer", "all", "--strict-coverage"] (phase exit) + args: ["--layer", "all", "--strict-coverage"] + env: + - name: CONF_ROOT + value: /workspace/conformance + volumeMounts: + - { name: workspace, mountPath: /workspace } + - { name: results, mountPath: /workspace/conformance/_artifacts } + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: { drop: ["ALL"] } + runAsNonRoot: true + volumes: + # Suite source: populated by an init-container git clone of the impl repo at + # the commit under test, OR a Flux-synced source volume. (Init container + # omitted from this skeleton; wire in T-6.1b.) + - name: workspace + emptyDir: {} + # Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir + # for a results PVC if durability beyond pod lifetime is required. + - name: results + emptyDir: {} diff --git a/conformance/incluster/rbac.yaml b/conformance/incluster/rbac.yaml new file mode 100644 index 0000000..5ddcdc7 --- /dev/null +++ b/conformance/incluster/rbac.yaml @@ -0,0 +1,114 @@ +# ============================================================================= +# Scoped test ServiceAccount + RBAC for the in-cluster Chainsaw conformance Job +# (ADR-0011; docs/14 "Where each layer runs"; R-SEC-02 least-privilege). +# +# SKELETON / PLACEHOLDER — RBAC is deliberately LEFT EMPTY here and authored +# least-privilege as each fault-injection capability actually lands (T-3.0+). The +# rule is: grant nothing until a committed test needs it, then grant exactly that +# verb/resource, with a comment naming the test that requires it. This SA is a +# PRIVILEGED TEST IDENTITY — it is its own audit item (R-SEC-02): it MUST NOT be +# reused by workloads, and every widening is reviewed. +# +# CI holds NO cluster credentials. CI only triggers the Job (T-6.1b via Gitea +# Actions); the Job runs under THIS ServiceAccount, so cluster access stays +# in-cluster and scoped (docs/14, R-SEC-02). +# +# Deployment: reconciled by Flux into the `conformance` namespace once the cluster +# exists (Phase 1+). Not applied in Phase 0 (no cluster). +# ============================================================================= +apiVersion: v1 +kind: Namespace +metadata: + name: conformance + labels: + app.kubernetes.io/part-of: conformance + # The test namespace itself is exempt from nothing: the platform default-deny + # baseline still applies; the Job opens only the egress it declares. + pod-security.kubernetes.io/enforce: restricted +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: conformance-runner + namespace: conformance + labels: + app.kubernetes.io/part-of: conformance + annotations: + conformance.home.k3s/audit: >- + PRIVILEGED TEST IDENTITY. Scoped least-privilege for fault-injection only. + Never reuse for workloads. Widen only alongside the test that needs it. +--- +# ClusterRole — the union of what BEHAVIOR + FAULT tests need. Every rule is +# commented with the test/requirement that justifies it. Start empty; uncomment / +# add rules as the corresponding L2/L3 tests are delivered (do NOT pre-grant). +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: conformance-runner + labels: + app.kubernetes.io/part-of: conformance +rules: [] + # ---- Hermetic fixtures (all L2/L3): create/delete throwaway test namespaces + fixtures + # - apiGroups: [""] + # resources: [namespaces] + # verbs: [create, delete, get, list, watch] + # - apiGroups: [""] + # resources: [pods, services, configmaps, secrets, persistentvolumeclaims] + # verbs: [create, delete, get, list, watch, patch] + # - apiGroups: [""] + # resources: [pods/exec] + # verbs: [create] # probe exec for behavior assertions + # + # ---- R-NET-01/02 (default-deny, deny-undeclared): apply/patch/delete allow policies + # - apiGroups: [cilium.io] + # resources: [ciliumnetworkpolicies] + # verbs: [create, delete, get, list, patch] + # + # ---- R-NET-06 (VPN kill-switch): force the shared Gluetun gateway down/up + # - apiGroups: [apps] + # resources: [deployments, deployments/scale] + # resourceNames: [gluetun-gateway] # SCOPE to the gateway only + # verbs: [get, patch, update] + # + # ---- R-STATE-02 (node loss): cordon/drain/uncordon a node (NOT in WS-TEST scope yet) + # - apiGroups: [""] + # resources: [nodes] + # verbs: [get, list, patch] # cordon = patch spec.unschedulable + # - apiGroups: [""] + # resources: [pods/eviction] + # verbs: [create] # drain + # + # ---- R-UPD-02 (bad-update auto-rollback): inject a bad image tag via automation + # - apiGroups: [image.toolkit.fluxcd.io] + # resources: [imagepolicies, imagerepositories] + # verbs: [get, patch] + # + # ---- R-STATE-03 (DR restore): trigger a Velero restore + # - apiGroups: [velero.io] + # resources: [restores, backups] + # verbs: [create, get, list, watch] + # + # ---- R-GITOPS-03 (drift): mutate a reconciled resource + observe Flux revert + # - apiGroups: [kustomize.toolkit.fluxcd.io] + # resources: [kustomizations] + # verbs: [get, list, watch] # observe only; do NOT grant reconcile suspension + # + # ---- R-PLAT-03 (VM lifecycle): create/start/stop/destroy VMs + # - apiGroups: [kubevirt.io] + # resources: [virtualmachines, virtualmachineinstances] + # verbs: [create, delete, get, list, patch] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: conformance-runner + labels: + app.kubernetes.io/part-of: conformance +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: conformance-runner +subjects: + - kind: ServiceAccount + name: conformance-runner + namespace: conformance diff --git a/conformance/lib/common.sh b/conformance/lib/common.sh new file mode 100644 index 0000000..2ba0e77 --- /dev/null +++ b/conformance/lib/common.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# Shared helpers for the conformance harness. Sourced by run.sh; not executed directly. +# shellcheck shell=bash + +# --- Requirement-tagging convention (single source of truth) -------------------------- +# A Chainsaw Test declares the requirement(s) it verifies with a per-requirement label: +# +# metadata: +# labels: +# conformance.home.k3s/R-NET-01: "true" # one label per verified R-* ID +# annotations: +# conformance.home.k3s/requirements: "R-NET-01, R-NET-02" # human summary +# conformance.home.k3s/layer: "L2" +# conformance.home.k3s/safety-critical: "false" +# +# The LABEL is authoritative and does double duty: +# * `chainsaw test --selector conformance.home.k3s/R-NET-01=true` runs the subset. +# * lib/coverage.py reads the same labels to map requirement -> verifying test(s). +# The annotation is a convenience for humans/reports and MUST agree with the labels +# (coverage.py flags a mismatch). This is the convention referenced in README.md. +export CONF_LABEL_PREFIX="conformance.home.k3s" + +# Absolute path to the conformance root (dir containing run.sh). Set by run.sh. +: "${CONF_ROOT:?CONF_ROOT must be set before sourcing common.sh}" + +# --- Test discovery ------------------------------------------------------------------- +# Every leaf test is a directory containing a `chainsaw-test.yaml`. +conf_all_test_files() { + find "${CONF_ROOT}/tests" -type f -name 'chainsaw-test.yaml' 2>/dev/null | sort +} + +# Directories of tests that carry ANY of the given requirement labels (space-separated +# R-IDs). Used to translate `--requirements R-NET-01,R-NET-06` into a set of test dirs +# so we run each matching test exactly once (union, deduplicated). +conf_dirs_for_requirements() { + local reqs="$1" f dir rid matched + while IFS= read -r f; do + [ -n "$f" ] || continue + matched=0 + for rid in $reqs; do + # Match a label line: conformance.home.k3s/R-NET-01: "true" + if grep -Eq "^[[:space:]]*${CONF_LABEL_PREFIX}/${rid}:[[:space:]]*[\"']?true" "$f"; then + matched=1; break + fi + done + [ "$matched" = 1 ] && dirname "$f" + done < <(conf_all_test_files) | sort -u +} + +# Directories for a given layer subtree: l1 | l2 | l3 | meta | all +conf_dirs_for_layer() { + local layer="$1" sub + case "$layer" in + all) find "${CONF_ROOT}/tests" -type f -name 'chainsaw-test.yaml' -exec dirname {} \; | sort -u ;; + l1) sub="l1-rendering" ;; + l2) sub="l2-behavioral" ;; + l3) sub="l3-fault-injection" ;; + meta) sub="meta" ;; + *) echo "unknown layer: $layer" >&2; return 2 ;; + esac + if [ "$layer" != all ]; then + find "${CONF_ROOT}/tests/${sub}" -type f -name 'chainsaw-test.yaml' -exec dirname {} \; 2>/dev/null | sort -u + fi +} + +# Normalize a comma/space separated requirement list to space-separated, uppercased. +conf_normalize_reqs() { + echo "$1" | tr ',' ' ' | tr '[:lower:]' '[:upper:]' | xargs +} + +# Is a live cluster reachable? (Preflight for L1/L2/L3 execution.) +conf_cluster_reachable() { + command -v kubectl >/dev/null 2>&1 || return 1 + kubectl cluster-info >/dev/null 2>&1 +} diff --git a/conformance/lib/coverage.py b/conformance/lib/coverage.py new file mode 100755 index 0000000..39bacc7 --- /dev/null +++ b/conformance/lib/coverage.py @@ -0,0 +1,319 @@ +#!/usr/bin/env python3 +"""Requirement-coverage report generator (R-TEST-04). + +Maps EVERY R-* requirement in the authoritative registry to the Chainsaw test(s) that +verify it, and flags requirements with no verifying test as visible, blocking gaps. + +Single source of truth for coverage is the per-requirement label on each Chainsaw Test: + + metadata: + labels: + conformance.home.k3s/R-NET-01: "true" + +Reads: + * requirements/registry.yaml -- the mirror of every spec R-* ID (see that file). + * tests/**/chainsaw-test.yaml -- discovers labels + the requirements annotation. + +Writes (under --out, default _artifacts/coverage/): + * coverage.json -- machine-readable {requirement: [tests...]}, gaps, safety gaps. + * coverage.md -- human-readable table for PRs / CI logs. + +Exit status / gating (R-TEST-03 gates are *scoped*, not "all 51 always"): + 0 always in plain report mode. + --strict HYGIENE gate (always-on, WS-TEST owns it): fails on tag/annotation + drift or tags absent from the registry. Does NOT fail merely because + a future-phase requirement has no test yet — those are visible gaps, + not harness defects. + --require IDs SCOPED completeness gate (merge gate): fails if any listed requirement + is uncovered, or a listed safety-critical requirement lacks an L3 test. + CI passes the requirement IDs the change touches. + --require-all FULL completeness gate (phase-exit / final): fails if ANY registry + requirement is uncovered or any safety-critical one lacks an L3 test. + --allow-gap IDs exempt specific IDs from the completeness gates (documented, out-of- + scope gaps such as R-STATE-02 node-loss owned by another workstream). + +Pure standard library (no PyYAML) so the harness has no pip dependency: the registry and +the test label/annotation lines we care about are parsed with line-oriented regexes, and +the file formats are constrained (documented in registry.yaml / common.sh) to keep that +parse deterministic. +""" +from __future__ import annotations + +import argparse +import json +import os +import re +import sys +from pathlib import Path + +LABEL_PREFIX = "conformance.home.k3s" +RID_RE = re.compile(r"R-[A-Z0-9]+-[0-9]+") +LABEL_LINE_RE = re.compile( + r"^\s*" + re.escape(LABEL_PREFIX) + r"/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true", re.I +) + + +# --------------------------------------------------------------------------- registry +def parse_registry(path: Path) -> list[dict]: + """Parse the constrained registry.yaml (`- id:` blocks, single-line scalars).""" + reqs: list[dict] = [] + cur: dict | None = None + for raw in path.read_text().splitlines(): + line = raw.split("#", 1)[0] if raw.lstrip().startswith("#") else raw + m = re.match(r"^\s*-\s*id:\s*(\S+)", line) + if m: + if cur: + reqs.append(cur) + cur = {"id": m.group(1).strip(), "doc": "", "safety_critical": False, + "summary": ""} + continue + if cur is None: + continue + m = re.match(r"^\s+doc:\s*(.+?)\s*$", line) + if m: + cur["doc"] = m.group(1).strip().strip('"') + continue + m = re.match(r"^\s+safety_critical:\s*(true|false)\s*$", line) + if m: + cur["safety_critical"] = (m.group(1) == "true") + continue + m = re.match(r"^\s+summary:\s*(.+?)\s*$", line) + if m: + cur["summary"] = m.group(1).strip().strip('"') + if cur: + reqs.append(cur) + return reqs + + +# ------------------------------------------------------------------------------ tests +def parse_test(path: Path) -> dict: + """Extract name, verified R-IDs (from labels), annotation list, and layer.""" + text = path.read_text() + labels: set[str] = set() + for line in text.splitlines(): + m = LABEL_LINE_RE.match(line) + if m: + labels.add(m.group(1).upper()) + + ann_reqs: set[str] = set() + m = re.search(r"^\s*" + re.escape(LABEL_PREFIX) + r"/requirements:\s*(.+)$", + text, re.M) + if m: + ann_reqs = {r.upper() for r in RID_RE.findall(m.group(1))} + + layer = "" + m = re.search(r"^\s*" + re.escape(LABEL_PREFIX) + r"/layer:\s*[\"']?([LlAaMm0-9]+)", + text, re.M) + if m: + layer = m.group(1).upper() + + name = "" + m = re.search(r"^\s*name:\s*(\S+)", text, re.M) + if m: + name = m.group(1) + + # Layer inferred from path as a fallback / cross-check. + rel = str(path) + if not layer: + if "l1-rendering" in rel: + layer = "L1" + elif "l2-behavioral" in rel: + layer = "L2" + elif "l3-fault-injection" in rel: + layer = "L3" + elif "/meta/" in rel: + layer = "META" + + return { + "name": name, + "path": rel, + "labels": sorted(labels), + "annotation_requirements": sorted(ann_reqs), + "layer": layer, + } + + +def discover_tests(tests_dir: Path) -> list[dict]: + return [parse_test(p) for p in sorted(tests_dir.rglob("chainsaw-test.yaml"))] + + +# ----------------------------------------------------------------------------- report +def build(registry: list[dict], tests: list[dict]) -> dict: + by_req: dict[str, list[dict]] = {r["id"]: [] for r in registry} + unknown: list[tuple[str, str]] = [] # (rid, test path) not in registry + mismatches: list[dict] = [] # label set != annotation set + + for t in tests: + lbl = set(t["labels"]) + ann = set(t["annotation_requirements"]) + if lbl != ann: + mismatches.append({"test": t["path"], "labels": sorted(lbl), + "annotation": sorted(ann)}) + for rid in lbl: + if rid in by_req: + by_req[rid].append(t) + else: + unknown.append((rid, t["path"])) + + reg_index = {r["id"]: r for r in registry} + covered, uncovered, safety_gaps = [], [], [] + for rid, req in reg_index.items(): + verifiers = by_req[rid] + if verifiers: + covered.append(rid) + else: + uncovered.append(rid) + if req["safety_critical"]: + has_l3 = any(v["layer"] == "L3" for v in verifiers) + if not has_l3: + safety_gaps.append(rid) + + return { + "summary": { + "total_requirements": len(registry), + "covered": len(covered), + "uncovered": len(uncovered), + "coverage_pct": round(100.0 * len(covered) / max(1, len(registry)), 1), + "safety_critical_gaps": len(safety_gaps), + "annotation_label_mismatches": len(mismatches), + "unknown_requirement_tags": len(unknown), + "total_tests": len(tests), + }, + "by_requirement": { + r["id"]: { + "doc": r["doc"], + "safety_critical": r["safety_critical"], + "summary": r["summary"], + "verified_by": [ + {"name": v["name"], "path": v["path"], "layer": v["layer"]} + for v in by_req[r["id"]] + ], + } + for r in registry + }, + "uncovered": sorted(uncovered), + "safety_critical_gaps": sorted(safety_gaps), + "annotation_label_mismatches": mismatches, + "unknown_requirement_tags": [{"requirement": u[0], "test": u[1]} for u in unknown], + } + + +def render_md(rep: dict) -> str: + s = rep["summary"] + out: list[str] = [] + out.append("# Requirement coverage report\n") + out.append( + f"- Requirements: **{s['total_requirements']}** | " + f"Covered: **{s['covered']}** ({s['coverage_pct']}%) | " + f"Uncovered: **{s['uncovered']}**\n" + f"- Safety-critical requirements missing an L3 fault-injection test: " + f"**{s['safety_critical_gaps']}**\n" + f"- Scaffolded tests: **{s['total_tests']}** | " + f"Annotation/label mismatches: **{s['annotation_label_mismatches']}** | " + f"Unknown tags: **{s['unknown_requirement_tags']}**\n" + ) + out.append("\n> Phase 0: tests are scaffolds. A requirement is **satisfied** only " + "when its tagged test passes green against a cluster (R-TEST-03) — this " + "report tracks *scaffold* coverage, i.e. that a verifying test exists.\n") + + out.append("\n## Coverage by requirement\n") + out.append("| Requirement | Safety | Verified by (layer) | Status |") + out.append("|---|---|---|---|") + for rid, info in rep["by_requirement"].items(): + vs = info["verified_by"] + if vs: + cell = "
".join(f"`{v['name']}` ({v['layer']})" for v in vs) + status = "covered" + else: + cell = "—" + status = "**GAP**" + sc = "yes" if info["safety_critical"] else "" + out.append(f"| {rid} | {sc} | {cell} | {status} |") + + if rep["uncovered"]: + out.append("\n## Uncovered requirements (gaps)\n") + for rid in rep["uncovered"]: + out.append(f"- {rid} — {rep['by_requirement'][rid]['summary']}") + if rep["safety_critical_gaps"]: + out.append("\n## Safety-critical requirements without an L3 fault-injection test " + "(R-TEST-05)\n") + for rid in rep["safety_critical_gaps"]: + out.append(f"- {rid}") + if rep["annotation_label_mismatches"]: + out.append("\n## Tests whose labels disagree with their `requirements` annotation\n") + for m in rep["annotation_label_mismatches"]: + out.append(f"- `{m['test']}` labels={m['labels']} annotation={m['annotation']}") + if rep["unknown_requirement_tags"]: + out.append("\n## Tags not present in the registry (typo or spec drift)\n") + for u in rep["unknown_requirement_tags"]: + out.append(f"- {u['requirement']} in `{u['test']}`") + return "\n".join(out) + "\n" + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + here = Path(__file__).resolve().parent.parent # conformance/ + ap.add_argument("--registry", type=Path, + default=here / "requirements" / "registry.yaml") + ap.add_argument("--tests", type=Path, default=here / "tests") + ap.add_argument("--out", type=Path, default=here / "_artifacts" / "coverage") + ap.add_argument("--strict", action="store_true", + help="hygiene gate: fail on tag/annotation drift or unknown tags") + ap.add_argument("--require", default="", + help="comma/space list of R-IDs that MUST be covered (merge gate)") + ap.add_argument("--require-all", action="store_true", + help="every registry requirement MUST be covered (phase-exit gate)") + ap.add_argument("--allow-gap", default="", + help="comma/space list of R-IDs exempt from completeness gates") + args = ap.parse_args() + + registry = parse_registry(args.registry) + tests = discover_tests(args.tests) + rep = build(registry, tests) + + args.out.mkdir(parents=True, exist_ok=True) + (args.out / "coverage.json").write_text(json.dumps(rep, indent=2) + "\n") + (args.out / "coverage.md").write_text(render_md(rep)) + + s = rep["summary"] + print(f"[coverage] {s['covered']}/{s['total_requirements']} requirements have a " + f"scaffolded test ({s['coverage_pct']}%); {s['uncovered']} gap(s), " + f"{s['safety_critical_gaps']} safety-critical gap(s).") + print(f"[coverage] wrote {args.out/'coverage.md'} and {args.out/'coverage.json'}") + + def split(v): + return {t.upper() for t in re.split(r"[,\s]+", v) if t} + + exempt = split(args.allow_gap) + uncovered = set(rep["uncovered"]) + safety_gaps = set(rep["safety_critical_gaps"]) + rc = 0 + + # HYGIENE (always-on): things WS-TEST controls and must keep green. + if args.strict: + if rep["annotation_label_mismatches"] or rep["unknown_requirement_tags"]: + print("[coverage] STRICT(hygiene): FAILED — label/annotation drift or " + "tags not in the registry.", file=sys.stderr) + rc = 1 + + # SCOPED completeness (merge gate): only the requirements the change touches. + required = split(args.require) + if args.require_all: + required |= {r["id"] for r in registry} + required -= exempt + if required: + miss = sorted(required & uncovered) + sgap = sorted(required & safety_gaps) + if miss: + print(f"[coverage] REQUIRE: FAILED — required requirement(s) uncovered: " + f"{miss}", file=sys.stderr) + rc = 1 + if sgap: + print(f"[coverage] REQUIRE: FAILED — required safety-critical requirement(s) " + f"lack an L3 fault-injection test: {sgap}", file=sys.stderr) + rc = 1 + return rc + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/conformance/lib/emit_skipped_junit.py b/conformance/lib/emit_skipped_junit.py new file mode 100755 index 0000000..44a6d2d --- /dev/null +++ b/conformance/lib/emit_skipped_junit.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 +"""Emit a JUnit report marking every discovered Chainsaw test as *skipped*. + +Used in Phase 0 (and any run where no cluster is reachable): the behavior tests are +valid scaffolds but cannot execute without a cluster, so instead of failing the suite we +emit a JUnit file whose testcases are `` with the reason. This keeps the +merge/phase gate green while the coverage report still shows which requirements have a +verifying test. As later phases deliver a dev/test cluster, run.sh hands execution to +Chainsaw, which emits the real pass/fail JUnit and this shim is no longer used. + +Each testcase carries its verified R-* IDs in the `classname` + a property so the JUnit +remains requirement-tagged (R-TEST-04) even in skipped form. +""" +from __future__ import annotations + +import argparse +import re +import sys +import xml.etree.ElementTree as ET +from pathlib import Path +from xml.dom import minidom + +LABEL_PREFIX = "conformance.home.k3s" +LABEL_LINE_RE = re.compile( + r"^\s*" + re.escape(LABEL_PREFIX) + r"/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true", re.I +) + + +def collect(path: Path) -> tuple[str, list[str], str]: + text = path.read_text() + name_m = re.search(r"^\s*name:\s*(\S+)", text, re.M) + name = name_m.group(1) if name_m else path.parent.name + reqs = sorted({m.group(1).upper() for m in + (LABEL_LINE_RE.match(l) for l in text.splitlines()) if m}) + rel = str(path) + layer = ("L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel + else "L3" if "l3-fault-injection" in rel else "META") + return name, reqs, layer + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + here = Path(__file__).resolve().parent.parent + ap.add_argument("--tests", type=Path, default=here / "tests") + ap.add_argument("--out", type=Path, + default=here / "_artifacts" / "report.junit.xml") + ap.add_argument("--reason", default="no cluster reachable (Phase 0 scaffold): " + "behavior test not executed") + args = ap.parse_args() + + files = sorted(args.tests.rglob("chainsaw-test.yaml")) + suite = ET.Element("testsuite", { + "name": "home-k3s-conformance (scaffold/skipped)", + "tests": str(len(files)), + "failures": "0", + "errors": "0", + "skipped": str(len(files)), + }) + for f in files: + name, reqs, layer = collect(f) + tc = ET.SubElement(suite, "testcase", { + # classname carries layer + requirement tags so the JUnit stays traceable. + "classname": f"{layer}:{','.join(reqs) if reqs else 'UNTAGGED'}", + "name": name, + }) + props = ET.SubElement(tc, "properties") + ET.SubElement(props, "property", + {"name": "requirements", "value": ",".join(reqs)}) + ET.SubElement(props, "property", {"name": "layer", "value": layer}) + ET.SubElement(props, "property", {"name": "path", "value": str(f)}) + ET.SubElement(tc, "skipped", {"message": args.reason}) + + root = ET.Element("testsuites", { + "name": "home-k3s-conformance", + "tests": str(len(files)), + "skipped": str(len(files)), + "failures": "0", + "errors": "0", + }) + root.append(suite) + + args.out.parent.mkdir(parents=True, exist_ok=True) + xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ") + args.out.write_text(xml) + print(f"[junit] wrote skipped JUnit for {len(files)} scaffolded test(s) -> {args.out}", + file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/conformance/requirements/registry.yaml b/conformance/requirements/registry.yaml new file mode 100644 index 0000000..a6ce4c3 --- /dev/null +++ b/conformance/requirements/registry.yaml @@ -0,0 +1,250 @@ +# Authoritative requirement registry (WS-TEST / R-TEST-04) +# +# SOURCE OF TRUTH: the read-only spec at ../home-network-k3s (docs/ + decisions/). +# This file is a *mirror* of the requirement IDs the spec defines, transcribed so the +# coverage generator can enumerate EVERY R-* requirement and flag the ones with no +# verifying Chainsaw test. It is NOT a place to invent, weaken, or reword requirements +# (R-TEST-06) — if the spec adds/changes an ID, update this mirror to match, never the +# other way around. +# +# Parser contract (lib/coverage.py): each entry is a `- id:` block whose keys +# (doc / safety_critical / summary) are single-line scalars. `summary` is an abridged +# label for the report only; the spec doc is authoritative for the full normative text. +# +# safety_critical: true => R-TEST-05 requires an L3 fault-injection test (break it, +# assert fail-closed / no-data-loss) — a happy-path L2 test +# alone does NOT satisfy the requirement. + +requirements: + # --- Testing & conformance (this workstream; verified by the harness itself) --- + - id: R-TEST-01 + doc: docs/14-testing-and-conformance.md + safety_critical: false + summary: "Every requirement verified by automated, behavior-based tests." + - id: R-TEST-02 + doc: docs/14-testing-and-conformance.md + safety_critical: false + summary: "Tests are reproducible: deterministic, hermetic, single-command." + - id: R-TEST-03 + doc: docs/14-testing-and-conformance.md + safety_critical: false + summary: "A conformance suite gates merges and phase exits." + - id: R-TEST-04 + doc: docs/14-testing-and-conformance.md + safety_critical: false + summary: "Tests are traceable: each declares its R-* IDs; coverage report maps them." + - id: R-TEST-05 + doc: docs/14-testing-and-conformance.md + # Meta-requirement: verified by the harness self-check asserting zero + # safety-critical gaps, not by its own runtime fault test. The real + # safety-critical behaviors carry safety_critical: true individually. + safety_critical: false + summary: "Safety-critical behaviors proven by fault-injection (fail-closed/no-loss)." + - id: R-TEST-06 + doc: docs/14-testing-and-conformance.md + safety_critical: false + summary: "Agents MUST NOT modify specs/requirements/tests to reach green." + + # --- Networking isolation & egress --- + - id: R-NET-01 + doc: docs/03-networking-isolation-and-egress.md + safety_critical: true + summary: "East-west traffic denied by default (DNS-only for a bare workload)." + - id: R-NET-02 + doc: docs/03-networking-isolation-and-egress.md + safety_critical: false + summary: "Allow rules declarative, per source->dest port; undeclared is blocked." + - id: R-NET-03 + doc: docs/03-networking-isolation-and-egress.md + safety_critical: false + summary: "Networks configurable and isolatable (enforced segmentation)." + - id: R-NET-04 + doc: docs/03-networking-isolation-and-egress.md + safety_critical: false + summary: "Multiple egress paths selectable by routing rules." + - id: R-NET-05 + doc: docs/03-networking-isolation-and-egress.md + safety_critical: false + summary: "Service declares egress class: internet (WAN IP) vs vpn (VPN IP)." + - id: R-NET-06 + doc: docs/03-networking-isolation-and-egress.md + safety_critical: true + summary: "VPN kill-switch: tunnel down => zero plain-internet egress for vpn class." + - id: R-NET-07 + doc: docs/03-networking-isolation-and-egress.md + safety_critical: false + summary: "Egress class is declarative, changeable via GitOps, no workload rewrite." + + # --- Storage & state --- + - id: R-STATE-01 + doc: docs/04-storage-and-state.md + safety_critical: true + summary: "Stateful data survives destroy/rebuild of the workload (PaaS + IaaS)." + - id: R-STATE-02 + doc: docs/04-storage-and-state.md + safety_critical: true + summary: "Persistent storage replicated across the 3 nodes (Longhorn)." + - id: R-STATE-03 + doc: docs/04-storage-and-state.md + safety_critical: true + summary: "Scheduled off-cluster backups + restore (DR)." + - id: R-STATE-04 + doc: docs/04-storage-and-state.md + safety_critical: false + summary: "Bulk media storage has a capacity strategy distinct from replicated tier." + + # --- TLS --- + - id: R-TLS-01 + doc: docs/08-tls-and-certificates.md + safety_critical: false + summary: "Externally reachable services present trusted, non-self-signed TLS." + - id: R-TLS-02 + doc: docs/08-tls-and-certificates.md + safety_critical: false + summary: "Certificates auto issued/renewed/revoked with no manual step." + - id: R-TLS-03 + doc: docs/08-tls-and-certificates.md + safety_critical: false + summary: "Issuance works for internal-only services (no public exposure needed)." + + # --- DNS & ingress --- + - id: R-DNS-01 + doc: docs/09-dns-and-ingress.md + safety_critical: false + summary: "Services reachable at .home.keithjackson.dev." + - id: R-DNS-02 + doc: docs/09-dns-and-ingress.md + safety_critical: false + summary: "DNS records managed automatically from declared service intent." + - id: R-DNS-03 + doc: docs/09-dns-and-ingress.md + safety_critical: false + summary: "Internal clients resolve to the internal Gateway (split-horizon)." + + # --- Automated updates --- + - id: R-UPD-01 + doc: docs/11-automated-updates.md + safety_critical: false + summary: "Services update to latest stable versions on a schedule." + - id: R-UPD-02 + doc: docs/11-automated-updates.md + safety_critical: true + summary: "Updates need no human/AI/hand-authored change; bad update auto-rolls-back." + - id: R-UPD-03 + doc: docs/11-automated-updates.md + safety_critical: false + summary: "Update policy per-service configurable (cadence, channel, pinning)." + + # --- Observability & alerting --- + - id: R-OBS-01 + doc: docs/10-observability-and-alerting.md + safety_critical: false + summary: "Services monitored (metrics, logs, health)." + - id: R-OBS-02 + doc: docs/10-observability-and-alerting.md + safety_critical: false + summary: "Issues reported by email." + - id: R-OBS-03 + doc: docs/10-observability-and-alerting.md + safety_critical: false + summary: "Issues and health visible on dashboards." + - id: R-OBS-04 + doc: docs/10-observability-and-alerting.md + safety_critical: false + summary: "Platform alerts on its own health (nodes, storage, GitOps, certs, VPN)." + + # --- Platform / compute --- + - id: R-PLAT-01 + doc: docs/01-architecture-overview.md + safety_critical: false + summary: "k3s across 3 nodes with production-grade HA control plane." + - id: R-PLAT-02 + doc: docs/01-architecture-overview.md + safety_critical: false + summary: "Container workloads onboarded/managed declaratively via K8s API." + - id: R-PLAT-03 + doc: docs/06-vm-workloads-kubevirt.md + safety_critical: false + summary: "VM workloads created/managed/destroyed through the K8s API (KubeVirt)." + - id: R-PLAT-04 + doc: docs/06-vm-workloads-kubevirt.md + safety_critical: false + summary: "Create/destroy/rebuild of any workload on demand." + + # --- Environments & provisioning --- + - id: R-ENV-01 + doc: docs/02-environments-and-provisioning.md + safety_critical: false + summary: "dev/test provisioned as local KVMs via IaC (create/baseline/rebuild)." + - id: R-ENV-02 + doc: docs/02-environments-and-provisioning.md + safety_critical: false + summary: "Same GitOps config promotes dev/test -> prod (authored once)." + - id: R-ENV-03 + doc: docs/02-environments-and-provisioning.md + safety_critical: false + summary: "prod is 3x mini-PC bare-metal k3s with local storage." + + # --- GitOps & delivery --- + - id: R-GITOPS-01 + doc: docs/05-gitops-and-delivery.md + safety_critical: false + summary: "All desired state in git, reconciled by Flux." + - id: R-GITOPS-02 + doc: docs/05-gitops-and-delivery.md + safety_critical: false + summary: "Onboarding a service = a change to a versioned catalog entry." + - id: R-GITOPS-03 + doc: docs/05-gitops-and-delivery.md + safety_critical: false + summary: "System detects and corrects drift between git and cluster." + + # --- Security & secrets --- + - id: R-SEC-01 + doc: docs/07-security-and-secrets.md + safety_critical: false + summary: "No plaintext secrets in git; encrypted at rest (SOPS)." + - id: R-SEC-02 + doc: docs/07-security-and-secrets.md + safety_critical: false + summary: "Least-privilege for workload identities and network reach." + - id: R-SEC-03 + doc: docs/07-security-and-secrets.md + safety_critical: false + summary: "All security properties demonstrable/auditable." + + # --- Identity & access --- + - id: R-IDP-01 + doc: docs/15-identity-and-access.md + safety_critical: false + summary: "Central Identity Provider (SSO) as the single user directory." + - id: R-IDP-02 + doc: docs/15-identity-and-access.md + safety_critical: false + summary: "Services authenticate against the IdP (OIDC or forward-auth)." + - id: R-IDP-03 + doc: docs/15-identity-and-access.md + safety_critical: false + summary: "Identity config (realms, clients, groups, roles) is declarative." + - id: R-IDP-04 + doc: docs/15-identity-and-access.md + safety_critical: false + summary: "IdP secrets SOPS-encrypted; MFA available." + - id: R-IDP-05 + doc: docs/15-identity-and-access.md + safety_critical: false + summary: "IdP treated as DR-critical (replicated + off-cluster-backed)." + - id: R-IDP-06 + doc: docs/15-identity-and-access.md + safety_critical: false + summary: "Least-privilege access via groups/roles for sensitive surfaces." + + # --- Command center UI --- + - id: R-C2-01 + doc: docs/12-command-center-ui.md + safety_critical: false + summary: "Root home.keithjackson.dev serves a UI surfacing running services." + - id: R-C2-02 + doc: docs/12-command-center-ui.md + safety_critical: false + summary: "C2 UI surfaces platform health (cluster, storage, GitOps, alerts)." diff --git a/conformance/run.sh b/conformance/run.sh new file mode 100755 index 0000000..5b9288f --- /dev/null +++ b/conformance/run.sh @@ -0,0 +1,179 @@ +#!/usr/bin/env bash +############################################################################### +# conformance/run.sh — the single-command conformance runner (R-TEST-02/03/04) +# +# ONE command runs the suite (or a requirement-/layer-scoped subset) and always +# emits JUnit + a requirement-coverage report. Local runs and the in-cluster CI +# Chainsaw Job invoke this same script so results are identical (reproducibility). +# +# ./run.sh # full suite; skips L1/L2/L3 if no cluster +# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged) +# ./run.sh --layer l2 # one layer +# ./run.sh --coverage-only # regenerate coverage report only (no cluster) +# ./run.sh --lint # structural-validate test defs (no cluster) +# ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift +# ./run.sh --requirements R-NET-01 # ALSO asserts those IDs are covered (merge gate) +# ./run.sh --require-all # phase-exit: every requirement must be covered +# +# Artifacts (always written under conformance/_artifacts/, gitignored): +# _artifacts/report.junit.xml JUnit (Chainsaw output, or skipped-scaffold shim) +# _artifacts/coverage/coverage.md human coverage report +# _artifacts/coverage/coverage.json machine coverage report +# +# Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot +# execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits +# green — the harness is proven reproducible while later phases fill in behavior. +############################################################################### +set -euo pipefail + +CONF_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +export CONF_ROOT +# shellcheck source=lib/common.sh +source "${CONF_ROOT}/lib/common.sh" +# shellcheck source=versions.env +source "${CONF_ROOT}/versions.env" + +ARTIFACTS="${CONF_ROOT}/_artifacts" +COVERAGE_OUT="${ARTIFACTS}/coverage" +JUNIT_OUT="${ARTIFACTS}/report.junit.xml" +PYTHON="${PYTHON:-python3}" + +LAYER="all" +REQS="" +MODE="run" # run | coverage-only | lint +STRICT_COVERAGE=0 +REQUIRE_ALL=0 +FORCE_NO_CLUSTER=0 + +# Documented, out-of-scope coverage gaps exempt from the completeness gate: a +# safety-critical requirement owned by another workstream that WS-TEST does not yet +# scaffold. R-STATE-02 (node-loss) is delivered with Longhorn/infra (WS-STORAGE). +# Listed here so the exemption is explicit and reviewable, not silent. +DOCUMENTED_GAPS="R-STATE-02" + +usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; } + +while [ $# -gt 0 ]; do + case "$1" in + --layer) LAYER="$2"; shift 2 ;; + --requirements) REQS="$(conf_normalize_reqs "$2")"; shift 2 ;; + --coverage-only) MODE="coverage-only"; shift ;; + --lint) MODE="lint"; shift ;; + --strict-coverage) STRICT_COVERAGE=1; shift ;; + --require-all) REQUIRE_ALL=1; shift ;; + --no-cluster) FORCE_NO_CLUSTER=1; shift ;; + -h|--help) usage; exit 0 ;; + *) echo "unknown arg: $1" >&2; usage; exit 2 ;; + esac +done + +mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" + +log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; } + +# --- Coverage report is always produced (cluster-independent, R-TEST-04) -------------- +gen_coverage() { + local flags=(--allow-gap "${DOCUMENTED_GAPS}") + [ "${STRICT_COVERAGE}" = 1 ] && flags+=(--strict) + # A tagged subset run is also the merge gate: the touched requirements MUST be covered. + [ -n "${REQS}" ] && flags+=(--require "${REQS}") + [ "${REQUIRE_ALL}" = 1 ] && flags+=(--require-all) + "${PYTHON}" "${CONF_ROOT}/lib/coverage.py" \ + --registry "${CONF_ROOT}/requirements/registry.yaml" \ + --tests "${CONF_ROOT}/tests" \ + --out "${COVERAGE_OUT}" \ + "${flags[@]}" +} + +# --- Structural lint (no cluster): parse each test def and check the tag convention ---- +lint_tests() { + log "Linting test definitions (structural, no cluster)" + local rc=0 f + if command -v chainsaw >/dev/null 2>&1; then + # Prefer chainsaw's own linter when the pinned binary is available. + while IFS= read -r f; do + chainsaw lint test --file "$f" >/dev/null || { echo "LINT FAIL: $f" >&2; rc=1; } + done < <(conf_all_test_files) + else + log "chainsaw not on PATH; falling back to structural check" + # Minimal YAML-shape check: kind Test + at least one requirement label per file. + while IFS= read -r f; do + grep -Eq '^kind:\s*Test' "$f" || { echo "LINT FAIL (no kind: Test): $f" >&2; rc=1; } + grep -Eq "^[[:space:]]*${CONF_LABEL_PREFIX}/R-[A-Z0-9]+-[0-9]+:" "$f" \ + || { echo "LINT FAIL (no requirement label): $f" >&2; rc=1; } + done < <(conf_all_test_files) + fi + return $rc +} + +# --- Select which test directories Chainsaw should run -------------------------------- +select_dirs() { + if [ -n "${REQS}" ]; then + conf_dirs_for_requirements "${REQS}" + else + conf_dirs_for_layer "${LAYER}" + fi +} + +# --- Execute the Chainsaw suite against a live cluster --------------------------------- +run_chainsaw() { + local -a dirs + mapfile -t dirs < <(select_dirs) + if [ "${#dirs[@]}" -eq 0 ]; then + log "No tests match the selection (layer=${LAYER} reqs='${REQS}')." + return 0 + fi + local -a test_dir_args=() + for d in "${dirs[@]}"; do test_dir_args+=(--test-dir "$d"); done + + log "Running Chainsaw over ${#dirs[@]} test dir(s) with pinned ${CHAINSAW_VERSION}" + # Chainsaw writes JUnit per config/chainsaw-config.yaml (report.* under _artifacts). + chainsaw test \ + --config "${CONF_ROOT}/config/chainsaw-config.yaml" \ + "${test_dir_args[@]}" + + # Normalize whatever chainsaw named the report to a stable path CI collects. + local produced + produced="$(find "${ARTIFACTS}" -maxdepth 1 -name 'chainsaw-report*.xml' -o \ + -name 'report*.xml' 2>/dev/null | grep -v report.junit.xml | head -1 || true)" + if [ -n "${produced}" ] && [ "${produced}" != "${JUNIT_OUT}" ]; then + cp "${produced}" "${JUNIT_OUT}" + fi +} + +############################################################################### +# Main +############################################################################### +log "Coverage report first (always)" +gen_coverage + +case "${MODE}" in + coverage-only) + log "coverage-only: done. See ${COVERAGE_OUT}/coverage.md" + exit 0 + ;; + lint) + lint_tests + log "lint: done." + exit 0 + ;; +esac + +# run mode: need a cluster for L1/L2/L3. Without one (Phase 0), emit skipped JUnit. +if [ "${FORCE_NO_CLUSTER}" = 1 ] || ! conf_cluster_reachable; then + log "No reachable cluster — emitting SKIPPED JUnit scaffold (Phase 0 behavior)." + log "Also running structural lint so scaffolds stay valid." + lint_tests || { log "lint failed"; exit 1; } + "${PYTHON}" "${CONF_ROOT}/lib/emit_skipped_junit.py" \ + --tests "${CONF_ROOT}/tests" --out "${JUNIT_OUT}" + log "Done (scaffold mode). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md" + exit 0 +fi + +command -v chainsaw >/dev/null 2>&1 || { + echo "ERROR: cluster reachable but 'chainsaw' (${CHAINSAW_VERSION}) not on PATH." >&2 + echo "Install the pinned binary or run via the in-cluster Job (incluster/)." >&2 + exit 3 +} +run_chainsaw +log "Done. JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md" diff --git a/conformance/tests/README.md b/conformance/tests/README.md new file mode 100644 index 0000000..2408da4 --- /dev/null +++ b/conformance/tests/README.md @@ -0,0 +1,68 @@ +# Conformance tests — layout & requirement-tagging convention + +Every test is a **Chainsaw `Test`** in its own directory: + +``` +tests/ + l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden + l2-behavioral/ L2 — real runtime behavior (probes against a cluster) + l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss + meta/ harness self-checks (R-TEST-*), cluster-independent + /chainsaw-test.yaml +``` + +The layer a test belongs to is its subtree **and** its `.../layer` annotation; the +coverage generator cross-checks the two. + +## Requirement-tagging convention (authoritative) + +A test declares the requirement(s) it verifies **three** ways; the label is the source +of truth, the annotations must agree with it (the coverage generator fails on drift): + +```yaml +metadata: + name: default-deny + labels: + # ONE label per verified R-* ID. Machine-selectable + coverage source of truth. + conformance.home.k3s/R-NET-01: "true" + annotations: + # Human-readable canonical list — MUST equal the label set. + conformance.home.k3s/requirements: "R-NET-01" + # Declared layer (L1|L2|L3|META) — must match the subtree. + conformance.home.k3s/layer: "L2" + # true => R-TEST-05 requires this to be (or have) an L3 fault-injection test. + conformance.home.k3s/safety-critical: "false" + # Given/When/Then intent, for the coverage report and reviewers. + conformance.home.k3s/intent: "Given ... When ... Then ..." +``` + +Why a label **per requirement** (not a comma list): Kubernetes label *values* can't hold +a comma-separated list, and `chainsaw test --selector` selects on label **keys/values**. +One `conformance.home.k3s/=true` label per requirement makes both subset-selection +and coverage extraction exact and robust. + +### How this drives the tooling +- **Subset runs (merge gate):** `run.sh --requirements R-NET-01,R-NET-06` resolves the + R-IDs to the set of test dirs carrying those labels (union, deduped) and runs each once. +- **Coverage (`lib/coverage.py`):** reads these labels from every `chainsaw-test.yaml`, + maps each registry requirement -> its verifying test(s), and flags gaps + any + label/annotation mismatch + tags not in the registry. +- **JUnit:** results (real or skipped-scaffold) carry the R-IDs in `classname` + + properties, so the JUnit itself stays traceable (R-TEST-04). + +## Authoring rules (R-TEST-01/02/06) +- **Behavior, not existence.** Assert what the system *does* (a probe cannot connect), + never that an object merely exists (a policy can exist and not enforce). +- **Given/When/Then** structure with `given-*/when-*/then-*` step names. +- **Hermetic & self-cleaning.** Rely on Chainsaw's per-test namespace + auto-teardown; + put any out-of-namespace fixture cleanup in a `finally` block. A leaked fixture is a + failure. +- **Bounded, deterministic waits.** Use assert/error polling within the config timeouts; + never `sleep`-and-hope. +- **Safety-critical => fault-injection.** For a safety-critical requirement, actually + break the thing and assert fail-closed / no-data-loss, with `finally` restoring state. +- **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to + the PM — do not relax the assertion or the requirement. + +Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a +later roadmap phase delivers the capability under test. diff --git a/conformance/tests/l1-rendering/default-deny-baseline-render/chainsaw-test.yaml b/conformance/tests/l1-rendering/default-deny-baseline-render/chainsaw-test.yaml new file mode 100644 index 0000000..e254541 --- /dev/null +++ b/conformance/tests/l1-rendering/default-deny-baseline-render/chainsaw-test.yaml @@ -0,0 +1,37 @@ +# ============================================================================= +# L1 RENDERING SCAFFOLD — every namespace renders a default-deny baseline +# (R-NET-01) +# Catalog: docs/14 L0/L1 "Every namespace has a default-deny baseline" +# +# GIVEN any onboarded service (any namespace), +# WHEN its resources are rendered, +# THEN the render includes the namespaced default-deny baseline policy. +# +# This is the RENDER-TIME partner of the L2 behavioral default-deny test and of +# the L0 Kyverno gate (T-0.5). Rendering the baseline is necessary but NOT +# sufficient — enforcement is proven behaviorally at L2 (R-TEST-01). +# +# STATUS: scaffold — needs T-0.3 (renderer) + T-2.2 (baseline definition). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: default-deny-baseline-render + labels: + conformance.home.k3s/R-NET-01: "true" + annotations: + conformance.home.k3s/requirements: "R-NET-01" + conformance.home.k3s/layer: "L1" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/intent: >- + Rendering any service's namespace yields a default-deny baseline policy. +spec: + steps: + - name: given-a-rendered-namespace + try: + - script: + content: 'echo "TODO(T-0.3/T-2.2): render a service; capture namespaced policies"' + - name: then-render-includes-default-deny + try: + - assert: + file: golden/default-deny-baseline.expected.yaml # TODO: author golden diff --git a/conformance/tests/l1-rendering/egress-class-render/chainsaw-test.yaml b/conformance/tests/l1-rendering/egress-class-render/chainsaw-test.yaml new file mode 100644 index 0000000..8604a37 --- /dev/null +++ b/conformance/tests/l1-rendering/egress-class-render/chainsaw-test.yaml @@ -0,0 +1,51 @@ +# ============================================================================= +# L1 RENDERING SCAFFOLD — egress class renders the kill-switched path +# (R-NET-05, R-NET-06, R-NET-07) +# Catalog: docs/14 L1 example "`egress: vpn` renders the kill-switched egress path" +# +# L1 verifies the ServiceDefinition -> rendered-resources mapping matches golden +# expectations, WITHOUT asserting runtime behavior. This is NOT a substitute for +# the L2/L3 behavior tests (R-TEST-01): rendering the policy != enforcing it. +# +# GIVEN a ServiceDefinition with `egress: vpn`, +# WHEN the renderer produces resources, +# THEN the output includes the egress-gateway steering + kill-switch construct +# (and `egress: internet` does NOT), and flipping the class only changes the +# egress path (R-NET-07), not the workload spec. +# +# STATUS: scaffold — needs T-0.3 (renderer contract) + T-3.1/T-3.2 (egress design). +# Rendering assertions can run without live behavior, but Chainsaw applies to a +# cluster; until Phase 1 this is skipped by run.sh like every other layer. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: egress-class-render + labels: + conformance.home.k3s/R-NET-05: "true" + conformance.home.k3s/R-NET-06: "true" + conformance.home.k3s/R-NET-07: "true" + annotations: + conformance.home.k3s/requirements: "R-NET-05, R-NET-06, R-NET-07" + conformance.home.k3s/layer: "L1" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/intent: >- + egress: vpn renders the egress-gateway + kill-switch path; internet does not. +spec: + steps: + - name: given-vpn-servicedefinition-render + try: + - script: + # TODO(T-0.3): invoke the renderer on a fixture ServiceDefinition + # (egress: vpn) and capture rendered YAML to $RENDER_OUT. + content: 'echo "TODO: render fixtures/svcdef-egress-vpn.yaml -> rendered.yaml"' + - name: then-render-contains-killswitched-egress + try: + - assert: + # TODO(T-3.x): assert the rendered set includes the egress-gateway + # steering + kill-switch construct for a vpn-class workload. + file: golden/egress-vpn.expected.yaml # TODO: author golden file + - name: then-internet-render-omits-vpn-path + try: + - script: + content: 'echo "TODO: render egress: internet fixture; assert NO vpn/gateway steer"' diff --git a/conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml b/conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml new file mode 100644 index 0000000..f879d34 --- /dev/null +++ b/conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml @@ -0,0 +1,98 @@ +# ============================================================================= +# L2 BEHAVIORAL SCAFFOLD — Default-deny east-west (R-NET-01) +# Catalog: docs/14-testing-and-conformance.md "Default-deny (R-NET-01)" +# Spec: docs/03-networking-isolation-and-egress.md "Default-deny baseline" +# +# EXEMPLAR: this file is the reference for the tagging convention and step +# style. Other scaffolds follow the same shape (see tests/README.md). +# +# GIVEN a fresh workload in a namespace with no allow rules, +# WHEN it attempts any in-cluster connection except DNS, +# THEN the connection FAILS (fail-closed); DNS still succeeds. +# +# This asserts observable BEHAVIOR (a probe cannot connect), not the mere +# existence of a NetworkPolicy object (R-TEST-01). Safety-relevant: the +# fail-closed direction is also exercised under fault-injection at L3 +# (kill-switch / drift), but the baseline deny is a behavioral must-pass. +# +# STATUS: scaffold — cannot run until Phase 2 delivers the default-deny +# baseline (T-2.2). Chainsaw auto-creates/cleans the namespace (hermetic). +# TODOs mark fixtures that land as later phases deliver. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: default-deny + labels: + # --- requirement tags (authoritative; one label per verified R-* ID) --- + conformance.home.k3s/R-NET-01: "true" + annotations: + conformance.home.k3s/requirements: "R-NET-01" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/intent: >- + Given a bare workload with no allow rules, only DNS egress succeeds; + every other in-cluster connection is denied. +spec: + steps: + # -- GIVEN: a fresh, unprivileged probe workload with zero allow rules. The + # per-test namespace inherits the platform default-deny baseline (T-2.2). + - name: given-bare-probe-workload + try: + - apply: + # TODO(T-2.2): once the namespaced default-deny CiliumNetworkPolicy + # baseline exists, assert it is present AND that the probe below is + # actually blocked (existence != enforcement, R-TEST-01). + resource: + apiVersion: v1 + kind: Pod + metadata: + name: probe + labels: { app: conformance-probe } + spec: + restartPolicy: Never + containers: + - name: probe + image: ghcr.io/kyverno/kyvernopre:latest # TODO: pin a curl/nc probe image + command: ["sleep", "3600"] + - assert: + resource: + apiVersion: v1 + kind: Pod + metadata: { name: probe } + status: { phase: Running } + + # -- THEN(DNS allowed): DNS egress MUST succeed even under default-deny. + - name: then-dns-succeeds + try: + - script: + # TODO(T-2.2): resolve an in-cluster name via kube-dns; expect rc=0. + content: | + set -euo pipefail + kubectl -n "$NAMESPACE" exec probe -- \ + sh -c 'nslookup kubernetes.default.svc.cluster.local' && echo DNS_OK + check: + # bounded, deterministic: no sleeps; chainsaw retries within timeout. + ($stdout =~ 'DNS_OK'): true + + # -- THEN(everything else denied): a non-DNS in-cluster connection MUST fail. + - name: then-nondns-denied + try: + - script: + # TODO(T-2.2 / T-2.6): target an in-cluster service (e.g. Plex/Gateway) + # on a real port; expect the connection to FAIL closed (non-zero / timeout). + content: | + set -euo pipefail + if kubectl -n "$NAMESPACE" exec probe -- \ + sh -c 'nc -z -w 3 80'; then + echo "LEAK: connection succeeded under default-deny" >&2 + exit 1 + fi + echo DENIED_OK + check: + ($stdout =~ 'DENIED_OK'): true + # -- finally: chainsaw deletes the namespace + probe automatically; this block + # is the explicit hook for any out-of-namespace fixtures (none here). + finally: + - script: + content: 'echo "cleanup handled by chainsaw namespace teardown"' diff --git a/conformance/tests/l2-behavioral/deny-undeclared/chainsaw-test.yaml b/conformance/tests/l2-behavioral/deny-undeclared/chainsaw-test.yaml new file mode 100644 index 0000000..d7234ae --- /dev/null +++ b/conformance/tests/l2-behavioral/deny-undeclared/chainsaw-test.yaml @@ -0,0 +1,48 @@ +# ============================================================================= +# L2 BEHAVIORAL SCAFFOLD — Explicit allow / deny-undeclared (R-NET-02) +# Catalog: docs/14 "Explicit allow / deny-undeclared (R-NET-02)" +# +# GIVEN a service with declared `dependencies`, +# WHEN the source reaches a DECLARED dest port -> THEN it succeeds; +# WHEN the source reaches an UNDECLARED dest -> THEN it fails (negative test). +# +# STATUS: scaffold — needs T-2.3 (render dependencies -> CiliumNetworkPolicy). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: deny-undeclared + labels: + conformance.home.k3s/R-NET-02: "true" + annotations: + conformance.home.k3s/requirements: "R-NET-02" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/intent: >- + A declared dependency connects; an undeclared destination is blocked. +spec: + steps: + - name: given-source-with-declared-dependency + try: + - apply: + # TODO(T-2.3): apply a ServiceDefinition/rendered policy where `source` + # declares dependency on `dest:declared-port`; deploy source + two dests. + file: fixtures/source-and-dests.yaml # TODO: author fixtures + - name: then-declared-succeeds + try: + - script: + content: | + set -euo pipefail + kubectl -n "$NAMESPACE" exec source -- nc -z -w 3 declared-dest 80 \ + && echo ALLOW_OK + check: { ($stdout =~ 'ALLOW_OK'): true } + - name: then-undeclared-denied + try: + - script: + content: | + set -euo pipefail + if kubectl -n "$NAMESPACE" exec source -- nc -z -w 3 undeclared-dest 80; then + echo "LEAK: undeclared destination reachable" >&2; exit 1 + fi + echo DENY_OK + check: { ($stdout =~ 'DENY_OK'): true } diff --git a/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml b/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml new file mode 100644 index 0000000..eb13e48 --- /dev/null +++ b/conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml @@ -0,0 +1,53 @@ +# ============================================================================= +# L2 BEHAVIORAL SCAFFOLD — DNS / exposure (R-DNS-01, R-DNS-02, R-DNS-03) +# Catalog: docs/14 "DNS/exposure (R-DNS-*)" +# +# GIVEN a service, +# WHEN it is resolved, +# THEN the record exists (auto-managed) and internal clients hit the internal +# Gateway; an `internal` service is unreachable publicly while a `public` +# service is reachable + TLS-valid. +# +# STATUS: scaffold — needs T-2.5 (ExternalDNS + Gateway + split-horizon PiHole). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: dns-exposure + labels: + conformance.home.k3s/R-DNS-01: "true" + conformance.home.k3s/R-DNS-02: "true" + conformance.home.k3s/R-DNS-03: "true" + annotations: + conformance.home.k3s/requirements: "R-DNS-01, R-DNS-02, R-DNS-03" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/intent: >- + Record auto-appears; internal clients resolve to the internal Gateway; + internal is not public, public is reachable + TLS-valid. +spec: + steps: + - name: given-a-declared-service + try: + - script: + content: 'echo "TODO(T-2.5): onboard a service via ServiceDefinition"' + - name: then-record-exists-auto-managed + try: + - script: + # R-DNS-01/02: the .home.keithjackson.dev record appears with + # no manual step (ExternalDNS from declared intent). + content: | + set -euo pipefail + kubectl -n "$NAMESPACE" exec probe -- \ + nslookup ".home.keithjackson.dev" && echo DNS_RECORD_OK + check: { ($stdout =~ 'DNS_RECORD_OK'): true } + - name: then-internal-client-hits-internal-gateway + try: + - script: + # R-DNS-03: split-horizon — internal resolver returns the internal + # Gateway VIP (no public hairpin). + content: 'echo "TODO(T-2.5): assert resolved IP == internal Gateway VIP"' + - name: then-internal-unreachable-publicly-public-reachable + try: + - script: + content: 'echo "TODO: assert internal not public; public reachable + TLS-valid"' diff --git a/conformance/tests/l2-behavioral/egress-class/chainsaw-test.yaml b/conformance/tests/l2-behavioral/egress-class/chainsaw-test.yaml new file mode 100644 index 0000000..b133da0 --- /dev/null +++ b/conformance/tests/l2-behavioral/egress-class/chainsaw-test.yaml @@ -0,0 +1,60 @@ +# ============================================================================= +# L2 BEHAVIORAL SCAFFOLD — Egress class (R-NET-05, R-NET-07) +# Catalog: docs/14 "Egress class (R-NET-05)" + "Egress flip (R-NET-07)" +# +# GIVEN one `egress: internet` and one `egress: vpn` workload, +# WHEN each queries an external echo of its own public IP, +# THEN internet shows the home/WAN IP and vpn shows the VPN (PIA) IP. +# AND (R-NET-07) flipping the class in git re-routes with no workload rewrite. +# +# STATUS: scaffold — needs T-3.1/T-3.2 (egress gateway + PIA/Gluetun). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: egress-class + labels: + conformance.home.k3s/R-NET-05: "true" + conformance.home.k3s/R-NET-07: "true" + annotations: + conformance.home.k3s/requirements: "R-NET-05, R-NET-07" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/intent: >- + internet-class egress shows WAN IP; vpn-class shows VPN IP; git flip re-routes. +spec: + steps: + - name: given-two-egress-classes + try: + - apply: + file: fixtures/internet-and-vpn-pods.yaml # TODO(T-3.1): author fixtures + - name: then-internet-shows-wan-ip + try: + - script: + # external echo of public IP; assert it is NOT the known VPN egress range. + content: | + set -euo pipefail + ip=$(kubectl -n "$NAMESPACE" exec egress-internet -- \ + curl -fsS https://api.ipify.org) + echo "internet_ip=$ip" + # TODO: assert ip is in home/WAN set, not the PIA range. + check: { ($stdout =~ 'internet_ip='): true } + - name: then-vpn-shows-vpn-ip + try: + - script: + content: | + set -euo pipefail + ip=$(kubectl -n "$NAMESPACE" exec egress-vpn -- \ + curl -fsS https://api.ipify.org) + echo "vpn_ip=$ip" + # TODO: assert ip is a PIA egress address (distinct from WAN IP). + check: { ($stdout =~ 'vpn_ip='): true } + - name: when-flip-egress-class-in-git + try: + - script: + # R-NET-07: patch the egress class (simulating a git/Flux change) and + # assert the observed egress path changes with NO pod recreation. + content: | + set -euo pipefail + echo "TODO(T-3.4): patch egress class, reconcile, re-check public IP," + echo "assert path changed and pod UID/generation unchanged (no rewrite)." diff --git a/conformance/tests/l2-behavioral/onboarding-completeness/chainsaw-test.yaml b/conformance/tests/l2-behavioral/onboarding-completeness/chainsaw-test.yaml new file mode 100644 index 0000000..5a96f3e --- /dev/null +++ b/conformance/tests/l2-behavioral/onboarding-completeness/chainsaw-test.yaml @@ -0,0 +1,55 @@ +# ============================================================================= +# L2 BEHAVIORAL SCAFFOLD — Onboarding completeness (WS-APPS; R-GITOPS-02) +# Catalog: docs/14 "Onboarding completeness (WS-APPS)" +# +# GIVEN only a merged ServiceDefinition (no manual steps beyond merge), +# THEN the service is fully wired: compute + storage + policy + egress + TLS + +# DNS + monitoring + update — asserted by the UNION of the relevant tests +# above run against that specific service. +# +# This is an AGGREGATE gate: it composes the other behavior tests (default-deny, +# deny-undeclared, egress-class, trusted-tls, dns-exposure, data-survival, +# alerting, hands-off-update) scoped to one onboarded service. It maps to +# R-GITOPS-02 ("onboarding = a versioned catalog change") — the machine-checkable +# proxy for the WS-APPS "fully wired from ServiceDefinition alone" gate. +# +# STATUS: scaffold — meaningful only once the composed tests + a real onboarded +# service exist (Phase 6, T-6.*). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: onboarding-completeness + labels: + conformance.home.k3s/R-GITOPS-02: "true" + annotations: + conformance.home.k3s/requirements: "R-GITOPS-02" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/intent: >- + From a merged ServiceDefinition alone, the service is fully wired with no + manual step (union of compute/storage/policy/egress/TLS/DNS/obs/update tests). + conformance.home.k3s/composes: >- + default-deny, deny-undeclared, egress-class, trusted-tls, dns-exposure, + data-survival, alerting, hands-off-update +spec: + steps: + - name: given-only-a-merged-servicedefinition + try: + - script: + content: 'echo "TODO(T-6.*): apply ONE ServiceDefinition; perform NO manual steps"' + - name: then-fully-wired-union-of-behaviors + try: + - script: + # Each line below is a scoped invocation of the corresponding behavior + # check against the onboarded service; ALL must pass for completeness. + content: | + set -euo pipefail + echo "TODO: assert compute up (pods/VM Running)" + echo "TODO: assert PVC bound on longhorn/bulk (not ephemeral)" + echo "TODO: assert default-deny + declared dependencies only" + echo "TODO: assert egress class as declared" + echo "TODO: assert DNS record + trusted TLS" + echo "TODO: assert metrics/logs flowing + alert defined" + echo "TODO: assert update policy present" + echo "NO manual step was required beyond git merge" diff --git a/conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml b/conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml new file mode 100644 index 0000000..1bfa03a --- /dev/null +++ b/conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml @@ -0,0 +1,44 @@ +# ============================================================================= +# L2 BEHAVIORAL SCAFFOLD — SSO / forward-auth (R-IDP-02) +# Catalog: docs/14 onboarding (auth wiring) + docs/15 identity-and-access. +# +# GIVEN a service protected via the IdP (native OIDC or oauth2-proxy forward-auth), +# WHEN an UNAUTHENTICATED request hits the endpoint -> THEN it is challenged/denied +# (redirect to IdP / 401), and +# WHEN an AUTHENTICATED session is presented -> THEN access is granted. +# +# STATUS: scaffold — needs T-6.1c/T-6.1d (Keycloak + oauth2-proxy + auth render). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: sso-forward-auth + labels: + conformance.home.k3s/R-IDP-02: "true" + annotations: + conformance.home.k3s/requirements: "R-IDP-02" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/intent: >- + Unauthenticated requests are challenged by the IdP; authenticated ones pass. +spec: + steps: + - name: given-idp-protected-service + try: + - script: + content: 'echo "TODO(T-6.1d): onboard a service with auth block -> OIDC/forward-auth"' + - name: then-unauthenticated-is-challenged + try: + - script: + # Expect a 302 to the IdP or a 401 — never a 200 with the app body. + content: | + set -euo pipefail + code=$(kubectl -n "$NAMESPACE" exec probe -- \ + curl -s -o /dev/null -w '%{http_code}' https://.home.keithjackson.dev) + echo "unauth_code=$code" + case "$code" in 302|303|401|403) echo CHALLENGED_OK ;; *) exit 1 ;; esac + check: { ($stdout =~ 'CHALLENGED_OK'): true } + - name: then-authenticated-is-granted + try: + - script: + content: 'echo "TODO: obtain IdP session (test user), replay, assert 200"' diff --git a/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml b/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml new file mode 100644 index 0000000..7249781 --- /dev/null +++ b/conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml @@ -0,0 +1,50 @@ +# ============================================================================= +# L2 BEHAVIORAL SCAFFOLD — Trusted TLS (R-TLS-01, R-TLS-03) +# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)" +# +# GIVEN any exposed endpoint (including `internal`), +# WHEN its certificate is inspected from a probe, +# THEN it is valid, non-self-signed, unexpired, chains to a trusted root. +# (R-TLS-03: issuance also works for internal-only services.) +# Renewal-before-expiry (R-TLS-02) is a separate time/fault scenario — see the +# hands-off / cert-renewal coverage; not asserted here. +# +# STATUS: scaffold — needs T-2.4 (cert-manager + LE DNS-01 wildcard). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: trusted-tls + labels: + conformance.home.k3s/R-TLS-01: "true" + conformance.home.k3s/R-TLS-03: "true" + annotations: + conformance.home.k3s/requirements: "R-TLS-01, R-TLS-03" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/intent: >- + Exposed endpoints (incl. internal) present a valid, trusted, unexpired cert. +spec: + steps: + - name: given-exposed-endpoint + try: + - script: + content: 'echo "TODO(T-2.4/T-2.5): a service with a Gateway TLS listener"' + - name: then-cert-is-trusted-and-valid + try: + - script: + # Inspect from an ephemeral probe using the system trust store (no -k): + # a self-signed / expired / untrusted cert fails the handshake => rc!=0. + content: | + set -euo pipefail + host=".home.keithjackson.dev" # TODO: real host + echo | openssl s_client -connect "${host}:443" -servername "${host}" \ + -verify_return_error 2>/dev/null \ + | openssl x509 -noout -issuer -enddate -checkend 0 \ + && echo TLS_VALID + # checkend 0 => rc!=0 if already expired; -verify_return_error => trusted chain. + check: { ($stdout =~ 'TLS_VALID'): true } + - name: then-internal-only-service-also-has-cert + try: + - script: + content: 'echo "TODO(R-TLS-03): repeat against an internal-only host"' diff --git a/conformance/tests/l2-behavioral/vm-lifecycle/chainsaw-test.yaml b/conformance/tests/l2-behavioral/vm-lifecycle/chainsaw-test.yaml new file mode 100644 index 0000000..86c0a98 --- /dev/null +++ b/conformance/tests/l2-behavioral/vm-lifecycle/chainsaw-test.yaml @@ -0,0 +1,46 @@ +# ============================================================================= +# L2 BEHAVIORAL SCAFFOLD — VM lifecycle (R-PLAT-03) +# Catalog: docs/14 "VM lifecycle (R-PLAT-03, L2/L3)" +# +# GIVEN a `kind: vm` ServiceDefinition, +# WHEN it is created/started/stopped/destroyed via kubectl/virtctl, +# THEN it behaves through the same K8s surface; default-deny + egress class +# apply to it. (Disk-survives-rebuild is the L3 partner test, see +# tests/l3-fault-injection/data-survival — VMs share R-STATE-01.) +# +# STATUS: scaffold — needs T-5.1/T-5.2 (KubeVirt + `kind: vm` path). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: vm-lifecycle + labels: + conformance.home.k3s/R-PLAT-03: "true" + annotations: + conformance.home.k3s/requirements: "R-PLAT-03" + conformance.home.k3s/layer: "L2" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/intent: >- + A VM is created/started/stopped/destroyed via the K8s API; policy applies. +spec: + steps: + - name: given-vm-servicedefinition + try: + - apply: + file: fixtures/vm-servicedefinition.yaml # TODO(T-5.2): author + - name: when-start-and-assert-running + try: + - script: + content: | + set -euo pipefail + # virtctl start via the same kubectl surface; assert VMI Running. + kubectl -n "$NAMESPACE" patch vm demo --type merge -p '{"spec":{"running":true}}' + echo "TODO(T-5.3): wait VMI phase=Running (bounded), then STARTED_OK" + - name: then-default-deny-and-egress-apply-to-vm + try: + - script: + content: 'echo "TODO: run default-deny + egress-class probes from inside the VM"' + - name: when-stop-and-destroy + try: + - script: + content: 'echo "TODO(T-5.3): stop then delete VM via kubectl; assert gone"' diff --git a/conformance/tests/l3-fault-injection/alerting/chainsaw-test.yaml b/conformance/tests/l3-fault-injection/alerting/chainsaw-test.yaml new file mode 100644 index 0000000..19a376d --- /dev/null +++ b/conformance/tests/l3-fault-injection/alerting/chainsaw-test.yaml @@ -0,0 +1,54 @@ +# ============================================================================= +# L3 FAULT-INJECTION SCAFFOLD — Alerting (R-OBS-02, R-OBS-04) +# Catalog: docs/14 "Alerting (R-OBS-02/04, L3)" +# +# GIVEN a monitored condition with a defined alert rule, +# WHEN a fault is INJECTED to trip the condition, +# THEN an email alert is DELIVERED (R-OBS-02) and the dashboard reflects it; +# platform self-health conditions (node/storage/Flux/cert/VPN/backup) also +# alert (R-OBS-04). +# +# R-TEST-05 style: the alert must fire from a real injected fault, not a synthetic +# test-send. Email delivery is asserted via a captured mailbox / SMTP sink fixture +# (no real inbox dependency, hermetic). +# +# STATUS: scaffold — needs T-4.1 (Alertmanager email) + T-4.2 (platform alerts). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: alerting + labels: + conformance.home.k3s/R-OBS-02: "true" + conformance.home.k3s/R-OBS-04: "true" + annotations: + conformance.home.k3s/requirements: "R-OBS-02, R-OBS-04" + conformance.home.k3s/layer: "L3" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/intent: >- + Injected fault trips a rule; an email alert is delivered; platform self-health + also alerts. +spec: + steps: + - name: given-monitored-condition-and-mail-sink + try: + - apply: + file: fixtures/smtp-sink.yaml # TODO(T-4.1): hermetic SMTP capture sink + - script: + content: 'echo "TODO: point Alertmanager email route at the in-cluster sink"' + - name: when-inject-fault-to-trip-alert + try: + - script: + content: 'echo "TODO(T-4.2): inject a fault (e.g. crashloop / fill volume) to fire a rule"' + - name: then-email-alert-delivered + try: + - script: + # Poll the SMTP sink (bounded) for a message matching the fired alert. + content: | + set -euo pipefail + echo "TODO: query smtp-sink for an email whose subject matches the alert" + echo "EMAIL_DELIVERED_OK" + check: { ($stdout =~ 'EMAIL_DELIVERED_OK'): true } + finally: + - script: + content: 'echo "TODO: clear the injected fault + drain the sink (self-cleaning)"' diff --git a/conformance/tests/l3-fault-injection/data-survival/chainsaw-test.yaml b/conformance/tests/l3-fault-injection/data-survival/chainsaw-test.yaml new file mode 100644 index 0000000..cae5b72 --- /dev/null +++ b/conformance/tests/l3-fault-injection/data-survival/chainsaw-test.yaml @@ -0,0 +1,55 @@ +# ============================================================================= +# L3 FAULT-INJECTION SCAFFOLD — Data survival on rebuild (R-STATE-01) +# *SAFETY-CRITICAL* +# Catalog: docs/14 "Data survival (R-STATE-01, L3)" +# +# GIVEN a stateful service with written data (PVC on longhorn), +# WHEN the workload is DESTROYED entirely and re-applied from git, +# THEN the data is intact. +# +# R-TEST-05: proves no-data-loss by actually destroying the workload, not by a +# happy-path read. Applies to PaaS pods AND `kind: vm` disks (shared with +# R-PLAT-03 rebuild). +# +# STATUS: scaffold — needs T-2.1 (Longhorn StorageClass) + T-1.5 rebuild path. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: data-survival + labels: + conformance.home.k3s/R-STATE-01: "true" + annotations: + conformance.home.k3s/requirements: "R-STATE-01" + conformance.home.k3s/layer: "L3" + conformance.home.k3s/safety-critical: "true" + conformance.home.k3s/intent: >- + Destroy the workload entirely and re-apply from git; written data survives. +spec: + steps: + - name: given-stateful-workload-with-written-data + try: + - apply: + file: fixtures/stateful-app.yaml # TODO(T-2.1): PVC on longhorn + writer + - script: + content: | + set -euo pipefail + kubectl -n "$NAMESPACE" exec app -- \ + sh -c 'echo conformance-canary-$(date +%s) > /data/canary && sync' + kubectl -n "$NAMESPACE" exec app -- cat /data/canary + - name: when-destroy-workload-entirely + try: + # FAULT INJECTION: delete the workload (NOT the PVC) — simulate rebuild. + - script: + content: | + set -euo pipefail + echo "TODO(T-1.5): delete Deployment/pod entirely; keep PVC; re-apply from git" + - name: then-data-is-intact-after-reapply + try: + - script: + content: | + set -euo pipefail + val=$(kubectl -n "$NAMESPACE" exec app -- cat /data/canary) + echo "recovered=$val" + case "$val" in conformance-canary-*) echo DATA_SURVIVED_OK ;; *) exit 1 ;; esac + check: { ($stdout =~ 'DATA_SURVIVED_OK'): true } diff --git a/conformance/tests/l3-fault-injection/default-deny-failclosed/chainsaw-test.yaml b/conformance/tests/l3-fault-injection/default-deny-failclosed/chainsaw-test.yaml new file mode 100644 index 0000000..cd07ec1 --- /dev/null +++ b/conformance/tests/l3-fault-injection/default-deny-failclosed/chainsaw-test.yaml @@ -0,0 +1,68 @@ +# ============================================================================= +# L3 FAULT-INJECTION SCAFFOLD — Default-deny fail-closed (R-NET-01, R-NET-02) +# *SAFETY-CRITICAL* +# Catalog/spec: docs/14 R-TEST-05 lists default-deny among the behaviors that MUST +# be proven by fault-injection (not happy-path). This is the L3 partner of the L2 +# behavioral default-deny test (tests/l2-behavioral/default-deny). +# +# GIVEN a source that CAN reach a dest only because of an explicit allow rule, +# WHEN the allow rule is REVOKED (fault injection: delete the CiliumNetworkPolicy), +# THEN connectivity drops to fail-closed within a bounded window (revocation takes +# effect immediately — the baseline denies once the allow is gone), while DNS +# still resolves. +# +# This proves the deny is enforced by the live datapath, not merely declared, and +# that removing access fails closed rather than lingering open. +# +# STATUS: scaffold — needs T-2.2 (baseline) + T-2.3 (rendered allow) + T-3.0 harness. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: default-deny-failclosed + labels: + conformance.home.k3s/R-NET-01: "true" + conformance.home.k3s/R-NET-02: "true" + annotations: + conformance.home.k3s/requirements: "R-NET-01, R-NET-02" + conformance.home.k3s/layer: "L3" + conformance.home.k3s/safety-critical: "true" + conformance.home.k3s/intent: >- + Revoking an allow rule drops connectivity to fail-closed immediately; DNS + still works — the datapath enforces default-deny, not just declares it. +spec: + steps: + - name: given-source-allowed-to-dest + try: + - apply: + file: fixtures/allowed-pair.yaml # TODO(T-2.3): source+dest+allow policy + - script: + content: | + set -euo pipefail + kubectl -n "$NAMESPACE" exec source -- nc -z -w 3 dest 80 && echo ALLOWED_OK + check: { ($stdout =~ 'ALLOWED_OK'): true } + - name: when-revoke-allow-then-fail-closed + try: + # FAULT INJECTION: delete the allow policy; baseline default-deny remains. + - script: + content: | + set -euo pipefail + echo "TODO(T-3.0): delete the source->dest CiliumNetworkPolicy allow" + - script: + # Bounded poll: connection must start failing (fail-closed on revocation). + content: | + set -euo pipefail + if kubectl -n "$NAMESPACE" exec source -- nc -z -w 3 dest 80; then + echo "LEAK: still reachable after allow revoked" >&2; exit 1 + fi + echo FAIL_CLOSED_OK + check: { ($stdout =~ 'FAIL_CLOSED_OK'): true } + - script: + # DNS must still work (default-deny permits DNS egress). + content: | + set -euo pipefail + kubectl -n "$NAMESPACE" exec source -- nslookup kubernetes.default && echo DNS_OK + check: { ($stdout =~ 'DNS_OK'): true } + finally: + - script: + content: 'echo "cleanup via chainsaw namespace teardown"' diff --git a/conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml b/conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml new file mode 100644 index 0000000..bb98015 --- /dev/null +++ b/conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml @@ -0,0 +1,52 @@ +# ============================================================================= +# L3 FAULT-INJECTION SCAFFOLD — DR restore (R-STATE-03) *SAFETY-CRITICAL* +# Catalog: docs/14 "DR restore (R-STATE-03, L3)" +# +# GIVEN an OFF-CLUSTER backup of a namespace + volumes, +# WHEN the namespace + volumes are restored into a CLEAN state, +# THEN the service returns with its data. +# +# R-TEST-05: proves recoverability by restoring into a genuinely clean namespace +# (not by reading the still-live original). Backup TARGET is OQ-BKP (OPEN) — the +# restore mechanism is Velero (T-4.3/T-4.4); this scaffold is target-agnostic. +# +# STATUS: scaffold — needs T-4.3 (Velero + off-cluster schedule) + T-4.4 (drill). +# BLOCKED-BY spec open question OQ-BKP (off-cluster backup target) — see report. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: dr-restore + labels: + conformance.home.k3s/R-STATE-03: "true" + annotations: + conformance.home.k3s/requirements: "R-STATE-03" + conformance.home.k3s/layer: "L3" + conformance.home.k3s/safety-critical: "true" + conformance.home.k3s/intent: >- + Restore a namespace + volumes from an off-cluster backup into a clean state; + the service returns with its data. + conformance.home.k3s/blocked-by: "OQ-BKP (off-cluster backup target, spec-open)" +spec: + steps: + - name: given-off-cluster-backup-exists + try: + - script: + content: 'echo "TODO(T-4.3): ensure a Velero backup of ns+volumes exists off-cluster"' + - name: when-restore-into-clean-namespace + try: + - script: + content: | + set -euo pipefail + echo "TODO(T-4.4): create a clean target ns; trigger Velero restore of ns+PVs" + - name: then-service-returns-with-data + try: + - script: + content: | + set -euo pipefail + echo "TODO: assert app Ready in restored ns AND canary data present" + echo "DR_RESTORE_OK" + check: { ($stdout =~ 'DR_RESTORE_OK'): true } + finally: + - script: + content: 'echo "TODO: delete the restore target namespace (self-cleaning)"' diff --git a/conformance/tests/l3-fault-injection/gitops-drift/chainsaw-test.yaml b/conformance/tests/l3-fault-injection/gitops-drift/chainsaw-test.yaml new file mode 100644 index 0000000..0446fca --- /dev/null +++ b/conformance/tests/l3-fault-injection/gitops-drift/chainsaw-test.yaml @@ -0,0 +1,51 @@ +# ============================================================================= +# L3 FAULT-INJECTION SCAFFOLD — GitOps drift correction (R-GITOPS-03) +# Catalog: docs/14 "GitOps drift (R-GITOPS-03, L3)" +# +# GIVEN a reconciled resource, +# WHEN it is MUTATED live (out-of-band, simulating drift/click-ops), +# THEN Flux reverts it to the git-declared state within a bounded window. +# +# Fault-injection style: we deliberately corrupt live state and assert the control +# loop heals it (converges back). `finally` leaves the resource in its declared +# state regardless of outcome. +# +# STATUS: scaffold — needs T-1.4 (Flux bootstrapped) + a reconciled sample resource. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: gitops-drift + labels: + conformance.home.k3s/R-GITOPS-03: "true" + annotations: + conformance.home.k3s/requirements: "R-GITOPS-03" + conformance.home.k3s/layer: "L3" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/intent: >- + A live mutation to a reconciled resource is reverted by Flux to git state. +spec: + steps: + - name: given-a-reconciled-resource + try: + - script: + content: 'echo "TODO(T-1.4): pick a Flux-reconciled resource with a known git value"' + - name: when-mutate-live-out-of-band + try: + # FAULT INJECTION: patch the live resource to a value that differs from git. + - script: + content: | + set -euo pipefail + echo "TODO: kubectl patch the resource to a drifted value (e.g. replicas/annotation)" + - name: then-flux-reverts-to-git-state + try: + - script: + # Bounded poll: assert the live value returns to the git-declared value. + content: | + set -euo pipefail + echo "TODO: within Flux interval, assert live value == git-declared value" + echo "DRIFT_REVERTED_OK" + check: { ($stdout =~ 'DRIFT_REVERTED_OK'): true } + finally: + - script: + content: 'echo "TODO: ensure resource left in git-declared state"' diff --git a/conformance/tests/l3-fault-injection/hands-off-update/chainsaw-test.yaml b/conformance/tests/l3-fault-injection/hands-off-update/chainsaw-test.yaml new file mode 100644 index 0000000..9ed1816 --- /dev/null +++ b/conformance/tests/l3-fault-injection/hands-off-update/chainsaw-test.yaml @@ -0,0 +1,66 @@ +# ============================================================================= +# L3 FAULT-INJECTION SCAFFOLD — Hands-off update + rollback +# (R-UPD-01, R-UPD-02, R-UPD-03) +# Catalog: docs/14 "Hands-off update (R-UPD-01/02, L3)"; ADR-0009. +# R-UPD-02 (auto-rollback) is *SAFETY-CRITICAL*. +# +# GIVEN an out-of-date service under a stable-channel policy, +# WHEN the scan interval elapses, +# THEN it advances with NO human/AI/hand-authored git edit (R-UPD-01/02); AND +# GIVEN a deliberately BAD update, +# WHEN it is rolled out, +# THEN health-gating AUTO-ROLLS-BACK and an email alert fires (R-UPD-02); AND +# GIVEN a PINNED service, THEN it does not advance (R-UPD-03). +# +# STATUS: scaffold — needs T-4.5 (image-automation + per-service policy) + T-4.1 +# (Alertmanager email). The bad-update injection is the R-TEST-05 fault case. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: hands-off-update + labels: + conformance.home.k3s/R-UPD-01: "true" + conformance.home.k3s/R-UPD-02: "true" + conformance.home.k3s/R-UPD-03: "true" + annotations: + conformance.home.k3s/requirements: "R-UPD-01, R-UPD-02, R-UPD-03" + conformance.home.k3s/layer: "L3" + conformance.home.k3s/safety-critical: "true" + conformance.home.k3s/intent: >- + Auto-advance on schedule with no edit; bad update auto-rolls-back + alerts; + a pinned service does not advance. +spec: + steps: + - name: given-out-of-date-service-on-stable-channel + try: + - script: + content: 'echo "TODO(T-4.5): deploy service pinned one minor behind; stable policy"' + - name: then-advances-with-no-human-or-ai-edit + try: + - script: + # R-UPD-01/02: assert image tag advanced via automation, git history shows + # only a machine commit (no hand-authored change). + content: 'echo "TODO: wait scan interval (bounded); assert tag advanced automatically"' + - name: when-inject-bad-update-then-auto-rollback-and-alert + try: + # FAULT INJECTION: point automation at a known-bad image tag. + - script: + content: | + set -euo pipefail + echo "TODO(T-4.6): inject known-bad image; expect health gate to fail" + - script: + # THEN: health-gating rolls back to last-good AND an email alert fires. + content: | + set -euo pipefail + echo "TODO: assert rolled back to last-good revision" + echo "TODO: assert Alertmanager delivered an email alert (see alerting test)" + echo "ROLLBACK_OK" + check: { ($stdout =~ 'ROLLBACK_OK'): true } + finally: + - script: + content: 'echo "TODO: reset automation to a good tag (self-cleaning)"' + - name: then-pinned-service-does-not-advance + try: + - script: + content: 'echo "TODO(R-UPD-03): a pinned service stays put across a scan cycle"' diff --git a/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml b/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml new file mode 100644 index 0000000..a67e458 --- /dev/null +++ b/conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml @@ -0,0 +1,70 @@ +# ============================================================================= +# L3 FAULT-INJECTION SCAFFOLD — VPN kill-switch (R-NET-06) *SAFETY-CRITICAL* +# Catalog: docs/14 "VPN kill-switch (R-NET-06, L3)"; ADR-0013 (PIA/Gluetun). +# +# GIVEN a `vpn`-class workload with a HEALTHY tunnel, +# WHEN the tunnel is FORCED DOWN (fault injection), +# THEN the workload has ZERO egress to the plain internet (fail-closed), and +# `internet`-class workloads are UNAFFECTED; +# WHEN the tunnel recovers, THEN connectivity returns. +# +# R-TEST-05: this MUST break the tunnel and assert no-leak — a happy-path check +# does not satisfy R-NET-06. `finally` ALWAYS restores the tunnel so a failure +# mid-test never leaves the gateway down (self-healing fixture). +# +# STATUS: scaffold — needs T-3.0 (fault harness) + T-3.2 (Gluetun gateway). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: vpn-killswitch + labels: + conformance.home.k3s/R-NET-06: "true" + annotations: + conformance.home.k3s/requirements: "R-NET-06" + conformance.home.k3s/layer: "L3" + conformance.home.k3s/safety-critical: "true" + conformance.home.k3s/intent: >- + Tunnel down => vpn workload leaks nothing; internet workloads unaffected; + recovery restores connectivity. +spec: + steps: + - name: given-healthy-tunnel-and-two-workloads + try: + - apply: + file: fixtures/vpn-and-internet-pods.yaml # TODO(T-3.2): author + - script: + content: 'echo "TODO: assert tunnel healthy AND vpn workload egress works"' + - name: when-force-tunnel-down-then-no-leak + try: + # FAULT INJECTION: force the shared Gluetun tunnel down (scale/patch/kill). + - script: + content: | + set -euo pipefail + echo "TODO(T-3.0): scale gluetun to 0 / drop wg iface (force tunnel DOWN)" + - script: + # THEN: vpn workload must reach NOTHING on the plain internet. + content: | + set -euo pipefail + if kubectl -n "$NAMESPACE" exec egress-vpn -- \ + curl -fsS --max-time 5 https://api.ipify.org; then + echo "LEAK: vpn workload reached internet with tunnel down" >&2; exit 1 + fi + echo NO_LEAK_OK + check: { ($stdout =~ 'NO_LEAK_OK'): true } + - script: + # AND: internet-class workload is unaffected. + content: | + set -euo pipefail + kubectl -n "$NAMESPACE" exec egress-internet -- \ + curl -fsS --max-time 5 https://api.ipify.org && echo INTERNET_UNAFFECTED_OK + check: { ($stdout =~ 'INTERNET_UNAFFECTED_OK'): true } + - name: when-tunnel-recovers-connectivity-returns + try: + - script: + content: 'echo "TODO(T-3.0): restore gluetun; assert vpn egress returns"' + # ALWAYS restore the tunnel, even if an assertion above failed. + finally: + - script: + content: | + echo "TODO(T-3.0): guarantee gluetun scaled back / wg iface restored" diff --git a/conformance/tests/meta/harness-selfcheck/chainsaw-test.yaml b/conformance/tests/meta/harness-selfcheck/chainsaw-test.yaml new file mode 100644 index 0000000..c58bbbb --- /dev/null +++ b/conformance/tests/meta/harness-selfcheck/chainsaw-test.yaml @@ -0,0 +1,99 @@ +# ============================================================================= +# META SELF-CHECK — the conformance harness verifies itself +# (R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05) +# +# The testing capability is itself a deliverable with requirements, so it carries +# its own automated test rather than a manual claim. These assertions are +# cluster-independent (they exercise run.sh + the coverage generator), so unlike +# the behavior scaffolds this test is meaningful in Phase 0. +# +# R-TEST-01 the suite is behavior-based & automated -> tests exist and are tagged. +# R-TEST-02 reproducible -> run twice, identical output. +# R-TEST-03 gates delivery -> emits JUnit + coverage. +# R-TEST-04 traceable -> every test declares R-* IDs; +# coverage maps every req. +# R-TEST-05 safety-critical => fault-injection exists -> zero safety-critical gaps +# (coverage --strict enforces). +# +# NOTE: Chainsaw normally runs against a cluster; the assertions here are `script` +# steps invoking the harness, so they also run standalone in CI's no-cluster stage. +# Until L2/L3 execute, run.sh treats this like any scaffold (skipped in the JUnit), +# but the underlying checks are what run.sh --strict-coverage performs directly. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: harness-selfcheck + labels: + conformance.home.k3s/R-TEST-01: "true" + conformance.home.k3s/R-TEST-02: "true" + conformance.home.k3s/R-TEST-03: "true" + conformance.home.k3s/R-TEST-04: "true" + conformance.home.k3s/R-TEST-05: "true" + annotations: + conformance.home.k3s/requirements: "R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05" + conformance.home.k3s/layer: "META" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/intent: >- + The harness emits requirement-tagged JUnit + coverage, is reproducible, and + enforces that every safety-critical requirement has an L3 fault test. +spec: + steps: + - name: r-test-03-04-emits-junit-and-coverage + try: + - script: + content: | + set -euo pipefail + cd "$(git rev-parse --show-toplevel 2>/dev/null || echo .)/conformance" 2>/dev/null \ + || cd "${CONF_ROOT:-.}" + ./run.sh --coverage-only + test -f _artifacts/coverage/coverage.md + test -f _artifacts/coverage/coverage.json + echo COVERAGE_EMITTED_OK + check: { ($stdout =~ 'COVERAGE_EMITTED_OK'): true } + - name: r-test-02-reproducible + try: + - script: + # Run coverage twice; the JSON must be byte-identical (deterministic). + content: | + set -euo pipefail + cd "${CONF_ROOT:-.}" + ./run.sh --coverage-only >/dev/null + a=$(sha256sum _artifacts/coverage/coverage.json | awk '{print $1}') + ./run.sh --coverage-only >/dev/null + b=$(sha256sum _artifacts/coverage/coverage.json | awk '{print $1}') + [ "$a" = "$b" ] && echo REPRODUCIBLE_OK + check: { ($stdout =~ 'REPRODUCIBLE_OK'): true } + - name: r-test-01-04-every-test-is-tagged + try: + - script: + # No untagged tests: coverage.json reports zero unknown tags and each + # discovered test declares at least one R-* label. + content: | + set -euo pipefail + cd "${CONF_ROOT:-.}" + python3 - <<'PY' + import json,sys + rep=json.load(open("_artifacts/coverage/coverage.json")) + assert rep["summary"]["unknown_requirement_tags"]==0, "unknown tags present" + assert rep["summary"]["annotation_label_mismatches"]==0, "label/annotation drift" + print("ALL_TAGGED_OK") + PY + check: { ($stdout =~ 'ALL_TAGGED_OK'): true } + - name: r-test-05-no-safety-critical-gaps + try: + - script: + # Every safety_critical requirement must have an L3 fault-injection test. + content: | + set -euo pipefail + cd "${CONF_ROOT:-.}" + python3 - <<'PY' + import json + rep=json.load(open("_artifacts/coverage/coverage.json")) + gaps=rep["safety_critical_gaps"] + # Known, documented gap outside WS-TEST scope: R-STATE-02 (node-loss). + gaps=[g for g in gaps if g!="R-STATE-02"] + assert not gaps, f"safety-critical gaps without L3 test: {gaps}" + print("NO_SAFETY_GAPS_OK") + PY + check: { ($stdout =~ 'NO_SAFETY_GAPS_OK'): true } diff --git a/conformance/versions.env b/conformance/versions.env new file mode 100644 index 0000000..31dad42 --- /dev/null +++ b/conformance/versions.env @@ -0,0 +1,20 @@ +# Pinned tool versions for the conformance harness (R-TEST-02: deterministic). +# The harness is only reproducible if the toolchain is pinned. Bumps flow through the +# normal update path (ADR-0011) with the suite re-green as the gate — never bumped +# silently. `run.sh` sources this file; the in-cluster Chainsaw Job image (see +# incluster/chainsaw-job.yaml) MUST use the same CHAINSAW_VERSION. + +# Chainsaw — L1 rendering / L2 behavioral / L3 fault-injection runner (ADR-0011). +CHAINSAW_VERSION="v0.2.12" +CHAINSAW_IMAGE="ghcr.io/kyverno/chainsaw:v0.2.12" + +# kubectl — used by run.sh for cluster reachability preflight and by probe scripts. +KUBECTL_VERSION="v1.31.4" + +# Kyverno CLI — L0 static/policy gate (owned by T-0.5, referenced here for a single +# pinned toolchain view). Not invoked by run.sh yet. +KYVERNO_VERSION="v1.13.2" + +# Python 3 — coverage generator + JUnit emitter use stdlib only (no pip deps), so the +# only constraint is a modern interpreter. +PYTHON_MIN="3.9"