T-0.4 Chainsaw conformance harness + coverage + behavior scaffolds (R-TEST-01..06)

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-16 00:08:32 -05:00
1 parent a778d923b6
commit 9d8e03892f
31 files changed
+2445 -2

No files matched your search

@@ -0,0 +1,98 @@
# =============================================================================
# L2 BEHAVIORAL SCAFFOLD — Default-deny east-west (R-NET-01)
# Catalog: docs/14-testing-and-conformance.md "Default-deny (R-NET-01)"
# Spec: docs/03-networking-isolation-and-egress.md "Default-deny baseline"
#
# EXEMPLAR: this file is the reference for the tagging convention and step
# style. Other scaffolds follow the same shape (see tests/README.md).
#
# GIVEN a fresh workload in a namespace with no allow rules,
# WHEN it attempts any in-cluster connection except DNS,
# THEN the connection FAILS (fail-closed); DNS still succeeds.
#
# This asserts observable BEHAVIOR (a probe cannot connect), not the mere
# existence of a NetworkPolicy object (R-TEST-01). Safety-relevant: the
# fail-closed direction is also exercised under fault-injection at L3
# (kill-switch / drift), but the baseline deny is a behavioral must-pass.
#
# STATUS: scaffold — cannot run until Phase 2 delivers the default-deny
# baseline (T-2.2). Chainsaw auto-creates/cleans the namespace (hermetic).
# TODOs mark fixtures that land as later phases deliver.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: default-deny
labels:
# --- requirement tags (authoritative; one label per verified R-* ID) ---
conformance.home.k3s/R-NET-01: "true"
annotations:
conformance.home.k3s/requirements: "R-NET-01"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/intent: >-
Given a bare workload with no allow rules, only DNS egress succeeds;
every other in-cluster connection is denied.
spec:
steps:
# -- GIVEN: a fresh, unprivileged probe workload with zero allow rules. The
# per-test namespace inherits the platform default-deny baseline (T-2.2).
- name: given-bare-probe-workload
try:
- apply:
# TODO(T-2.2): once the namespaced default-deny CiliumNetworkPolicy
# baseline exists, assert it is present AND that the probe below is
# actually blocked (existence != enforcement, R-TEST-01).
resource:
apiVersion: v1
kind: Pod
metadata:
name: probe
labels: { app: conformance-probe }
spec:
restartPolicy: Never
containers:
- name: probe
image: ghcr.io/kyverno/kyvernopre:latest # TODO: pin a curl/nc probe image
command: ["sleep", "3600"]
- assert:
resource:
apiVersion: v1
kind: Pod
metadata: { name: probe }
status: { phase: Running }
# -- THEN(DNS allowed): DNS egress MUST succeed even under default-deny.
- name: then-dns-succeeds
try:
- script:
# TODO(T-2.2): resolve an in-cluster name via kube-dns; expect rc=0.
content: |
set -euo pipefail
kubectl -n "$NAMESPACE" exec probe -- \
sh -c 'nslookup kubernetes.default.svc.cluster.local' && echo DNS_OK
check:
# bounded, deterministic: no sleeps; chainsaw retries within timeout.
($stdout =~ 'DNS_OK'): true
# -- THEN(everything else denied): a non-DNS in-cluster connection MUST fail.
- name: then-nondns-denied
try:
- script:
# TODO(T-2.2 / T-2.6): target an in-cluster service (e.g. Plex/Gateway)
# on a real port; expect the connection to FAIL closed (non-zero / timeout).
content: |
set -euo pipefail
if kubectl -n "$NAMESPACE" exec probe -- \
sh -c 'nc -z -w 3 <in-cluster-service> 80'; then
echo "LEAK: connection succeeded under default-deny" >&2
exit 1
fi
echo DENIED_OK
check:
($stdout =~ 'DENIED_OK'): true
# -- finally: chainsaw deletes the namespace + probe automatically; this block
# is the explicit hook for any out-of-namespace fixtures (none here).
finally:
- script:
content: 'echo "cleanup handled by chainsaw namespace teardown"'
@@ -0,0 +1,48 @@
# =============================================================================
# L2 BEHAVIORAL SCAFFOLD — Explicit allow / deny-undeclared (R-NET-02)
# Catalog: docs/14 "Explicit allow / deny-undeclared (R-NET-02)"
#
# GIVEN a service with declared `dependencies`,
# WHEN the source reaches a DECLARED dest port -> THEN it succeeds;
# WHEN the source reaches an UNDECLARED dest -> THEN it fails (negative test).
#
# STATUS: scaffold — needs T-2.3 (render dependencies -> CiliumNetworkPolicy).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: deny-undeclared
labels:
conformance.home.k3s/R-NET-02: "true"
annotations:
conformance.home.k3s/requirements: "R-NET-02"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/intent: >-
A declared dependency connects; an undeclared destination is blocked.
spec:
steps:
- name: given-source-with-declared-dependency
try:
- apply:
# TODO(T-2.3): apply a ServiceDefinition/rendered policy where `source`
# declares dependency on `dest:declared-port`; deploy source + two dests.
file: fixtures/source-and-dests.yaml # TODO: author fixtures
- name: then-declared-succeeds
try:
- script:
content: |
set -euo pipefail
kubectl -n "$NAMESPACE" exec source -- nc -z -w 3 declared-dest 80 \
&& echo ALLOW_OK
check: { ($stdout =~ 'ALLOW_OK'): true }
- name: then-undeclared-denied
try:
- script:
content: |
set -euo pipefail
if kubectl -n "$NAMESPACE" exec source -- nc -z -w 3 undeclared-dest 80; then
echo "LEAK: undeclared destination reachable" >&2; exit 1
fi
echo DENY_OK
check: { ($stdout =~ 'DENY_OK'): true }
@@ -0,0 +1,53 @@
# =============================================================================
# L2 BEHAVIORAL SCAFFOLD — DNS / exposure (R-DNS-01, R-DNS-02, R-DNS-03)
# Catalog: docs/14 "DNS/exposure (R-DNS-*)"
#
# GIVEN a service,
# WHEN it is resolved,
# THEN the record exists (auto-managed) and internal clients hit the internal
# Gateway; an `internal` service is unreachable publicly while a `public`
# service is reachable + TLS-valid.
#
# STATUS: scaffold — needs T-2.5 (ExternalDNS + Gateway + split-horizon PiHole).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: dns-exposure
labels:
conformance.home.k3s/R-DNS-01: "true"
conformance.home.k3s/R-DNS-02: "true"
conformance.home.k3s/R-DNS-03: "true"
annotations:
conformance.home.k3s/requirements: "R-DNS-01, R-DNS-02, R-DNS-03"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/intent: >-
Record auto-appears; internal clients resolve to the internal Gateway;
internal is not public, public is reachable + TLS-valid.
spec:
steps:
- name: given-a-declared-service
try:
- script:
content: 'echo "TODO(T-2.5): onboard a service via ServiceDefinition"'
- name: then-record-exists-auto-managed
try:
- script:
# R-DNS-01/02: the <service>.home.keithjackson.dev record appears with
# no manual step (ExternalDNS from declared intent).
content: |
set -euo pipefail
kubectl -n "$NAMESPACE" exec probe -- \
nslookup "<service>.home.keithjackson.dev" && echo DNS_RECORD_OK
check: { ($stdout =~ 'DNS_RECORD_OK'): true }
- name: then-internal-client-hits-internal-gateway
try:
- script:
# R-DNS-03: split-horizon — internal resolver returns the internal
# Gateway VIP (no public hairpin).
content: 'echo "TODO(T-2.5): assert resolved IP == internal Gateway VIP"'
- name: then-internal-unreachable-publicly-public-reachable
try:
- script:
content: 'echo "TODO: assert internal not public; public reachable + TLS-valid"'
@@ -0,0 +1,60 @@
# =============================================================================
# L2 BEHAVIORAL SCAFFOLD — Egress class (R-NET-05, R-NET-07)
# Catalog: docs/14 "Egress class (R-NET-05)" + "Egress flip (R-NET-07)"
#
# GIVEN one `egress: internet` and one `egress: vpn` workload,
# WHEN each queries an external echo of its own public IP,
# THEN internet shows the home/WAN IP and vpn shows the VPN (PIA) IP.
# AND (R-NET-07) flipping the class in git re-routes with no workload rewrite.
#
# STATUS: scaffold — needs T-3.1/T-3.2 (egress gateway + PIA/Gluetun).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: egress-class
labels:
conformance.home.k3s/R-NET-05: "true"
conformance.home.k3s/R-NET-07: "true"
annotations:
conformance.home.k3s/requirements: "R-NET-05, R-NET-07"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/intent: >-
internet-class egress shows WAN IP; vpn-class shows VPN IP; git flip re-routes.
spec:
steps:
- name: given-two-egress-classes
try:
- apply:
file: fixtures/internet-and-vpn-pods.yaml # TODO(T-3.1): author fixtures
- name: then-internet-shows-wan-ip
try:
- script:
# external echo of public IP; assert it is NOT the known VPN egress range.
content: |
set -euo pipefail
ip=$(kubectl -n "$NAMESPACE" exec egress-internet -- \
curl -fsS https://api.ipify.org)
echo "internet_ip=$ip"
# TODO: assert ip is in home/WAN set, not the PIA range.
check: { ($stdout =~ 'internet_ip='): true }
- name: then-vpn-shows-vpn-ip
try:
- script:
content: |
set -euo pipefail
ip=$(kubectl -n "$NAMESPACE" exec egress-vpn -- \
curl -fsS https://api.ipify.org)
echo "vpn_ip=$ip"
# TODO: assert ip is a PIA egress address (distinct from WAN IP).
check: { ($stdout =~ 'vpn_ip='): true }
- name: when-flip-egress-class-in-git
try:
- script:
# R-NET-07: patch the egress class (simulating a git/Flux change) and
# assert the observed egress path changes with NO pod recreation.
content: |
set -euo pipefail
echo "TODO(T-3.4): patch egress class, reconcile, re-check public IP,"
echo "assert path changed and pod UID/generation unchanged (no rewrite)."
@@ -0,0 +1,55 @@
# =============================================================================
# L2 BEHAVIORAL SCAFFOLD — Onboarding completeness (WS-APPS; R-GITOPS-02)
# Catalog: docs/14 "Onboarding completeness (WS-APPS)"
#
# GIVEN only a merged ServiceDefinition (no manual steps beyond merge),
# THEN the service is fully wired: compute + storage + policy + egress + TLS +
# DNS + monitoring + update — asserted by the UNION of the relevant tests
# above run against that specific service.
#
# This is an AGGREGATE gate: it composes the other behavior tests (default-deny,
# deny-undeclared, egress-class, trusted-tls, dns-exposure, data-survival,
# alerting, hands-off-update) scoped to one onboarded service. It maps to
# R-GITOPS-02 ("onboarding = a versioned catalog change") — the machine-checkable
# proxy for the WS-APPS "fully wired from ServiceDefinition alone" gate.
#
# STATUS: scaffold — meaningful only once the composed tests + a real onboarded
# service exist (Phase 6, T-6.*).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: onboarding-completeness
labels:
conformance.home.k3s/R-GITOPS-02: "true"
annotations:
conformance.home.k3s/requirements: "R-GITOPS-02"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/intent: >-
From a merged ServiceDefinition alone, the service is fully wired with no
manual step (union of compute/storage/policy/egress/TLS/DNS/obs/update tests).
conformance.home.k3s/composes: >-
default-deny, deny-undeclared, egress-class, trusted-tls, dns-exposure,
data-survival, alerting, hands-off-update
spec:
steps:
- name: given-only-a-merged-servicedefinition
try:
- script:
content: 'echo "TODO(T-6.*): apply ONE ServiceDefinition; perform NO manual steps"'
- name: then-fully-wired-union-of-behaviors
try:
- script:
# Each line below is a scoped invocation of the corresponding behavior
# check against the onboarded service; ALL must pass for completeness.
content: |
set -euo pipefail
echo "TODO: assert compute up (pods/VM Running)"
echo "TODO: assert PVC bound on longhorn/bulk (not ephemeral)"
echo "TODO: assert default-deny + declared dependencies only"
echo "TODO: assert egress class as declared"
echo "TODO: assert DNS record + trusted TLS"
echo "TODO: assert metrics/logs flowing + alert defined"
echo "TODO: assert update policy present"
echo "NO manual step was required beyond git merge"
@@ -0,0 +1,44 @@
# =============================================================================
# L2 BEHAVIORAL SCAFFOLD — SSO / forward-auth (R-IDP-02)
# Catalog: docs/14 onboarding (auth wiring) + docs/15 identity-and-access.
#
# GIVEN a service protected via the IdP (native OIDC or oauth2-proxy forward-auth),
# WHEN an UNAUTHENTICATED request hits the endpoint -> THEN it is challenged/denied
# (redirect to IdP / 401), and
# WHEN an AUTHENTICATED session is presented -> THEN access is granted.
#
# STATUS: scaffold — needs T-6.1c/T-6.1d (Keycloak + oauth2-proxy + auth render).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: sso-forward-auth
labels:
conformance.home.k3s/R-IDP-02: "true"
annotations:
conformance.home.k3s/requirements: "R-IDP-02"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/intent: >-
Unauthenticated requests are challenged by the IdP; authenticated ones pass.
spec:
steps:
- name: given-idp-protected-service
try:
- script:
content: 'echo "TODO(T-6.1d): onboard a service with auth block -> OIDC/forward-auth"'
- name: then-unauthenticated-is-challenged
try:
- script:
# Expect a 302 to the IdP or a 401 — never a 200 with the app body.
content: |
set -euo pipefail
code=$(kubectl -n "$NAMESPACE" exec probe -- \
curl -s -o /dev/null -w '%{http_code}' https://<service>.home.keithjackson.dev)
echo "unauth_code=$code"
case "$code" in 302|303|401|403) echo CHALLENGED_OK ;; *) exit 1 ;; esac
check: { ($stdout =~ 'CHALLENGED_OK'): true }
- name: then-authenticated-is-granted
try:
- script:
content: 'echo "TODO: obtain IdP session (test user), replay, assert 200"'
@@ -0,0 +1,50 @@
# =============================================================================
# L2 BEHAVIORAL SCAFFOLD — Trusted TLS (R-TLS-01, R-TLS-03)
# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)"
#
# GIVEN any exposed endpoint (including `internal`),
# WHEN its certificate is inspected from a probe,
# THEN it is valid, non-self-signed, unexpired, chains to a trusted root.
# (R-TLS-03: issuance also works for internal-only services.)
# Renewal-before-expiry (R-TLS-02) is a separate time/fault scenario — see the
# hands-off / cert-renewal coverage; not asserted here.
#
# STATUS: scaffold — needs T-2.4 (cert-manager + LE DNS-01 wildcard).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: trusted-tls
labels:
conformance.home.k3s/R-TLS-01: "true"
conformance.home.k3s/R-TLS-03: "true"
annotations:
conformance.home.k3s/requirements: "R-TLS-01, R-TLS-03"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/intent: >-
Exposed endpoints (incl. internal) present a valid, trusted, unexpired cert.
spec:
steps:
- name: given-exposed-endpoint
try:
- script:
content: 'echo "TODO(T-2.4/T-2.5): a service with a Gateway TLS listener"'
- name: then-cert-is-trusted-and-valid
try:
- script:
# Inspect from an ephemeral probe using the system trust store (no -k):
# a self-signed / expired / untrusted cert fails the handshake => rc!=0.
content: |
set -euo pipefail
host="<service>.home.keithjackson.dev" # TODO: real host
echo | openssl s_client -connect "${host}:443" -servername "${host}" \
-verify_return_error 2>/dev/null \
| openssl x509 -noout -issuer -enddate -checkend 0 \
&& echo TLS_VALID
# checkend 0 => rc!=0 if already expired; -verify_return_error => trusted chain.
check: { ($stdout =~ 'TLS_VALID'): true }
- name: then-internal-only-service-also-has-cert
try:
- script:
content: 'echo "TODO(R-TLS-03): repeat against an internal-only host"'
@@ -0,0 +1,46 @@
# =============================================================================
# L2 BEHAVIORAL SCAFFOLD — VM lifecycle (R-PLAT-03)
# Catalog: docs/14 "VM lifecycle (R-PLAT-03, L2/L3)"
#
# GIVEN a `kind: vm` ServiceDefinition,
# WHEN it is created/started/stopped/destroyed via kubectl/virtctl,
# THEN it behaves through the same K8s surface; default-deny + egress class
# apply to it. (Disk-survives-rebuild is the L3 partner test, see
# tests/l3-fault-injection/data-survival — VMs share R-STATE-01.)
#
# STATUS: scaffold — needs T-5.1/T-5.2 (KubeVirt + `kind: vm` path).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vm-lifecycle
labels:
conformance.home.k3s/R-PLAT-03: "true"
annotations:
conformance.home.k3s/requirements: "R-PLAT-03"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/intent: >-
A VM is created/started/stopped/destroyed via the K8s API; policy applies.
spec:
steps:
- name: given-vm-servicedefinition
try:
- apply:
file: fixtures/vm-servicedefinition.yaml # TODO(T-5.2): author
- name: when-start-and-assert-running
try:
- script:
content: |
set -euo pipefail
# virtctl start via the same kubectl surface; assert VMI Running.
kubectl -n "$NAMESPACE" patch vm demo --type merge -p '{"spec":{"running":true}}'
echo "TODO(T-5.3): wait VMI phase=Running (bounded), then STARTED_OK"
- name: then-default-deny-and-egress-apply-to-vm
try:
- script:
content: 'echo "TODO: run default-deny + egress-class probes from inside the VM"'
- name: when-stop-and-destroy
try:
- script:
content: 'echo "TODO(T-5.3): stop then delete VM via kubectl; assert gone"'