jakman85andClaude Opus 5
b389566641
All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`.
ci/l1.sh has passed --include-scope whole-repo since it was written, with a
comment saying no L1 test declared the scope yet and it was wired ahead of time
so the next whole-repo invariant "at any layer" would be picked up by annotation
alone. Nothing ever declared it. The consequence is that the entire L1 rendering
layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or
R-NET-05/06/07 — a fail-open hole for every other change in the repo.
The coupling requirement-scoped selection cannot see is that all three tests
exercise ONE code path: catalog/render.py against
schemas/servicedefinition.schema.json. Anything that changes rendered output
invalidates these goldens no matter which requirement the touched path maps to.
The 2026-09-04 external-dns merge is the proof, measured rather than asserted.
ci/touched-requirements.sh attributes platform/externaldns/helmrelease.yaml to
R-ENV-02, R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — none of the
rendering requirements — yet that change added
external-dns.alpha.kubernetes.io/controller: none
to the rendered output of all three goldens. Via --print-selection:
requirement-scoped only (the 2026-09-04 behaviour): 0 L1 tests selected
with --include-scope whole-repo (how ci/l1.sh runs): 3 L1 tests selected
So the merge gated GREEN while leaving every rendering golden stale, and the
staleness surfaced only on a later full run.
Also corrects the now-false comment in ci/l1.sh that said no L1 test declares the
scope, and records why the annotation is load-bearing there.
Verified: offline l0,l1,meta --include-scope whole-repo in the pinned toolbox —
25 tests, 0 failures, 0 errors, exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
58 lines
3.3 KiB
YAML
58 lines
3.3 KiB
YAML
# =============================================================================
|
|
# L1 RENDERING SCAFFOLD — egress class renders the kill-switched path
|
|
# (R-NET-05, R-NET-06, R-NET-07)
|
|
# Catalog: docs/14 L1 example "`egress: vpn` renders the kill-switched egress path"
|
|
#
|
|
# L1 verifies the ServiceDefinition -> rendered-resources mapping matches golden
|
|
# expectations, WITHOUT asserting runtime behavior. This is NOT a substitute for
|
|
# the L2/L3 behavior tests (R-TEST-01): rendering the policy != enforcing it.
|
|
#
|
|
# GIVEN a ServiceDefinition with `egress: vpn`,
|
|
# WHEN the renderer produces resources,
|
|
# THEN the output includes the egress-gateway steering + kill-switch construct
|
|
# (and `egress: internet` does NOT), and flipping the class only changes the
|
|
# egress path (R-NET-07), not the workload spec.
|
|
#
|
|
# STATUS: IMPLEMENTED (T-0.3). The render assertion is a pure offline check
|
|
# (render-check.sh: renderer + golden diff, no cluster). conformance/run.sh --layer l1
|
|
# and ci/l1.sh run render-check.sh directly WITHOUT a cluster (docs/14: "L1 rendering
|
|
# assertions run in CI without a cluster"). This Chainsaw Test wraps the SAME
|
|
# render-check.sh in a script step so an in-cluster Chainsaw run executes the identical
|
|
# assertion — rendering the policy, NOT enforcing it (enforcement is proven at L2/L3).
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: egress-class-render
|
|
labels:
|
|
conformance.home.k3s/R-NET-05: "true"
|
|
conformance.home.k3s/R-NET-06: "true"
|
|
conformance.home.k3s/R-NET-07: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-NET-05, R-NET-06, R-NET-07"
|
|
conformance.home.k3s/layer: "L1"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
|
|
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
|
|
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
|
|
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
|
|
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
|
|
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
|
|
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
|
|
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
|
|
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
|
|
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
|
|
conformance.home.k3s/scope: "whole-repo"
|
|
conformance.home.k3s/intent: >-
|
|
egress: vpn renders the egress-gateway + kill-switch path; internet does not.
|
|
spec:
|
|
steps:
|
|
- name: render-and-assert-egress-class
|
|
description: >-
|
|
Render the vpn + internet fixtures and assert: each matches its golden; the vpn
|
|
render carries the egress-class=vpn steering label and internet does not; and the
|
|
two renders differ ONLY in the egress-class value (R-NET-07).
|
|
try:
|
|
- script:
|
|
content: bash ./render-check.sh
|