jakman85andClaude Opus 5
830983b791
The implementation landed in the preceding WIP commit had never been executed. This finishes it: the arithmetic is re-measured, the guard and its allowlist are attacked, and the header's one factual error is corrected. WHAT ENVSUBST ACTUALLY IS. The header credited `github.com/drone/envsubst`. The pinned flux 2.4.0 binary in the dev image embeds `github.com/fluxcd/pkg/[email protected]` — Flux's own fork of drone/envsubst ("This package is a fork of drone/envsubst", its README) — read straight out of the binary with `grep -ao`. Same behaviour, different module; the header now says so and says how to check, because a claim about a dependency that nobody can re-derive is how the last wrong assumption survived. RE-MEASURED, NOT ASSERTED. Every row of the DOLLAR-RUN ARITHMETIC table was re-run through `flux envsubst` (flux 2.4.0, DEFINED=VALUE) and every row holds: `$$`->`$`, `$$$`->`$$`, `$${D}`->`${D}`, `$$${D}`->`$VALUE` (odd run: the brace is LIVE), `$$$${D}`->`$${D}`, `DO $$ x $$;`->`DO $ x $;`, and `$$${ARR[0]}` dies with "missing closing brace" while `$${ARR[0]}` is fine. `--strict` output was byte-identical to non-strict, exit 0 both ways (`cmp`) — strict mode is blind to this whole class, which is why a source assertion is the only guard available. The header now carries the 30-second reproduction, flagged to be re-measured on any FLUX_VERSION bump. END TO END ON THE REAL PATH. With the pre-16b7fcd manifest restored, `kubectl kustomize apps/wave-0` emits `DO $$` — valid SQL, and exactly what the `kubectl apply` verification saw — while the same stream through `flux envsubst --strict` emits `DO $` and `$;` and exits 0. That two-command contrast IS the incident. THE GUARD IS NOT TRIVIALLY SATISFIABLE — 16 adversarial cases, all as intended, run against a throwaway copy of the tree so nothing could be left behind: * an unmarked `$$` under a substituted path FAILS (DOLLAR-COLLAPSE); so do an odd `$$${...}` (DOLLAR-HALF-ESCAPED) and an even `$$$${...}` (OVER-ESCAPED); * a stale entry FAILS three ways — context no longer in the file, context that only covers an already-accepted `$${` escape and so excuses nothing, and a path that is not a file in the repo; * the allowlist cannot blanket-exempt a path: a 2-char context is rejected (<24), a `.*` wildcard is matched VERBATIM and therefore rejected as stale, a 24-char context carrying no dollar run is rejected, a 4-char reason is rejected (<40) — and, the central case, a genuine entry covers only the span its context matches, so a SECOND `$$` elsewhere in the same file still fails. Two genuine entries for two real occurrences do pass, so the mechanism is usable rather than merely unsatisfiable; * the CLASS, not the instance: the pre-fix `DO $$ ... $$;` reconstructed in a NEW ConfigMap under apps/wave-2 fails at both dollar runs, and restoring the real pre-16b7fcd apps/wave-0/nextcloud-db.yaml verbatim fails the check at lines 384 and 391. The guard rejects the exact commit that broke production. The scratch fixtures lived only in the throwaway copy; `git status` on this branch stayed clean throughout, which is the other half of the last attempt's unfinished business. LEGITIMATE USES EXIST AND STAY GREEN. 18 `$${...}` escapes (unifi cloud-init, the gitea and nextcloud OIDC bootstrap scripts, two longhorn comments) pass unremarked, and the 2 non-braced `$$` occurrences are the two allowlist entries. Those two are shell comments inside block scalars, and `kubectl kustomize` confirms the comment text really does survive into the stream envsubst reads — so they are genuine, reaching content whose collapse is cosmetic, not a hypothetical the allowlist was invented to excuse. STILL NOT PROVEN, AND STILL NOT PROXIED: that a change was verified through a substituting path. Which command a human ran is a fact about a shell session; git stores the manifest, never the invocation, so it is not statically decidable here and no honest formulation exists. Grepping commit prose for `kubectl apply` would be satisfiable by rewording and would fire on the break-glass runbooks whose whole content is that command. The answer is to move the proof off the verification path and onto the source, which is what assertion (7) does. No ci/l0.sh stage: the `conformance.home.k3s/scope: "whole-repo"` annotation is what wires it, and whole-repo-invariants-wired fails the gate on a hand-written stage. Co-Authored-By: Claude Opus 5 <[email protected]>
Conformance tests — layout & requirement-tagging convention
Every test is a Chainsaw Test in its own directory:
tests/
l0-static/ L0 — whole-repo invariants over the tracked tree (no cluster)
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
meta/ harness self-checks (R-TEST-*), cluster-independent
<test-name>/chainsaw-test.yaml
The layer a test belongs to is its subtree and its .../layer annotation; the
coverage generator cross-checks the two.
L0 static (l0-static/) — what belongs here, and what does not
docs/14's L0 row is "Pre-apply, in CI · manifests/ServiceDefinitions obey invariants".
Kyverno (platform/kyverno/) covers the invariants expressible over a single
resource. This subtree covers the L0 invariants that are properties of the repository
as a whole and that no per-resource policy can state — e.g. no plaintext secret exists
anywhere in the tracked tree, no tracked RoleBinding grants a workload identity a
privileged role, the spec-immutability gate actually blocks a modified spec.
Each l0-static/<name>/ dir carries a check.sh: a self-contained offline assertion
script (git + Python stdlib/PyYAML; no cluster, no network) that exits 0/1. It is executed:
- by
conformance/run.sh --layer l0(and as the offline leg of a full run), vialib/run_l1_offline.py, producing a REAL pass/fail JUnit entry — never a skip; and - by the merge gate's
ci/l0.shstage, so these assertions run on every PR; and - by the wrapping Chainsaw
Test, which runs the SAMEcheck.sh, so an in-cluster run executes byte-identical assertions.
Two authoring rules specific to this layer:
- State the gap. A statically-provable half of a requirement is still only half.
Every
check.shheader ends with an explicit "what this does NOT prove" list naming the cluster-layer remainder. Covering a requirement at L0 does not finish it. - Exemptions must be self-policing. Where a check must tolerate something (a
deliberately-plaintext policy fixture, an
.exampletemplate), the exemption is conditioned on a property that would break if the exemption were abused — e.g. the plaintext Secret fixture is exempt only while akyverno-test.yamlasserts the policy rejects it. Never a bare path allowlist.
Requirement-tagging convention (authoritative)
A test declares the requirement(s) it verifies three ways; the label is the source of truth, the annotations must agree with it (the coverage generator fails on drift):
metadata:
name: default-deny
labels:
# ONE label per verified R-* ID. Machine-selectable + coverage source of truth.
conformance.home.k3s/R-NET-01: "true"
annotations:
# Human-readable canonical list — MUST equal the label set.
conformance.home.k3s/requirements: "R-NET-01"
# Declared layer (L0|L1|L2|L3|META) — must match the subtree.
conformance.home.k3s/layer: "L2"
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
conformance.home.k3s/safety-critical: "false"
# Given/When/Then intent, for the coverage report and reviewers.
conformance.home.k3s/intent: "Given ... When ... Then ..."
Why a label per requirement (not a comma list): Kubernetes label values can't hold
a comma-separated list, and chainsaw test --selector selects on label keys/values.
One conformance.home.k3s/<R-ID>=true label per requirement makes both subset-selection
and coverage extraction exact and robust.
How this drives the tooling
- Subset runs (merge gate):
run.sh --requirements R-NET-01,R-NET-06resolves the R-IDs to the set of test dirs carrying those labels (union, deduped) and runs each once. - Coverage (
lib/coverage.py): reads these labels from everychainsaw-test.yaml, maps each registry requirement -> its verifying test(s), and flags gaps + any label/annotation mismatch + tags not in the registry. - JUnit: results (real or skipped-scaffold) carry the R-IDs in
classname+ properties, so the JUnit itself stays traceable (R-TEST-04).
Authoring rules (R-TEST-01/02/06)
-
Behavior, not existence. Assert what the system does (a probe cannot connect), never that an object merely exists (a policy can exist and not enforce).
-
Given/When/Then structure with
given-*/when-*/then-*step names. -
Hermetic & self-cleaning. Rely on Chainsaw's per-test namespace + auto-teardown; put any out-of-namespace fixture cleanup in a
finallyblock. A leaked fixture is a failure. -
Bounded, deterministic waits. Use assert/error polling within the config timeouts; never
sleep-and-hope. -
Safety-critical => fault-injection. For a safety-critical requirement, actually break the thing and assert fail-closed / no-data-loss, with
finallyrestoring state. -
Never weaken a test to go green (R-TEST-06). If the spec can't be met, escalate to the PM — do not relax the assertion or the requirement.
-
Never write a per-environment value into a test (R-ENV-02). No hostname suffix, no Gateway VIP, no LB-pool range. One authored suite must run green against dev and prod, so a test asks the cluster it is pointed at what it is. The mechanism is a chainsaw binding over the same ConfigMap Flux substitutes from:
spec: bindings: - name: baseDomain value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)Declare it at test level (step
outputs/bindingsare step-scoped), then consume it:($baseDomain)for a podenv:value,(join('.', ['myhost', $baseDomain]))where a full hostname is needed — including inside afixtures/*.yamlapplied withapply: file:, which chainsaw templates with the same bindings. Prefer reading the actual observed value over restating a range (e.g. assert the DNS answer equalsGATEWAY_VIP, or equals the LB IP the Service was assigned) — it is both stricter and environment-independent. Never take the environment from a CLI flag or an exported variable: a flag someone forgets is how a suite ends up silently asserting the other cluster's expectations. L1 goldens are the one exception: they are rendering fixtures and carry the literal${BASE_DOMAIN}token.
How a scaffold declares itself (and how status.py tells one from a verifier)
Scaffolds in this tree carry TODO(T-x.y) markers where a fixture/assertion lands as a
later roadmap phase delivers the capability under test. lib/status.py classifies each
test EXECUTABLE or SCAFFOLD from the test's own files — chainsaw-test.yaml,
render-check.sh, check.sh — and either signal below outranks
conformance.home.k3s/executable: "true", so a placeholder body can never be reported as
an executable test. Declare a scaffold in both positions:
# STATUS: scaffold — <what is missing>in the file's header comment block. The declared state must be the first token afterSTATUS:(executable tests say# STATUS: EXECUTABLE/IMPLEMENTED/RUNNABLE).- A
TODO/FIXMEmarker carrying a message, in the body — the file from its first non-comment line onward.echo "TODO(T-4.2): inject a fault to fire a rule", or an inline comment on a missing fixture:file: fixtures/smtp-sink.yaml # TODO(T-4.1): hermetic SMTP capture sink.
What is not a scaffold marker, and why the distinction is load-bearing: a marker
mentioned in the header prose (the Given/When/Then and the mandatory "what this does
NOT prove" list), and a marker that is the entire content of a quoted literal — a
placeholder detector's own vocabulary, e.g. SEED = ["TBD", "TODO", "FIXME", ...] in
l0-static/placeholder-guard-coupling/check.sh. A whole-file "TODO" in text test used to
classify that 32-assertion check as a scaffold and, through it, report the safety-critical
R-STATE-03 as exec=1 scaffold=1. If your check must hold a marker as data, keep each
idiom a standalone quoted literal in a list; a marker with text attached (a regex
alternation, several idioms in one string) still classifies as a scaffold, because
ambiguity here only ever downgrades.