jakman85andClaude Opus 5
3ade1d440f
`has_todo = "TODO" in text` — a bare whole-file substring. The new L0 check
`placeholder-guard-coupling/check.sh` ENUMERATES placeholder idioms as data
(`SEED = ["TBD","TODO","FIXME",...]`) because detecting them is its entire job, so
the harness classified a 32-assertion executable check as a TODO scaffold.
Consequence: R-STATE-03 — SAFETY-CRITICAL — reported `exec=1 scaffold=1` and
`not-derivable`, because status.py requires all of a requirement's tagged tests to
be proven. A safety requirement mis-reported by a substring match, plus a standing
--strict-coverage conflict that a previous engineer worked AROUND by rewording
prose instead of fixing the detector.
Replaced with two REDUNDANT structural signals, derived from the corpus (all 43
tests read), not invented:
1. `# STATUS: scaffold` with the state as the FIRST token after `STATUS:`. Exactly
3 tests declare it — onboarding-completeness, alerting, gitops-drift — and they
are exactly the three real scaffolds. 19 others declare EXECUTABLE/RUNNABLE, so
a loose `.*scaffold` would have matched prose.
2. A TODO/FIXME marker CARRYING A MESSAGE in the body, where the body starts after
the header comment block. A marker that is the ENTIRE content of a quoted
literal is data, not a note, and is blanked before the scan. Every genuine
scaffold marker in the corpus carries a message, so mid-body comments still count.
The line-322 intent is preserved exactly: each file is classified on its own
structure, either signal in either file sets scaffold + annotation_conflict, the
`executable: "true"` annotation is consulted only AFTER the body verdict, and
ambiguity still only downgrades. Vocabulary stays TODO/FIXME only — TBD/PLACEHOLDER
are excluded because two checks use them in real executable code.
NON-VACUITY, and the redundancy is the point: deleting the `# STATUS: scaffold` line
from all three real scaffolds still catches all three via the body rule. The decisive
perturbation is the reverse — the exact file that caused the bug, degraded to a
TODO-only body while KEEPING its `STATUS: EXECUTABLE` header, its idiom-naming header
prose and its `executable: "true"` annotation, is caught as a scaffold WITH the
annotation conflict raised. The annotation still cannot outrank the body.
TAGGING: the agent RECOMMENDED KEEPING `R-STATE-03` on placeholder-guard-coupling and
reversed my lean, on a ground I had missed and which is load-bearing:
ci/touched-requirements.sh maps `bootstrap/*` -> R-STATE-03, and `run.sh
--requirements` resolves R-IDs to test dirs THROUGH THESE LABELS. Removing the tag
would reopen the exact merge-gate hole a984442 closed. It also measured the
counterfactual — with the tag removed R-STATE-03 becomes `stale-proof`, not `proven`
— so the tag is not what holds PROVEN at zero.
R-STATE-03 before -> after: `exec=1 scaffold=1` / not-derivable + strict conflict, to
`exec=2 scaffold=0` / not-derivable with an ACCURATE reason (no ledger row for the L0
check; dr-restore is stale-proof).
LEDGER MECHANISM CONFIRMED CORRECT, not worked around. R-STATE-02's node-loss proof
(sha 57d4df8c) went stale because a984442 touched its evidence set — attributed
precisely to the single path `ci/touched-requirements.sh`. Fail-closed and by design.
Re-proving needs a real L3 run recorded from a clean tree at HEAD-or-later: a
live-cluster action.
OBSERVATION FOR LATER (not changed): scan_implementation counts
ci/touched-requirements.sh as implementation evidence for EVERY R-ID it names, so any
edit to that dispatch table invalidates every proof it mentions. Both current
stale/not-derivable verdicts trace to that one file. Fail-closed is the right
direction; whether a CI path-map should be implementation evidence is a separate call.
VERIFIED BY ME: `status.py --selftest` PASS (8 new assertions, both directions);
`--status --strict-coverage` exit 1 -> 0; `--layer l0` 11/11; the three
`STATUS: scaffold` files are exactly the three real scaffolds; kyverno L0 29/29 and
the full L0 stage green IN THE CONTAINER (the agent saw the stage skip on the host,
where the pinned kyverno CLI is absent). Full CI gate PASS. Counts unchanged:
coverage 48/51, scaffold-only 4, PROVEN 0 (stale 1).
Refs R-TEST-01, R-TEST-04, R-TEST-05, R-STATE-03.
Co-Authored-By: Claude Opus 5 <[email protected]>
145 lines
8.3 KiB
Markdown
145 lines
8.3 KiB
Markdown
# Conformance tests — layout & requirement-tagging convention
|
|
|
|
Every test is a **Chainsaw `Test`** in its own directory:
|
|
|
|
```
|
|
tests/
|
|
l0-static/ L0 — whole-repo invariants over the tracked tree (no cluster)
|
|
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
|
|
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
|
|
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
|
|
meta/ harness self-checks (R-TEST-*), cluster-independent
|
|
<test-name>/chainsaw-test.yaml
|
|
```
|
|
|
|
The layer a test belongs to is its subtree **and** its `.../layer` annotation; the
|
|
coverage generator cross-checks the two.
|
|
|
|
### L0 static (`l0-static/`) — what belongs here, and what does not
|
|
|
|
docs/14's L0 row is *"Pre-apply, in CI · manifests/ServiceDefinitions obey invariants"*.
|
|
Kyverno (`platform/kyverno/`) covers the invariants expressible over a **single
|
|
resource**. This subtree covers the L0 invariants that are **properties of the repository
|
|
as a whole** and that no per-resource policy can state — e.g. *no plaintext secret exists
|
|
anywhere in the tracked tree*, *no tracked RoleBinding grants a workload identity a
|
|
privileged role*, *the spec-immutability gate actually blocks a modified spec*.
|
|
|
|
Each `l0-static/<name>/` dir carries a **`check.sh`**: a self-contained offline assertion
|
|
script (git + Python stdlib/PyYAML; no cluster, no network) that exits 0/1. It is executed:
|
|
|
|
- by `conformance/run.sh --layer l0` (and as the offline leg of a full run), via
|
|
`lib/run_l1_offline.py`, producing a REAL pass/fail JUnit entry — never a skip; and
|
|
- by the merge gate's **`ci/l0.sh`** stage, so these assertions run on every PR; and
|
|
- by the wrapping Chainsaw `Test`, which runs the SAME `check.sh`, so an in-cluster run
|
|
executes byte-identical assertions.
|
|
|
|
Two authoring rules specific to this layer:
|
|
|
|
1. **State the gap.** A statically-provable half of a requirement is still only half.
|
|
Every `check.sh` header ends with an explicit *"what this does NOT prove"* list naming
|
|
the cluster-layer remainder. Covering a requirement at L0 does not finish it.
|
|
2. **Exemptions must be self-policing.** Where a check must tolerate something (a
|
|
deliberately-plaintext policy fixture, an `.example` template), the exemption is
|
|
conditioned on a property that would break if the exemption were abused — e.g. the
|
|
plaintext Secret fixture is exempt *only while* a `kyverno-test.yaml` asserts the
|
|
policy rejects it. Never a bare path allowlist.
|
|
|
|
## Requirement-tagging convention (authoritative)
|
|
|
|
A test declares the requirement(s) it verifies **three** ways; the label is the source
|
|
of truth, the annotations must agree with it (the coverage generator fails on drift):
|
|
|
|
```yaml
|
|
metadata:
|
|
name: default-deny
|
|
labels:
|
|
# ONE label per verified R-* ID. Machine-selectable + coverage source of truth.
|
|
conformance.home.k3s/R-NET-01: "true"
|
|
annotations:
|
|
# Human-readable canonical list — MUST equal the label set.
|
|
conformance.home.k3s/requirements: "R-NET-01"
|
|
# Declared layer (L0|L1|L2|L3|META) — must match the subtree.
|
|
conformance.home.k3s/layer: "L2"
|
|
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
|
|
conformance.home.k3s/safety-critical: "false"
|
|
# Given/When/Then intent, for the coverage report and reviewers.
|
|
conformance.home.k3s/intent: "Given ... When ... Then ..."
|
|
```
|
|
|
|
Why a label **per requirement** (not a comma list): Kubernetes label *values* can't hold
|
|
a comma-separated list, and `chainsaw test --selector` selects on label **keys/values**.
|
|
One `conformance.home.k3s/<R-ID>=true` label per requirement makes both subset-selection
|
|
and coverage extraction exact and robust.
|
|
|
|
### How this drives the tooling
|
|
- **Subset runs (merge gate):** `run.sh --requirements R-NET-01,R-NET-06` resolves the
|
|
R-IDs to the set of test dirs carrying those labels (union, deduped) and runs each once.
|
|
- **Coverage (`lib/coverage.py`):** reads these labels from every `chainsaw-test.yaml`,
|
|
maps each registry requirement -> its verifying test(s), and flags gaps + any
|
|
label/annotation mismatch + tags not in the registry.
|
|
- **JUnit:** results (real or skipped-scaffold) carry the R-IDs in `classname` +
|
|
properties, so the JUnit itself stays traceable (R-TEST-04).
|
|
|
|
## Authoring rules (R-TEST-01/02/06)
|
|
- **Behavior, not existence.** Assert what the system *does* (a probe cannot connect),
|
|
never that an object merely exists (a policy can exist and not enforce).
|
|
- **Given/When/Then** structure with `given-*/when-*/then-*` step names.
|
|
- **Hermetic & self-cleaning.** Rely on Chainsaw's per-test namespace + auto-teardown;
|
|
put any out-of-namespace fixture cleanup in a `finally` block. A leaked fixture is a
|
|
failure.
|
|
- **Bounded, deterministic waits.** Use assert/error polling within the config timeouts;
|
|
never `sleep`-and-hope.
|
|
- **Safety-critical => fault-injection.** For a safety-critical requirement, actually
|
|
break the thing and assert fail-closed / no-data-loss, with `finally` restoring state.
|
|
- **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to
|
|
the PM — do not relax the assertion or the requirement.
|
|
- **Never write a per-environment value into a test (R-ENV-02).** No hostname suffix, no
|
|
Gateway VIP, no LB-pool range. One authored suite must run green against dev *and* prod,
|
|
so a test **asks the cluster it is pointed at** what it is. The mechanism is a chainsaw
|
|
binding over the same ConfigMap Flux substitutes from:
|
|
|
|
```yaml
|
|
spec:
|
|
bindings:
|
|
- name: baseDomain
|
|
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
|
|
```
|
|
|
|
Declare it at **test level** (step `outputs`/`bindings` are step-scoped), then consume it:
|
|
`($baseDomain)` for a pod `env:` value, `(join('.', ['myhost', $baseDomain]))` where a
|
|
full hostname is needed — including inside a `fixtures/*.yaml` applied with `apply: file:`,
|
|
which chainsaw templates with the same bindings. Prefer reading the *actual* observed value
|
|
over restating a range (e.g. assert the DNS answer equals `GATEWAY_VIP`, or equals the LB IP
|
|
the Service was assigned) — it is both stricter and environment-independent. Never take the
|
|
environment from a CLI flag or an exported variable: a flag someone forgets is how a suite
|
|
ends up silently asserting the other cluster's expectations. L1 goldens are the one
|
|
exception: they are rendering fixtures and carry the literal `${BASE_DOMAIN}` token.
|
|
|
|
### How a scaffold declares itself (and how `status.py` tells one from a verifier)
|
|
|
|
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
|
|
later roadmap phase delivers the capability under test. `lib/status.py` classifies each
|
|
test EXECUTABLE or SCAFFOLD from the test's own files — `chainsaw-test.yaml`,
|
|
`render-check.sh`, `check.sh` — and either signal below **outranks**
|
|
`conformance.home.k3s/executable: "true"`, so a placeholder body can never be reported as
|
|
an executable test. Declare a scaffold in **both** positions:
|
|
|
|
1. **`# STATUS: scaffold — <what is missing>`** in the file's header comment block. The
|
|
declared state must be the first token after `STATUS:` (executable tests say
|
|
`# STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`).
|
|
2. **A `TODO`/`FIXME` marker carrying a message, in the body** — the file from its first
|
|
non-comment line onward. `echo "TODO(T-4.2): inject a fault to fire a rule"`,
|
|
or an inline comment on a missing fixture:
|
|
`file: fixtures/smtp-sink.yaml # TODO(T-4.1): hermetic SMTP capture sink`.
|
|
|
|
What is **not** a scaffold marker, and why the distinction is load-bearing: a marker
|
|
mentioned in the **header prose** (the Given/When/Then and the mandatory "what this does
|
|
NOT prove" list), and a marker that is the **entire content of a quoted literal** — a
|
|
placeholder *detector*'s own vocabulary, e.g. `SEED = ["TBD", "TODO", "FIXME", ...]` in
|
|
`l0-static/placeholder-guard-coupling/check.sh`. A whole-file `"TODO" in text` test used to
|
|
classify that 32-assertion check as a scaffold and, through it, report the safety-critical
|
|
R-STATE-03 as `exec=1 scaffold=1`. If your check must hold a marker as data, keep each
|
|
idiom a **standalone quoted literal in a list**; a marker with text attached (a regex
|
|
alternation, several idioms in one string) still classifies as a scaffold, because
|
|
ambiguity here only ever downgrades.
|