jakman85andClaude Opus 5
87c3d83dff
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3 SKIPPED on every gate run since it landed, because its check keyed its whole existence on `bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is no deletion commit for it anywhere; it landed only on unmerged prod-standup branches (`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is the quiet version of the failure it was written to prevent. The skip was also hiding a RED. Run against the real `standup.sh` from `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both documented there as load-bearing). It would have false-blocked the merge landing its own input. NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard: HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE cluster-vars value; an empty effective value is STATED by the overlay rather than inherited in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the 2026-09-05 outcome asserted at the source instead of at the guard — if prod's S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does not exist. Input surface is `clusters/`, all of it on `main`. HALF B (self-activating): the real predicate extracted and executed against those same values, the consumer trace, and the mid-string anchoring property. The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded: any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at `bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When no guard exists the gap is printed loudly every run instead of swallowed as a skip. LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is always a defect in an overlay, while `""` is held to the explicitness rule instead — and still counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard author's declaration that empty is not a usable target. Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present, reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34. L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged — R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps. Also: * `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale comment corrected. It claimed standup.sh "carries the placeholder guards", describing a file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the change that lands a placeholder guard selected by nothing. * `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8) locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather than a silent loss of selection. All three ci/tests pass (12 + 31 + 15). * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a placeholder-shaped overlay value" is no longer true; corrected. * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that can break it attributes a requirement it declares. Co-Authored-By: Claude Opus 5 <[email protected]>
110 lines
7.5 KiB
YAML
110 lines
7.5 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — no environment overlay ships a placeholder value, and the guard that owns
|
|
# one refuses it
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
#
|
|
# THE BUG CLASS THIS CLOSES (it has now bitten three times in one file). A guard in
|
|
# `bootstrap/standup.sh` keys on the SHAPE of a value that lives in
|
|
# `clusters/base/cluster-vars.yaml`, and nothing tied the two together. On 2026-09-05 the
|
|
# S3 placeholder changed from `TBD-PROD-S3-BACKUP-ENDPOINT` to
|
|
# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` (commit 32d6766); the three guards
|
|
# matched the glob `TBD-*`, a PREFIX match, and `fnmatch('https://TBD-R2-…','TBD-*')` is
|
|
# False. All three guards silently stopped matching, and `standup.sh resume` would have
|
|
# resumed `velero` on prod against a nonexistent hostname: installs clean, reports healthy,
|
|
# NO BACKUPS EXIST. It was found by accident.
|
|
#
|
|
# WHY THIS TEST WAS RESTRUCTURED (2026-09-13). It was PERMANENTLY SKIPPED. `check.sh` exited
|
|
# 3 when `bootstrap/standup.sh` was absent — and that file has NEVER been on `main`: it was
|
|
# not deleted (no deletion commit exists anywhere), it landed only on unmerged prod-standup
|
|
# branches. So an incident-derived test reported SKIP on every gate run, and SKIPPED IS NOT
|
|
# GREEN here. The skip was also hiding a RED: run against the real `standup.sh` from
|
|
# `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions on false positives
|
|
# from treating the empty string as a placeholder for knobs cluster-vars authors empty on
|
|
# purpose. Both are fixed. The invariant is now split into a half that needs no guard and a
|
|
# half that activates when one lands, and THIS CHECK NEVER EXITS 3.
|
|
#
|
|
# GIVEN the repository as authored,
|
|
# THEN HALF A — always, needs no guard, input surface `clusters/`:
|
|
# (A1) no environment overlay's EFFECTIVE cluster-vars value is placeholder-shaped by
|
|
# an INDEPENDENT detector, for any knob no guard refuses (base is exempt: it is
|
|
# not a bootstrap target and its loud sentinels exist to be overridden — but a
|
|
# sentinel an overlay forgets to override IS caught, as that overlay's value);
|
|
# (A2) an empty effective value is STATED explicitly by the overlay, never inherited
|
|
# in silence — cluster-vars' own documented convention, and the rule that
|
|
# separates "authored empty" from "somebody forgot";
|
|
# (A3) a LOUD base sentinel is explicitly overridden by every overlay, so the
|
|
# `TBD-PROD-*` convention cannot be defeated by an omitted patch op;
|
|
# (A4) the detector itself recognises the exact 2026-09-05 string MID-STRING, accepts
|
|
# both real endpoints the repo ships, and does not call the empty string loud;
|
|
# (A5) ci/l0.sh runs the L0 layer and ci/gate.sh blocks the merge when it fails.
|
|
# HALF B — when a `*_is_placeholder` predicate exists ANYWHERE in the tree (the guard
|
|
# is DISCOVERED, not hardcoded, so a guard landing at bootstrap/standup.sh or moving to
|
|
# a sourceable library activates this half with no edit here):
|
|
# (B1) every authored value of a guarded knob the detector calls placeholder-shaped is
|
|
# REFUSED by the REAL predicate, extracted from the real file and executed (never
|
|
# reimplemented — see check.sh "HOW THE REAL GUARD IS EXERCISED"), and every
|
|
# usable value is ACCEPTED;
|
|
# (B2) every function that reads a guarded knob applies the guard to it (the "three
|
|
# guards died at once" regression);
|
|
# (B3) the guard matches its own vocabulary ANYWHERE in a value, not just as a prefix
|
|
# — the exact anchoring defect above — and still accepts a real endpoint;
|
|
# (B4) the extraction is faithful, exactly one file and one definition own the
|
|
# vocabulary, and the check fails LOUDLY rather than silently testing a copy if
|
|
# the guard stops being a self-contained predicate.
|
|
#
|
|
# STATUS: EXECUTABLE, and it RUNS — never SKIP. `check.sh` is a pure offline assertion
|
|
# script (standard library + PyYAML + bash); conformance/run.sh --layer l0 runs it directly
|
|
# and this Chainsaw Test wraps the SAME script.
|
|
#
|
|
# NOT proven: while no guard exists on `main`, HALF B IS NOT PROVEN, and the check SAYS SO on
|
|
# stdout every run rather than swallowing it as a skip — what is proven instead is the
|
|
# stronger-at-the-source half, that the values a guard would be asked about are not
|
|
# placeholders at all. Also not proven: that the endpoint is REACHABLE or that backups exist
|
|
# (runtime facts — `standup.sh s3check` probes the former from a pod, l3-fault-injection/
|
|
# dr-restore proves the latter); that guards for knobs OTHER than the placeholder-refusing
|
|
# ones are correct; and that the guard runs at all in a real standup (the dispatch path is
|
|
# shell, executed only against a live cluster). A1 reads cluster-vars, not every manifest: a
|
|
# placeholder hardcoded in a manifest rather than in a knob is out of scope. R-STATE-03 is
|
|
# tagged for the narrow static slice only: the authored backup target cannot be a placeholder.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: placeholder-guard-coupling
|
|
labels:
|
|
conformance.home.k3s/R-ENV-02: "true"
|
|
conformance.home.k3s/R-STATE-03: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03"
|
|
conformance.home.k3s/layer: "L0"
|
|
# NO `scope` ANNOTATION, DELIBERATELY — this invariant IS requirement-scoped, and was
|
|
# re-audited as such on 2026-09-13 after the restructure changed the input surface.
|
|
# HALF A reads clusters/base/cluster-vars.yaml plus clusters/<env>/kustomization.yaml;
|
|
# ci/touched-requirements.sh keys clusters/* to R-ENV-02 + R-GITOPS-01, so every path
|
|
# that can break HALF A attributes a requirement declared above. HALF B reads whichever
|
|
# file defines the guard; bootstrap/* is keyed to R-STATE-03 + R-GITOPS-01, which covers
|
|
# the place a standup guard lands. Requirement-scoped selection therefore cannot hide a
|
|
# breaking change, and declaring whole-repo scope would only add runtime, not coverage.
|
|
# The one residual: (A5) reads ci/l0.sh and ci/gate.sh, and ci/* attributes neither
|
|
# R-ENV-02 nor R-STATE-03 — but that wiring is separately and unconditionally asserted by
|
|
# the whole-repo-scoped tests/l0-static/whole-repo-invariants-wired, so (A5) here is
|
|
# belt-and-braces rather than the only proof.
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
No environment overlay can ship a placeholder-shaped cluster-vars value that nothing
|
|
refuses, and where a real bootstrap guard owns such a knob it actually recognises
|
|
every authored value as a placeholder — so a value change in one file cannot silently
|
|
disarm a guard in another and enable velero against an endpoint that does not exist.
|
|
spec:
|
|
steps:
|
|
- name: refuse-authored-placeholders-and-couple-them-to-any-real-guard
|
|
description: >-
|
|
Screen every authored effective cluster-vars value of every environment overlay
|
|
against an independent placeholder detector, assert empty values are stated rather
|
|
than inherited, and — when a *_is_placeholder predicate exists anywhere in the tree
|
|
— extract and execute the REAL guard against those same values.
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|