Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever

Browse Source 
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3
SKIPPED on every gate run since it landed, because its check keyed its whole existence on
`bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is
no deletion commit for it anywhere; it landed only on unmerged prod-standup branches
(`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never
merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is
the quiet version of the failure it was written to prevent.

The skip was also hiding a RED. Run against the real `standup.sh` from
`work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from
treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on
purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both
documented there as load-bearing). It would have false-blocked the merge landing its own input.

NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard:

  HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE
  cluster-vars value; an empty effective value is STATED by the overlay rather than inherited
  in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the
  2026-09-05 outcome asserted at the source instead of at the guard — if prod's
  S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does
  not exist. Input surface is `clusters/`, all of it on `main`.

  HALF B (self-activating): the real predicate extracted and executed against those same
  values, the consumer trace, and the mid-string anchoring property.

The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded:
any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at
`bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already
recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When
no guard exists the gap is printed loudly every run instead of swallowed as a skip.

LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is
always a defect in an overlay, while `""` is held to the explicitness rule instead — and still
counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard
author's declaration that empty is not a usable target.

Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the
literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's
explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER
with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present,
reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the
pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34.

L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged —
R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps.

Also:
  * `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale
    comment corrected. It claimed standup.sh "carries the placeholder guards", describing a
    file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the
    change that lands a placeholder guard selected by nothing.
  * `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8)
    locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather
    than a silent loss of selection. All three ci/tests pass (12 + 31 + 15).
  * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a
    placeholder-shaped overlay value" is no longer true; corrected.
  * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that
    can break it attributes a requirement it declares.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-13 00:10:37 -05:00
1 parent 55b90849d1
commit 87c3d83dff
6 files changed
+716 -299
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml
+77 -44
View File
@@ -1,43 +1,71 @@
# =============================================================================
# L0 STATIC — a placeholder value in cluster-vars is REFUSED by the guard that owns it
# L0 STATIC — no environment overlay ships a placeholder value, and the guard that owns
# one refuses it
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# THE BUG CLASS THIS CLOSES (it has now bitten three times in one file). A guard in
# `bootstrap/standup.sh` keys on the SHAPE of a value that lives in
# `clusters/base/cluster-vars.yaml`, and nothing tied the two together. On 2026-09-05 the
# S3 placeholder changed from `TBD-PROD-S3-BACKUP-ENDPOINT` to
# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`; the three guards matched the glob
# `TBD-*`, a PREFIX match, and `fnmatch('https://TBD-R2-…','TBD-*')` is False. All three
# guards silently stopped matching, and `standup.sh resume` would have resumed `velero` on
# prod against a nonexistent hostname: installs clean, reports healthy, NO BACKUPS EXIST.
# It was found by accident. The guard is fixed (`s3_is_placeholder`, substring); the
# COUPLING was untested, which is what makes the class reintroducible.
# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` (commit 32d6766); the three guards
# matched the glob `TBD-*`, a PREFIX match, and `fnmatch('https://TBD-R2-…','TBD-*')` is
# False. All three guards silently stopped matching, and `standup.sh resume` would have
# resumed `velero` on prod against a nonexistent hostname: installs clean, reports healthy,
# NO BACKUPS EXIST. It was found by accident.
#
# WHY THIS TEST WAS RESTRUCTURED (2026-09-13). It was PERMANENTLY SKIPPED. `check.sh` exited
# 3 when `bootstrap/standup.sh` was absent — and that file has NEVER been on `main`: it was
# not deleted (no deletion commit exists anywhere), it landed only on unmerged prod-standup
# branches. So an incident-derived test reported SKIP on every gate run, and SKIPPED IS NOT
# GREEN here. The skip was also hiding a RED: run against the real `standup.sh` from
# `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions on false positives
# from treating the empty string as a placeholder for knobs cluster-vars authors empty on
# purpose. Both are fixed. The invariant is now split into a half that needs no guard and a
# half that activates when one lands, and THIS CHECK NEVER EXITS 3.
#
# GIVEN the repository as authored,
# WHEN the REAL predicate is extracted from bootstrap/standup.sh and executed by bash
# (never reimplemented — see check.sh "HOW THE REAL GUARD IS EXERCISED"),
# THEN (1) every authored value of a guarded knob that is placeholder-shaped by an
# INDEPENDENT detector is actually refused by that guard, in base and in every
# environment overlay;
# (2) no environment overlay carries a placeholder-shaped value for a knob no guard
# refuses;
# (3) every function in standup.sh that reads a guarded knob applies the guard to it
# (the "three guards died at once" regression);
# (4) the guard matches its own vocabulary ANYWHERE in a value, not just as a prefix
# — the exact anchoring defect above — and still accepts a real endpoint;
# AND (5) the extraction is faithful, and the check fails LOUDLY rather than silently
# testing a copy if the guard stops being a self-contained predicate.
# THEN HALF A — always, needs no guard, input surface `clusters/`:
# (A1) no environment overlay's EFFECTIVE cluster-vars value is placeholder-shaped by
# an INDEPENDENT detector, for any knob no guard refuses (base is exempt: it is
# not a bootstrap target and its loud sentinels exist to be overridden — but a
# sentinel an overlay forgets to override IS caught, as that overlay's value);
# (A2) an empty effective value is STATED explicitly by the overlay, never inherited
# in silence — cluster-vars' own documented convention, and the rule that
# separates "authored empty" from "somebody forgot";
# (A3) a LOUD base sentinel is explicitly overridden by every overlay, so the
# `TBD-PROD-*` convention cannot be defeated by an omitted patch op;
# (A4) the detector itself recognises the exact 2026-09-05 string MID-STRING, accepts
# both real endpoints the repo ships, and does not call the empty string loud;
# (A5) ci/l0.sh runs the L0 layer and ci/gate.sh blocks the merge when it fails.
# HALF B — when a `*_is_placeholder` predicate exists ANYWHERE in the tree (the guard
# is DISCOVERED, not hardcoded, so a guard landing at bootstrap/standup.sh or moving to
# a sourceable library activates this half with no edit here):
# (B1) every authored value of a guarded knob the detector calls placeholder-shaped is
# REFUSED by the REAL predicate, extracted from the real file and executed (never
# reimplemented — see check.sh "HOW THE REAL GUARD IS EXERCISED"), and every
# usable value is ACCEPTED;
# (B2) every function that reads a guarded knob applies the guard to it (the "three
# guards died at once" regression);
# (B3) the guard matches its own vocabulary ANYWHERE in a value, not just as a prefix
# — the exact anchoring defect above — and still accepts a real endpoint;
# (B4) the extraction is faithful, exactly one file and one definition own the
# vocabulary, and the check fails LOUDLY rather than silently testing a copy if
# the guard stops being a self-contained predicate.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
# PyYAML + bash); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test
# wraps the SAME script.
# STATUS: EXECUTABLE, and it RUNS — never SKIP. `check.sh` is a pure offline assertion
# script (standard library + PyYAML + bash); conformance/run.sh --layer l0 runs it directly
# and this Chainsaw Test wraps the SAME script.
#
# NOT proven: that the endpoint is REACHABLE or that backups exist (runtime facts —
# `standup.sh s3check` probes the former from a pod, l3-fault-injection/dr-restore proves
# the latter); that guards for knobs OTHER than the placeholder-refusing ones are correct;
# and that the guard runs at all in a real standup (the dispatch path is shell, executed
# only against a live cluster). R-STATE-03 is tagged for the narrow static slice only: the
# authored backup target cannot be a placeholder that the guard fails to recognise.
# NOT proven: while no guard exists on `main`, HALF B IS NOT PROVEN, and the check SAYS SO on
# stdout every run rather than swallowing it as a skip — what is proven instead is the
# stronger-at-the-source half, that the values a guard would be asked about are not
# placeholders at all. Also not proven: that the endpoint is REACHABLE or that backups exist
# (runtime facts — `standup.sh s3check` probes the former from a pod, l3-fault-injection/
# dr-restore proves the latter); that guards for knobs OTHER than the placeholder-refusing
# ones are correct; and that the guard runs at all in a real standup (the dispatch path is
# shell, executed only against a live cluster). A1 reads cluster-vars, not every manifest: a
# placeholder hardcoded in a manifest rather than in a knob is out of scope. R-STATE-03 is
# tagged for the narrow static slice only: the authored backup target cannot be a placeholder.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
@@ -50,27 +78,32 @@ metadata:
conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03"
conformance.home.k3s/layer: "L0"
# NO `scope` ANNOTATION, DELIBERATELY — this invariant IS requirement-scoped, and was
# audited as such (2026-09-12). Its entire input surface is bootstrap/standup.sh (the
# placeholder guards) plus clusters/base/cluster-vars.yaml and clusters/<env>/
# (the values they guard). ci/touched-requirements.sh keys bootstrap/* to
# R-STATE-03 + R-GITOPS-01 and clusters/* to R-ENV-02 + R-GITOPS-01 — so EVERY path that
# can break this check already attributes one of the two requirements it declares above.
# Requirement-scoped selection therefore cannot hide a breaking change here, and declaring
# whole-repo scope would only add runtime, not coverage.
# re-audited as such on 2026-09-13 after the restructure changed the input surface.
# HALF A reads clusters/base/cluster-vars.yaml plus clusters/<env>/kustomization.yaml;
# ci/touched-requirements.sh keys clusters/* to R-ENV-02 + R-GITOPS-01, so every path
# that can break HALF A attributes a requirement declared above. HALF B reads whichever
# file defines the guard; bootstrap/* is keyed to R-STATE-03 + R-GITOPS-01, which covers
# the place a standup guard lands. Requirement-scoped selection therefore cannot hide a
# breaking change, and declaring whole-repo scope would only add runtime, not coverage.
# The one residual: (A5) reads ci/l0.sh and ci/gate.sh, and ci/* attributes neither
# R-ENV-02 nor R-STATE-03 — but that wiring is separately and unconditionally asserted by
# the whole-repo-scoped tests/l0-static/whole-repo-invariants-wired, so (A5) here is
# belt-and-braces rather than the only proof.
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every placeholder-shaped value authored in cluster-vars, for any environment
overlay, is actually recognised as a placeholder by the real bootstrap guard that
is supposed to refuse it — so a value change in one file cannot silently disarm a
guard in another and enable velero against an endpoint that does not exist.
No environment overlay can ship a placeholder-shaped cluster-vars value that nothing
refuses, and where a real bootstrap guard owns such a knob it actually recognises
every authored value as a placeholder — so a value change in one file cannot silently
disarm a guard in another and enable velero against an endpoint that does not exist.
spec:
steps:
- name: couple-authored-placeholders-to-the-real-guard
- name: refuse-authored-placeholders-and-couple-them-to-any-real-guard
description: >-
Extract the real placeholder predicate from bootstrap/standup.sh, run it against
every authored cluster-vars value the repo actually carries, and assert the guard
refuses each one an independent detector calls a placeholder.
Screen every authored effective cluster-vars value of every environment overlay
against an independent placeholder detector, assert empty values are stated rather
than inherited, and — when a *_is_placeholder predicate exists anywhere in the tree
— extract and execute the REAL guard against those same values.
try:
- script:
content: bash ./check.sh