jakman85andClaude Opus 5
79c703bc79
Six discovery sites walked ROOT without pruning dot-directories, so they descended into .claude/worktrees/agent-*/ — up to 18 full copies of the repo, created for isolated agent runs and gitignored (.gitignore:43). The visible half was false FAILs: four ServiceMonitors reported as selecting no Service, and nine PrometheusRule manifests reported as "under test but not in the repo", all of them worktree paths. The whole-repo L0 gate was red for reasons that had nothing to do with authored surface. The dangerous half is quieter. A check that asks "does anything in the repo declare X" can be satisfied by a worktree copy instead of real source, so the same bug that invents failures can also manufacture a pass. That is the reason to fix all six sites rather than only the ones that happened to be failing. egress-paths-selected-by-rules already had this fix and documents it (it once saw 5 CiliumEgressGatewayPolicy objects instead of the 1 authored); this applies the same dot-directory predicate everywhere else, which skips dot-DIRECTORIES while still matching dot-FILES such as .sops.yaml. platform/monitoring/tests/run.sh is the other side of the same failure: its `--list` reported each manifest once per worktree, and the L0 check compares that listing against the tree. Both find calls now prune .claude alongside .git. `--list` returns 21 real pairs where it previously returned them ~19x over. Verified in the pinned toolbox: L0 --include-scope whole-repo is 21 cases, 0 failures, 0 skips, 0 unwired (was 4 failures, then 1). L1 3/3. The 18 stale worktrees are all on already-merged branches and remain on disk; cleaning them up is a separate change and is no longer load-bearing for the gate. Co-Authored-By: Claude Opus 5 <[email protected]>
711 lines
41 KiB
Bash
Executable File
711 lines
41 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# L0 static check (offline, no cluster) — placeholder-guard-coupling
|
|
# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)."
|
|
# — spec docs/02-environments-and-provisioning.md)
|
|
# R-STATE-03 ("Scheduled off-cluster backups + restore (DR)."
|
|
# — spec docs/04-storage-and-state.md)
|
|
#
|
|
# WHY THIS EXISTS — A GUARD KEYED ON A VALUE THAT LIVES IN ANOTHER FILE
|
|
# --------------------------------------------------------------------
|
|
# `bootstrap/standup.sh` refuses to resume tiers whose configuration is still a
|
|
# placeholder. Three of those guards keyed on the SHAPE of `S3_BACKUP_ENDPOINT`, which is
|
|
# authored in `clusters/base/cluster-vars.yaml`. On 2026-09-05 the placeholder changed
|
|
# from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`
|
|
# (the MinIO -> Cloudflare R2 move, commit 32d6766). The guards matched the glob `TBD-*` —
|
|
# a PREFIX match — and `fnmatch('https://TBD-R2-…','TBD-*')` is FALSE. All three guards
|
|
# silently stopped matching. `standup.sh resume` would then have resumed `velero` on prod
|
|
# against a hostname that does not exist: the HelmRelease installs clean, the Kustomization
|
|
# reports Ready, and NO BACKUPS EXIST. It was found by accident.
|
|
#
|
|
# WHY THIS CHECK WAS REWRITTEN (2026-09-13) — IT WAS SKIPPING FOREVER
|
|
# ------------------------------------------------------------------
|
|
# The original version keyed its ENTIRE existence on `bootstrap/standup.sh` and exited 3
|
|
# (SKIP) when that file was absent. Established by measurement, not assumption:
|
|
#
|
|
# * `bootstrap/standup.sh` HAS NEVER EXISTED ON `main`. It was not deleted — there is no
|
|
# deletion commit for it anywhere (`git log --all --diff-filter=D -- bootstrap/standup.sh`
|
|
# is empty). It lives on unmerged prod-standup branches (`work/prod-standup-campaign`,
|
|
# `prod/*`, `feat/prod-hardening`) and never landed. `git ls-tree main bootstrap/` lists
|
|
# only README.md, flux/, gitea/, sops/. docs/BULK-NFS-STORAGE.md §7 already recorded
|
|
# this, including the consequence: "nothing in the gate currently refuses a
|
|
# placeholder-shaped overlay value".
|
|
# * So this check reported SKIP on every gate run since it landed, and SKIPPED IS NOT
|
|
# GREEN in this repo. An incident-derived test that can only ever skip is the quiet
|
|
# version of the failure it was written to prevent.
|
|
# * Worse, the skip was HIDING A RED. Run against the real `standup.sh` from
|
|
# `work/prod-standup-campaign`, the original check FAILED 2 of 34 assertions — both
|
|
# false positives from treating the EMPTY STRING as placeholder-shaped for knobs whose
|
|
# emptiness is deliberately authored (`BACKUP_CREDENTIALS_SUFFIX` and
|
|
# `VELERO_BACKUP_PREFIX`, both `""` on dev). cluster-vars documents that emptiness as
|
|
# load-bearing ("THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING … an
|
|
# EMPTY VALUE IS SAFE UNDER StrictPostBuildSubstitutions"). So the check would have
|
|
# false-blocked the moment its own input landed. That is fixed below.
|
|
#
|
|
# THE HAZARD IS NOT GONE, so this coverage is NOT obsolete and was NOT deleted. The
|
|
# invariant has two independent halves, and only one of them needs `standup.sh`:
|
|
#
|
|
# HALF A — THE AUTHORED VALUES (provable on `main` today, needs NO guard).
|
|
# No environment overlay may ship a placeholder-shaped effective cluster-vars value.
|
|
# This is the OUTCOME the 2026-09-05 guard existed to prevent, asserted at the source
|
|
# instead of at the guard: if prod's `S3_BACKUP_ENDPOINT` is never a placeholder, velero
|
|
# cannot be pointed at a hostname that does not exist, whether or not a guard catches
|
|
# it. Its entire input surface is `clusters/`, all of which is on `main`.
|
|
# HALF B — THE COUPLING (activates automatically when a guard lands).
|
|
# The real predicate, extracted from the real file and executed, agrees with an
|
|
# INDEPENDENT detector about every authored value; every consumer of a guarded knob
|
|
# applies the guard; the guard matches its vocabulary ANYWHERE, not as a prefix.
|
|
#
|
|
# THIS CHECK NEVER EXITS 3. Half A always runs, against real files, with an assertion
|
|
# floor. Half B runs when a guard exists and is reported as NOT PROVEN when one does not —
|
|
# stated loudly in the output rather than swallowed as a skip.
|
|
#
|
|
# THE GUARD IS DISCOVERED, NOT HARDCODED (this is the "re-point" made automatic)
|
|
# -----------------------------------------------------------------------------
|
|
# The original hardcoded `bootstrap/standup.sh`, which is exactly why a file that never
|
|
# landed could disable the whole check. Instead, every tracked `*.sh` in the tree is
|
|
# scanned for a `<name>_is_placeholder()` definition:
|
|
# * exactly one such FILE -> that is the guard; Half B runs against it, wherever it is.
|
|
# A guard that lands at `bootstrap/standup.sh`, or moves to `bootstrap/lib/placeholder.sh`,
|
|
# is picked up with no edit to this check.
|
|
# * none -> Half A only, with the gap named in the output.
|
|
# * more than one file -> FAIL. Two files defining the placeholder vocabulary is the
|
|
# 2026-09-05 divergence with a second file's name on it; this check cannot say which
|
|
# one a standup run would source.
|
|
# `PLACEHOLDER_GUARD_FILE` overrides discovery (repo-relative) for negative fixtures.
|
|
# NOTE: discovery is shell-only, because the guard family is shell. If the refusal ever
|
|
# moves to a non-shell mechanism (a Kyverno policy, a CEL admission rule), discovery finds
|
|
# nothing and says so — re-point this check deliberately in that change.
|
|
#
|
|
# HOW THE REAL GUARD IS EXERCISED (and why not a copy)
|
|
# ---------------------------------------------------
|
|
# `standup.sh` is NOT sourceable: at top level it parses argv, hard-exits when
|
|
# --kubeconfig/--expect-server/<step> are absent, `cd`s to the repo root, exports
|
|
# KUBECONFIG, installs an EXIT trap and finally dispatches steps against a live cluster.
|
|
# So this check EXTRACTS the predicate's own source lines from the file and sources ONLY
|
|
# those, then calls the real function. That is the real bytes, not a reimplementation:
|
|
# * every function whose name ends `_is_placeholder` is discovered, not named here;
|
|
# * a duplicate definition of one is a failure (which of the two would run?);
|
|
# * the extracted body must be a SELF-CONTAINED predicate — `case`/`return` and nothing
|
|
# else, no command substitution, no call to another function. If the guard ever grows
|
|
# a dependency, extraction stops being faithful and THIS CHECK FAILS LOUDLY, naming the
|
|
# remedy (move the predicate into a small sourceable library) rather than quietly
|
|
# falling back to testing a copy of the logic;
|
|
# * a CANARY pins the extraction itself: the extracted function must refuse the empty
|
|
# string and accept a realistic endpoint.
|
|
# There is deliberately NO second implementation of the matching logic anywhere in this file.
|
|
#
|
|
# THE INDEPENDENT DETECTOR (and the empty-string distinction the old version got wrong)
|
|
# ------------------------------------------------------------------------------------
|
|
# Asking "does the guard agree with the guard?" proves nothing, so placeholder-shaped is
|
|
# decided by a detector written HERE, whose vocabulary is the UNION of two independent
|
|
# sources:
|
|
# * SEED — universal loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER, REPLACE-ME,
|
|
# CHANGE-ME, FILL-ME, SET-ME, XXXX), plus `<angle-bracket>` templates. NOT derived from
|
|
# any guard, so DELETING a guard arm cannot make a value stop being placeholder-shaped.
|
|
# * the guard's OWN vocabulary (when a guard exists), read out of its case-arms with the
|
|
# glob anchors STRIPPED. Stripping the anchors is what makes the comparison non-circular
|
|
# in the dimension that actually broke: the detector matches a word ANYWHERE,
|
|
# case-insensitively, and the guard must agree. A prefix-anchored guard cannot pass.
|
|
#
|
|
# LOUD vs EMPTY — the correction. A LOUD placeholder is a positive statement that a value
|
|
# was never decided, and it is always a defect in an overlay. The EMPTY STRING is not:
|
|
# cluster-vars authors `""` deliberately for suffix/prefix knobs, and Flux strict
|
|
# substitution fails on an UNDEFINED variable, not on one defined as `""`. So emptiness is
|
|
# NOT treated as a loud placeholder. It is held to a different, real rule instead (A2): an
|
|
# empty effective value must be STATED EXPLICITLY by that overlay, never inherited in
|
|
# silence — which is cluster-vars' own documented convention ("Both overlays state this
|
|
# knob explicitly … so neither inherits it silently"). Emptiness still counts as
|
|
# placeholder-shaped for a knob a GUARD covers, because a `""` case-arm in the guard is the
|
|
# guard author's declaration that empty is not a usable target for that knob.
|
|
#
|
|
# GIVEN the repository as authored,
|
|
# THEN HALF A (always):
|
|
# (A1) VALUES: no environment overlay's EFFECTIVE cluster-vars value is LOUD
|
|
# placeholder-shaped, for any knob that no guard refuses. (Base alone is exempt:
|
|
# it is not a bootstrap target — `flux bootstrap --path=clusters/<env>` — and its
|
|
# loud sentinels exist to be overridden. A base placeholder an overlay forgets to
|
|
# override IS caught, because it becomes that overlay's effective value.)
|
|
# (A2) EXPLICITNESS: an empty effective value is stated explicitly by that overlay.
|
|
# (A3) SENTINEL DISCIPLINE: if base carries a LOUD sentinel for a knob, every overlay
|
|
# states that knob explicitly — the `TBD-PROD-*` convention only works if no
|
|
# overlay can inherit the sentinel by omission.
|
|
# (A4) DETECTOR CANARIES: the detector recognises the exact 2026-09-05 string
|
|
# MID-STRING, accepts both real endpoints the repo actually ships, and does NOT
|
|
# call the empty string loud.
|
|
# (A5) LOAD-BEARING: ci/l0.sh runs the L0 conformance layer on every gate run and
|
|
# ci/gate.sh sets the merge block when that stage fails.
|
|
# HALF B (only when a guard file is discovered):
|
|
# (B1) VALUES: for every knob a guard is responsible for, every authored value — the
|
|
# base default AND each overlay's value — that the detector calls placeholder-
|
|
# shaped is REFUSED by the real guard, and every value it calls usable is
|
|
# ACCEPTED (a guard that refuses a real endpoint leaves velero suspended forever,
|
|
# the same class of silent failure).
|
|
# (B2) CONSUMERS: every function that reads a guarded knob (via `cvar`, following one
|
|
# hop of variable aliasing) applies a placeholder guard to it. This is the "all
|
|
# three guards died at once" regression stated as a property.
|
|
# (B3) ANCHORING: for each word in the guard's own vocabulary, the guard refuses a
|
|
# value carrying that word in the MIDDLE (not at the start) — the precise defect
|
|
# of 2026-09-05 — while still accepting a realistic endpoint.
|
|
# (B4) HARNESS: exactly one definition per predicate, one guard FILE in the tree, the
|
|
# extracted body verbatim and self-contained, the canary holds, no divergent
|
|
# second copy of the matching logic, and assertion floors keep a pass from
|
|
# degenerating into a no-op.
|
|
#
|
|
# WHAT THIS DOES *NOT* PROVE (honest gaps)
|
|
# - While no guard exists on `main`, HALF B IS NOT PROVEN AT ALL, and the check says so on
|
|
# stdout every run. What is proven is the stronger-at-the-source half: the values a
|
|
# guard would be asked about are not placeholders in the first place. This is a real
|
|
# narrowing of coverage versus the intent, and it is reported, not hidden.
|
|
# - Nothing about RUNTIME: not that the endpoint resolves, not that credentials work, not
|
|
# that a backup exists. `standup.sh s3check` probes reachability from a pod (operator-
|
|
# run, cluster-gated) and conformance/tests/l3-fault-injection/dr-restore proves restore.
|
|
# - Half B does not execute the dispatch path, so it cannot prove the guard is REACHED in
|
|
# a real run — only that, when reached, it refuses what it must. (B2) is a text-level
|
|
# taint trace over one file, not an interpreter.
|
|
# - It says nothing about guards keyed on shapes OTHER than placeholders (e.g. the
|
|
# EXTERNALDNS_TXT_OWNER_ID == 'default' test), nor about placeholder-shaped values
|
|
# outside cluster-vars (the Ansible inventory's BLOCKED/NOT-A-DEVICE sentinels are
|
|
# asserted by the node_baseline role itself, at play time).
|
|
# - A1 reads cluster-vars, not every manifest. A placeholder hardcoded in a manifest
|
|
# rather than in a knob is out of scope here.
|
|
#
|
|
# SELECTION. `ci/touched-requirements.sh` keys `clusters/*` -> R-ENV-02 (Half A's whole
|
|
# input surface) and `bootstrap/*` -> R-STATE-03 (where a guard lands), so every path that
|
|
# can break this check attributes a requirement it declares. Deliberately requirement-scoped,
|
|
# not whole-repo — see chainsaw-test.yaml.
|
|
# =============================================================================
|
|
set -euo pipefail
|
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
|
PYTHON="${PYTHON:-python3}"
|
|
GUARD_OVERRIDE="${PLACEHOLDER_GUARD_FILE:-}"
|
|
|
|
cd "${ROOT}"
|
|
|
|
# The authored half of the coupling is the one thing this check cannot do without.
|
|
if [ ! -f "${ROOT}/clusters/base/cluster-vars.yaml" ]; then
|
|
echo "CHECK FAIL (placeholder-guard-coupling): clusters/base/cluster-vars.yaml is absent — the authored values this check audits do not exist. That is not a SKIP: the knobs this invariant is about have been moved or deleted, and this check must be re-pointed in the SAME change." >&2
|
|
exit 1
|
|
fi
|
|
|
|
WORK="$(mktemp -d)"
|
|
trap 'rm -rf "${WORK}"' EXIT
|
|
|
|
# ---------------------------------------------------------------------------------------
|
|
# PHASE 1 (python): discover the guard, derive the corpus from the real files, and run
|
|
# every assertion that does not need a shell. Writes:
|
|
# ${WORK}/guards.sh the predicate source lines, verbatim, and nothing else (may be empty)
|
|
# ${WORK}/corpus.tsv <guard>\t<expect: placeholder|usable>\t<origin>\t<value> (may be empty)
|
|
# ${WORK}/findings.txt one line per failed assertion
|
|
# ${WORK}/counts.txt <assertions-run>
|
|
# ${WORK}/mode.txt the discovered guard's repo-relative path, or empty
|
|
# ---------------------------------------------------------------------------------------
|
|
"${PYTHON}" - "${ROOT}" "${WORK}" "${GUARD_OVERRIDE}" <<'PY'
|
|
import re
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
try:
|
|
import yaml
|
|
except ImportError: # pragma: no cover
|
|
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
|
file=sys.stderr)
|
|
raise SystemExit(1)
|
|
|
|
ROOT, WORK, OVERRIDE = Path(sys.argv[1]), Path(sys.argv[2]), sys.argv[3]
|
|
FAILURES: list[str] = []
|
|
CHECKS = 0
|
|
|
|
|
|
def check(cond, msg):
|
|
global CHECKS
|
|
CHECKS += 1
|
|
if not cond:
|
|
FAILURES.append(msg)
|
|
|
|
|
|
def bail(msg):
|
|
"""A failure that makes the rest of the check meaningless: report and stop."""
|
|
print(f"CHECK FAIL (placeholder-guard-coupling): {msg}", file=sys.stderr)
|
|
(WORK / "findings.txt").write_text(msg + "\n")
|
|
(WORK / "counts.txt").write_text("0\n")
|
|
(WORK / "corpus.tsv").write_text("")
|
|
(WORK / "guards.sh").write_text("")
|
|
(WORK / "mode.txt").write_text("")
|
|
raise SystemExit(1)
|
|
|
|
|
|
DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{")
|
|
SELF_DIR = ROOT / "conformance" / "tests" / "l0-static" / "placeholder-guard-coupling"
|
|
SKIP_DIRS = {".git", ".claude", "_artifacts", "node_modules", ".venv"}
|
|
|
|
# ------------------------------------------------------------------ discover the guard FILE
|
|
# Hardcoding a path is what let a file that never landed disable this whole check.
|
|
if OVERRIDE:
|
|
cand = [ROOT / OVERRIDE]
|
|
if not cand[0].is_file():
|
|
bail(f"PLACEHOLDER_GUARD_FILE={OVERRIDE} does not exist")
|
|
else:
|
|
cand = []
|
|
for p in sorted(ROOT.rglob("*.sh")):
|
|
if SKIP_DIRS & set(p.parts) or SELF_DIR in p.parents:
|
|
continue
|
|
try:
|
|
text = p.read_text()
|
|
except (OSError, UnicodeDecodeError): # pragma: no cover
|
|
continue
|
|
if any(DEF_RE.match(line) for line in text.splitlines()):
|
|
cand.append(p)
|
|
|
|
check(len(cand) <= 1,
|
|
"more than one file defines a *_is_placeholder predicate ("
|
|
+ ", ".join(str(p.relative_to(ROOT)) for p in cand)
|
|
+ ") — two files owning the placeholder vocabulary is the 2026-09-05 divergence with a "
|
|
"second file's name on it, and this check cannot say which one a standup run would "
|
|
"source. Keep exactly one, and source it from the other.")
|
|
|
|
GUARD_REL = str(cand[0].relative_to(ROOT)) if len(cand) == 1 else ""
|
|
(WORK / "mode.txt").write_text(GUARD_REL + "\n")
|
|
|
|
# --------------------------------------------------------- HALF B setup: extract the guard
|
|
guard_src: list[str] = []
|
|
guard_words: dict[str, list[str]] = {}
|
|
funcs: dict[str, list[str]] = {}
|
|
CALL_RE = re.compile(r"\b([a-z0-9_]+_is_placeholder)\s+\"([^\"]*)\"")
|
|
|
|
if GUARD_REL:
|
|
TEXT = (ROOT / GUARD_REL).read_text()
|
|
LINES = TEXT.splitlines()
|
|
|
|
defs: dict[str, list[int]] = {}
|
|
for i, line in enumerate(LINES):
|
|
m = DEF_RE.match(line)
|
|
if m:
|
|
defs.setdefault(m.group(1), []).append(i)
|
|
|
|
ARM_RE = re.compile(r"^(?P<pats>[^()]*?)\)\s*return\s+[01]\s*;;$")
|
|
ALLOWED = (
|
|
re.compile(r"^$"),
|
|
re.compile(r"^#"),
|
|
re.compile(r"^case\s+\"\$\{?[0-9A-Za-z_]+\}?\"\s+in$"),
|
|
re.compile(r"^esac$"),
|
|
re.compile(r"^return\s+[01]$"),
|
|
ARM_RE,
|
|
)
|
|
|
|
for name, at in sorted(defs.items()):
|
|
check(len(at) == 1,
|
|
f"{GUARD_REL} defines {name}() {len(at)} times (lines "
|
|
f"{', '.join(str(a + 1) for a in at)}) — which one runs depends on file order, and "
|
|
"this check can no longer say which one it extracted. Keep exactly one definition.")
|
|
if len(at) != 1:
|
|
continue
|
|
start = at[0]
|
|
end = next((j for j in range(start + 1, len(LINES)) if LINES[j] == "}"), None)
|
|
if end is None:
|
|
bail(f"{name}() at {GUARD_REL}:{start + 1} has no closing brace at column 0 — the "
|
|
"extraction boundary this check relies on is gone. Restore the conventional "
|
|
"layout, or move the predicate into a sourceable library.")
|
|
body = LINES[start:end + 1]
|
|
block = "\n".join(body)
|
|
check(block in TEXT,
|
|
f"extracted {name}() is not a verbatim substring of {GUARD_REL} — extraction bug; "
|
|
"the harness would be testing something the script does not contain")
|
|
|
|
words: list[str] = []
|
|
for raw in body[1:-1]:
|
|
s = raw.strip()
|
|
if not any(rx.match(s) for rx in ALLOWED) or "$(" in s or "`" in s:
|
|
bail(f"{name}() at {GUARD_REL}:{start + 1} is no longer a SELF-CONTAINED "
|
|
f"case/return predicate (offending line: {s!r}). This check exercises the "
|
|
"REAL function by sourcing its own lines; a body that calls out to anything "
|
|
"else cannot be extracted faithfully, and silently testing a COPY of the "
|
|
"logic is exactly the failure mode this check exists to prevent. REMEDY: "
|
|
"move the predicate into a tiny library that the script sources (e.g. "
|
|
"bootstrap/lib/placeholder.sh) — discovery will find it — do NOT "
|
|
"reimplement the matching here.")
|
|
m = ARM_RE.match(s)
|
|
if not m:
|
|
continue
|
|
for pat in m.group("pats").split("|"):
|
|
pat = pat.strip()
|
|
if pat in ('""', "''", ""):
|
|
continue # the empty-value arm: covered by the canary
|
|
words.append(pat.strip("'\"").strip("*"))
|
|
check(len(words) >= 1,
|
|
f"{name}() has no non-empty placeholder pattern left — every arm that recognises a "
|
|
"loud placeholder string has been removed, so the guard now only refuses the empty "
|
|
"value and any 'TBD'-style endpoint would sail straight through it")
|
|
guard_words[name] = words
|
|
guard_src.append(block)
|
|
|
|
if not guard_words:
|
|
bail(f"{GUARD_REL} was discovered as the guard file but yielded no usable predicate — "
|
|
"the definitions are there but none could be extracted. Fix the extraction in the "
|
|
"SAME change; do not let this degrade to a pass.")
|
|
|
|
# ------------------------------------------- no divergent second copy of the logic
|
|
# The guard's own vocabulary in EXECUTABLE text outside the predicate means a second,
|
|
# independently-drifting implementation of the same test (which is how three guards came
|
|
# to disagree with one value in the first place).
|
|
guard_line_nums = {j for name, at in defs.items() if len(at) == 1
|
|
for j in range(at[0], next(k for k in range(at[0] + 1, len(LINES))
|
|
if LINES[k] == "}") + 1)}
|
|
divergent: list[str] = []
|
|
_words = sorted({w.upper() for ws in guard_words.values() for w in ws if w})
|
|
for j, line in enumerate(LINES):
|
|
if j in guard_line_nums or line.strip().startswith("#"):
|
|
continue
|
|
code = line.split("#", 1)[0]
|
|
hit = [w for w in _words if w in code.upper()]
|
|
if hit:
|
|
divergent.append(f"{GUARD_REL}:{j + 1} ({', '.join(hit)}): {line.strip()!r}")
|
|
check(not divergent,
|
|
"the guard's own placeholder vocabulary appears in EXECUTABLE text outside the "
|
|
"predicate — a SECOND implementation of the same test, which is precisely how a value "
|
|
"change disarmed three guards at once. Call the predicate instead. Sites: "
|
|
+ " | ".join(divergent))
|
|
|
|
# function bodies, for the consumer trace
|
|
FUNC_RE = re.compile(r"^([a-z0-9_]+)\(\)\s*\{")
|
|
i = 0
|
|
while i < len(LINES):
|
|
m = FUNC_RE.match(LINES[i])
|
|
if m:
|
|
end = next((j for j in range(i + 1, len(LINES)) if LINES[j] == "}"), len(LINES) - 1)
|
|
funcs[m.group(1)] = LINES[i + 1:end]
|
|
i = end + 1
|
|
continue
|
|
i += 1
|
|
check(len(funcs) > 5, f"parsed only {len(funcs)} function(s) out of {GUARD_REL} — the "
|
|
"consumer assertion (B2) would be vacuous")
|
|
|
|
(WORK / "guards.sh").write_text(("\n".join(guard_src) + "\n") if guard_src else "")
|
|
|
|
# ------------------------------------------------------------- cluster-vars, as authored
|
|
CV = ROOT / "clusters" / "base" / "cluster-vars.yaml"
|
|
base: dict[str, str] = {}
|
|
for d in yaml.safe_load_all(CV.read_text()):
|
|
if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
|
|
and (d.get("metadata") or {}).get("name") == "cluster-vars":
|
|
base = {str(k): "" if v is None else str(v) for k, v in (d.get("data") or {}).items()}
|
|
check(len(base) > 5, f"clusters/base/cluster-vars.yaml yielded {len(base)} knob(s) — the "
|
|
"ConfigMap parse failed, so every value assertion below would be vacuous")
|
|
|
|
# Effective values per overlay, and WHICH knobs that overlay states explicitly. The second
|
|
# half is what lets emptiness be judged as authored-or-forgotten instead of guessed.
|
|
overlays: dict[str, dict[str, str]] = {}
|
|
stated: dict[str, set[str]] = {}
|
|
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
|
env = overlay.parent.name
|
|
if env == "base":
|
|
continue
|
|
eff = dict(base)
|
|
explicit: set[str] = set()
|
|
try:
|
|
k = yaml.safe_load(overlay.read_text()) or {}
|
|
except Exception as exc: # noqa: BLE001
|
|
check(False, f"clusters/{env}/kustomization.yaml does not parse as YAML ({exc}) — this "
|
|
"check cannot compute that overlay's effective values")
|
|
continue
|
|
for p in (k.get("patches") or []):
|
|
tgt = p.get("target") or {}
|
|
if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
|
|
continue
|
|
try:
|
|
ops = yaml.safe_load(p.get("patch") or "") or []
|
|
except Exception: # noqa: BLE001
|
|
continue
|
|
for op in ops if isinstance(ops, list) else []:
|
|
if not isinstance(op, dict):
|
|
continue
|
|
m = re.match(r"^/data/([A-Za-z0-9_]+)$", str(op.get("path", "")))
|
|
if m and str(op.get("op", "")) in ("replace", "add"):
|
|
eff[m.group(1)] = "" if op.get("value") is None else str(op["value"])
|
|
explicit.add(m.group(1))
|
|
overlays[env] = eff
|
|
stated[env] = explicit
|
|
check(len(overlays) >= 1,
|
|
"no clusters/<env>/kustomization.yaml overlay was parsed — assertion (A1) would be vacuous")
|
|
|
|
# ------------------------------------------ which knobs is a guard responsible for? (HALF B)
|
|
# A one-hop taint trace from `x="$(cvar KEY)"` to a `*_is_placeholder "${x}"` call. Both
|
|
# halves are read out of the file; nothing is named here.
|
|
CVAR_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\(\s*cvar\s+([A-Z0-9_]+)\s*\)\"")
|
|
ALIAS_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\{([A-Za-z_][A-Za-z0-9_]*)[:}]")
|
|
|
|
guarded_keys: set[str] = set()
|
|
consumers: list[tuple[str, str, bool]] = [] # (func, key, guarded-in-this-func)
|
|
for fname, body in funcs.items():
|
|
text = "\n".join(body)
|
|
keys_of_var: dict[str, set[str]] = {}
|
|
for m in CVAR_RE.finditer(text):
|
|
keys_of_var.setdefault(m.group(1), set()).add(m.group(2))
|
|
# one hop of aliasing: ep="${live}" carries live's taint
|
|
for _ in range(3):
|
|
for m in ALIAS_RE.finditer(text):
|
|
src = keys_of_var.get(m.group(2))
|
|
if src:
|
|
keys_of_var.setdefault(m.group(1), set()).update(src)
|
|
called_on = {m.group(2) for m in CALL_RE.finditer(text)}
|
|
called_vars = {re.sub(r"^\$\{?|\}?$", "", a) for a in called_on}
|
|
guarded_here: dict[str, bool] = {}
|
|
for var, keys in keys_of_var.items():
|
|
for key in keys:
|
|
guarded_here[key] = guarded_here.get(key, False) or (var in called_vars)
|
|
for key, g in guarded_here.items():
|
|
consumers.append((fname, key, g))
|
|
if g:
|
|
guarded_keys.add(key)
|
|
|
|
if GUARD_REL and not guarded_keys:
|
|
bail(f"no cluster-vars knob in {GUARD_REL} is passed to a *_is_placeholder predicate. "
|
|
"Either the guards were removed, or the way the script reads cluster-vars changed and "
|
|
"this check's derivation must be updated in the SAME change — it must never silently "
|
|
"verify an empty set.")
|
|
|
|
# (B2) CONSUMERS: every reader of a guarded knob must apply the guard to it.
|
|
for fname, key, guarded in sorted(consumers):
|
|
if key not in guarded_keys:
|
|
continue # nothing claims responsibility for this knob at all
|
|
check(guarded,
|
|
f"{GUARD_REL}: {fname}() reads cluster-vars knob {key} but never passes it to a "
|
|
"*_is_placeholder guard, while other functions do. On 2026-09-05 all three consumers "
|
|
"of S3_BACKUP_ENDPOINT were disarmed at once; a consumer that reads the knob and skips "
|
|
"the guard is that failure in one function.")
|
|
|
|
# ------------------------------------------------------------------ independent detector
|
|
# Deliberately NOT derived from the guard alone: SEED is universal loud-placeholder
|
|
# vocabulary, and the guard's own words are added so that a guard whose ANCHORING is wrong
|
|
# still has its vocabulary tested against real values.
|
|
SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", "REPLACE-ME", "REPLACEME",
|
|
"CHANGE-ME", "CHANGEME", "FILL-ME", "FILLME", "SET-ME", "XXXX"]
|
|
all_words = sorted({w.upper() for ws in guard_words.values() for w in ws if w})
|
|
VOCAB = sorted({w.upper() for w in SEED} | set(all_words))
|
|
ANGLE = re.compile(r"<[^>\s]+>")
|
|
|
|
|
|
def loud(value: str) -> bool:
|
|
"""A positive statement that the value was never decided. The EMPTY STRING IS NOT LOUD —
|
|
see the header: cluster-vars authors "" deliberately, and treating it as a placeholder is
|
|
the false positive that made the original check red against its own input."""
|
|
up = value.upper()
|
|
return any(w in up for w in VOCAB) or bool(ANGLE.search(value))
|
|
|
|
|
|
def guard_shaped(value: str) -> bool:
|
|
"""What a GUARD must refuse: loud, or empty. Empty counts here only because a `""`
|
|
case-arm is the guard author's declaration that empty is not a usable target."""
|
|
return value.strip() == "" or loud(value)
|
|
|
|
|
|
# ------------------------------------------------------------------- (A4) DETECTOR CANARIES
|
|
# The detector is now the thing doing the refusing in HALF A, so it gets the same
|
|
# no-op interrogation the guard does. These strings are the real ones from the incident.
|
|
check(loud("https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com"),
|
|
"the detector does not recognise the exact 2026-09-05 placeholder "
|
|
"'https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com' — it is prefix-anchored, which is "
|
|
"the defect this check exists to catch, now in the check itself")
|
|
check(loud("TBD-PROD-BULK-NFS-SERVER"),
|
|
"the detector does not recognise the repo's own TBD-PROD-* sentinel convention "
|
|
"(clusters/README.md) — assertion (A1) would pass over the very values it is about")
|
|
check(loud("<your-account-id>"),
|
|
"the detector does not recognise an <angle-bracket> template as a placeholder")
|
|
check(not loud("https://0123456789abcdef0123456789abcd.r2.cloudflarestorage.com"),
|
|
"the detector calls a REAL Cloudflare R2 endpoint a placeholder — it would false-block "
|
|
"every prod backup target")
|
|
check(not loud("http://192.168.1.9:9000"),
|
|
"the detector calls a REAL MinIO endpoint a placeholder — it would false-block dev")
|
|
check(not loud(""),
|
|
"the detector calls the EMPTY STRING loud. cluster-vars authors \"\" deliberately for "
|
|
"suffix/prefix knobs and documents it as load-bearing; emptiness is held to the "
|
|
"explicitness rule (A2) instead. This exact false positive made the original check FAIL "
|
|
"2 of 34 assertions against the real guard.")
|
|
|
|
# --------------------------------- (A1) no overlay ships a LOUD value nothing refuses
|
|
a1_rows = 0
|
|
for env, eff in sorted(overlays.items()):
|
|
for key, value in sorted(eff.items()):
|
|
if key in guarded_keys:
|
|
continue # a guard owns this knob: HALF B (B1) asserts it refuses it
|
|
a1_rows += 1
|
|
check(not loud(value),
|
|
f"clusters/{env}: knob {key} resolves to the placeholder-shaped value {value!r}, "
|
|
"and nothing refuses it"
|
|
+ (f" — no guard in {GUARD_REL} covers this knob." if GUARD_REL
|
|
else " — and no placeholder guard exists in the tree at all.")
|
|
+ " Either give it a real value before it reaches an overlay, or guard it the way "
|
|
"the S3 endpoint is guarded. An overlay that ships a placeholder no guard "
|
|
"recognises is the 2026-09-05 failure with a different knob's name on it: it "
|
|
"applies clean, reports Ready, and the thing it configures does not exist.")
|
|
|
|
# ------------------------- (A2) an empty effective value must be STATED, never inherited
|
|
a2_rows = 0
|
|
for env, eff in sorted(overlays.items()):
|
|
for key, value in sorted(eff.items()):
|
|
if value.strip() != "":
|
|
continue
|
|
a2_rows += 1
|
|
check(key in stated[env],
|
|
f"clusters/{env}: knob {key} resolves to the EMPTY STRING by INHERITANCE from "
|
|
"clusters/base — the overlay never states it. An empty value can be correct (dev's "
|
|
"BACKUP_CREDENTIALS_SUFFIX and VELERO_BACKUP_PREFIX are authored empty on purpose), "
|
|
"but only when the overlay SAYS SO: cluster-vars' own convention is that both "
|
|
"overlays state such a knob explicitly so neither inherits it silently. An empty "
|
|
"value nobody stated is indistinguishable from a knob somebody forgot.")
|
|
|
|
# ---------------- (A3) a LOUD base sentinel must be overridden EXPLICITLY by every overlay
|
|
a3_rows = 0
|
|
for key, value in sorted(base.items()):
|
|
if not loud(value):
|
|
continue
|
|
for env in sorted(overlays):
|
|
a3_rows += 1
|
|
check(key in stated[env],
|
|
f"clusters/base/cluster-vars.yaml authors the LOUD sentinel {key}={value!r}, and "
|
|
f"clusters/{env} does not state that knob — so that overlay INHERITS the sentinel. "
|
|
"The TBD-PROD-* convention only works if a sentinel cannot be inherited by "
|
|
"omission; an omitted op is how prod silently ships a value nobody decided.")
|
|
|
|
# --------------------------------------- HALF B corpus: authored values + anchoring + canaries
|
|
rows: list[tuple[str, str, str, str]] = [] # guard, expect, origin, value
|
|
|
|
# (B1) VALUES: base default and every overlay's value, for every guarded knob.
|
|
for key in sorted(guarded_keys):
|
|
seen: list[tuple[str, str]] = []
|
|
if key in base:
|
|
seen.append((f"clusters/base/cluster-vars.yaml:{key}", base[key]))
|
|
for env, eff in sorted(overlays.items()):
|
|
seen.append((f"clusters/{env} effective:{key}", eff.get(key, "")))
|
|
check(bool(seen), f"guarded knob {key} is not defined in cluster-vars at all — the guard "
|
|
"protects a knob nothing authors")
|
|
for origin, value in seen:
|
|
expect = "placeholder" if guard_shaped(value) else "usable"
|
|
for name in sorted(guard_words):
|
|
rows.append((name, expect, origin, value))
|
|
|
|
# (B3) ANCHORING: the guard's own vocabulary must match ANYWHERE, not just as a prefix,
|
|
# and a realistic endpoint must still be accepted.
|
|
for name, words in sorted(guard_words.items()):
|
|
for w in sorted(set(words)):
|
|
rows.append((name, "placeholder",
|
|
f"anchoring:{name}:mid-string({w})",
|
|
f"https://account-{w}-id.r2.cloudflarestorage.com"))
|
|
rows.append((name, "placeholder", f"anchoring:{name}:tail({w})",
|
|
f"https://r2-endpoint.example.invalid/{w}"))
|
|
rows.append((name, "placeholder", f"canary:{name}:empty", ""))
|
|
rows.append((name, "usable", f"canary:{name}:real-r2",
|
|
"https://0123456789abcdef0123456789abcd.r2.cloudflarestorage.com"))
|
|
rows.append((name, "usable", f"canary:{name}:real-minio", "http://192.168.1.9:9000"))
|
|
|
|
# Every guarded knob must reach the oracle once per authored source (base + each overlay),
|
|
# so a knob silently dropping out of the corpus is a failure rather than a quiet pass.
|
|
if GUARD_REL:
|
|
authored = [r for r in rows if not r[2].startswith(("anchoring:", "canary:"))]
|
|
expected_rows = len(guarded_keys) * (1 + len(overlays)) * len(guard_words)
|
|
check(len(authored) == expected_rows,
|
|
f"{len(authored)} authored value(s) reached the guard, expected {expected_rows} "
|
|
f"({len(guarded_keys)} guarded knob(s) x base+{len(overlays)} overlay(s) x "
|
|
f"{len(guard_words)} predicate(s)) — a guarded knob is missing from cluster-vars or "
|
|
"from an overlay's effective values, and the coupling is only partly exercised")
|
|
|
|
# --------------------------------------------------- (A5) the check is load-bearing
|
|
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
|
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
|
check("--layer l0" in L0,
|
|
"ci/l0.sh does not run the L0 conformance layer — this check would never execute in the gate")
|
|
check(bool(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)) and bool(re.search(r"fail\).*BLOCK=1", GATE)),
|
|
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
|
|
|
# ------------------------------------------------------- non-vacuity floor for HALF A
|
|
# HALF A is the half that runs unconditionally, so it carries its own floor: if the
|
|
# authored-value sweep shrinks to nothing, a pass here means nothing.
|
|
check(a1_rows >= 20,
|
|
f"only {a1_rows} authored overlay value(s) were audited by (A1) — the sweep over "
|
|
"clusters/<env> effective cluster-vars has collapsed, and a pass would be meaningless. "
|
|
"Expected roughly (knobs x overlays); check the ConfigMap and overlay parse above.")
|
|
|
|
# --------------------------------------------------------------------------- handoff
|
|
with (WORK / "corpus.tsv").open("w") as fh:
|
|
for name, expect, origin, value in rows:
|
|
fh.write(f"{name}\t{expect}\t{origin}\t{value}\n")
|
|
(WORK / "findings.txt").write_text("".join(f"{f}\n" for f in FAILURES))
|
|
(WORK / "counts.txt").write_text(f"{CHECKS}\n")
|
|
|
|
if GUARD_REL:
|
|
print(f"guard file: {GUARD_REL} predicates: {', '.join(sorted(guard_words))}")
|
|
print(f"guarded knobs: {', '.join(sorted(guarded_keys))} "
|
|
f"consumers traced: {len(consumers)}")
|
|
else:
|
|
print("guard file: NONE FOUND in the tree — HALF B (guard/value coupling) IS NOT PROVEN "
|
|
"ON THIS TREE. No *_is_placeholder predicate exists to extract: bootstrap/standup.sh "
|
|
"has never been on main (no deletion commit exists; it lives on unmerged prod-standup "
|
|
"branches). HALF A below runs against the real authored values and is what currently "
|
|
"prevents the 2026-09-05 outcome. When a guard lands ANYWHERE in the tree, discovery "
|
|
"picks it up and HALF B activates with no edit to this check.")
|
|
print(f"detector vocabulary: {', '.join(VOCAB)}")
|
|
print(f"HALF A: overlays {', '.join(sorted(overlays))}; {len(base)} base knob(s); "
|
|
f"(A1) {a1_rows} authored overlay value(s) screened; (A2) {a2_rows} empty value(s) "
|
|
f"checked for explicit statement; (A3) {a3_rows} base-sentinel inheritance check(s)")
|
|
for name, expect, origin, value in rows:
|
|
if not origin.startswith(("anchoring:", "canary:")):
|
|
print(f" authored {origin} = {value!r} -> expect {expect}")
|
|
PY
|
|
|
|
GUARD_REL="$(tr -d '\n' < "${WORK}/mode.txt")"
|
|
|
|
# ---------------------------------------------------------------------------------------
|
|
# PHASE 2 (bash): source the REAL predicate lines and run the corpus through them.
|
|
# Empty when no guard exists — HALF A has already run and carries its own floor.
|
|
# ---------------------------------------------------------------------------------------
|
|
# shellcheck disable=SC1090,SC1091
|
|
source "${WORK}/guards.sh"
|
|
|
|
oracle_failures=0
|
|
oracle_rows=0
|
|
while IFS=$'\t' read -r guard expect origin value; do
|
|
[ -n "${guard}" ] || continue
|
|
oracle_rows=$((oracle_rows + 1))
|
|
if "${guard}" "${value}"; then verdict=placeholder; else verdict=usable; fi
|
|
if [ "${verdict}" != "${expect}" ]; then
|
|
oracle_failures=$((oracle_failures + 1))
|
|
if [ "${expect}" = placeholder ]; then
|
|
printf '%s\n' "${guard}() in ${GUARD_REL} ACCEPTS '${value}' (from ${origin}) as a usable value, but it is placeholder-shaped. This is the 2026-09-05 failure exactly: the guard is asked about a value it does not recognise, returns 'fine', and the tier it protects is enabled against something that does not exist. Fix the predicate (match the idiom anywhere, not as a prefix) or stop authoring that idiom." >> "${WORK}/findings.txt"
|
|
else
|
|
printf '%s\n' "${guard}() in ${GUARD_REL} REFUSES '${value}' (from ${origin}), which is a usable value. An over-matching guard leaves the tier it protects suspended forever and reports it as an operator decision — a silent failure in the other direction." >> "${WORK}/findings.txt"
|
|
fi
|
|
fi
|
|
done < "${WORK}/corpus.tsv"
|
|
|
|
# ------------------------------------------------------------------------------ verdict
|
|
py_checks="$(cat "${WORK}/counts.txt")"
|
|
total=$((py_checks + oracle_rows))
|
|
findings="$(cat "${WORK}/findings.txt")"
|
|
|
|
# NO-OP GUARDS: this repo has been bitten by a check that passed when perturbed because it
|
|
# asked a question that was always satisfiable. The floors below are the minimum evidence
|
|
# for the verdict to mean anything, not a style rule. The oracle floor applies only when a
|
|
# guard exists — when none does, HALF A's own floor (a1_rows >= 20, asserted in phase 1) is
|
|
# what stops this from degenerating, and the absence is reported on stdout.
|
|
if [ -n "${GUARD_REL}" ] && [ "${oracle_rows}" -lt 6 ]; then
|
|
findings="${findings}only ${oracle_rows} value(s) were put to the real guard in ${GUARD_REL} — the coupling is not actually being exercised
|
|
"
|
|
fi
|
|
if [ "${total}" -lt 20 ]; then
|
|
findings="${findings}only ${total} assertions ran — the check degenerated to a near-no-op and its pass would be meaningless
|
|
"
|
|
fi
|
|
|
|
if [ -n "${findings}" ]; then
|
|
echo "CHECK FAIL (placeholder-guard-coupling): $(printf '%s' "${findings}" | grep -c . ) of ${total} assertions failed" >&2
|
|
printf '%s' "${findings}" | sed 's/^/ - /' >&2
|
|
exit 1
|
|
fi
|
|
|
|
if [ -n "${GUARD_REL}" ]; then
|
|
echo "check OK: placeholder-guard-coupling — ${total} assertions. HALF A: no environment overlay ships a placeholder-shaped effective cluster-vars value, every empty value is stated explicitly rather than inherited, and the detector canaries hold (including the exact 2026-09-05 string, matched mid-string). HALF B: ${oracle_rows} value(s) run through the REAL extracted predicate(s) from ${GUARD_REL} (no reimplementation), covering every authored value of every guarded knob in base and each overlay, the guard's vocabulary matched mid-string, and the empty/real-endpoint canaries. Static slice only: refusal, not reachability — see the header."
|
|
else
|
|
echo "check OK: placeholder-guard-coupling — ${total} assertions. HALF A ONLY: no environment overlay ships a placeholder-shaped effective cluster-vars value, every empty value is stated explicitly rather than inherited, no LOUD base sentinel is inherited by omission, and the detector canaries hold (including the exact 2026-09-05 string 'https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com', matched mid-string). HALF B (guard/value coupling) IS NOT PROVEN: no *_is_placeholder predicate exists anywhere in the tree to extract — see the stdout note above and the header. This is a REPORTED gap, not a skip: what a guard would be asked about is proven not to be a placeholder in the first place."
|
|
fi
|