#!/usr/bin/env bash # ============================================================================= # L0 static check (offline, no cluster) — placeholder-guard-coupling # Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)." # — spec docs/02-environments-and-provisioning.md) # R-STATE-03 ("Scheduled off-cluster backups + restore (DR)." # — spec docs/04-storage-and-state.md) # # WHY THIS EXISTS — A GUARD KEYED ON A VALUE THAT LIVES IN ANOTHER FILE # -------------------------------------------------------------------- # `bootstrap/standup.sh` refuses to resume tiers whose configuration is still a # placeholder. Three of those guards keyed on the SHAPE of `S3_BACKUP_ENDPOINT`, which is # authored in `clusters/base/cluster-vars.yaml`. On 2026-09-05 the placeholder changed # from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` # (the MinIO -> Cloudflare R2 move, commit 32d6766). The guards matched the glob `TBD-*` — # a PREFIX match — and `fnmatch('https://TBD-R2-…','TBD-*')` is FALSE. All three guards # silently stopped matching. `standup.sh resume` would then have resumed `velero` on prod # against a hostname that does not exist: the HelmRelease installs clean, the Kustomization # reports Ready, and NO BACKUPS EXIST. It was found by accident. # # WHY THIS CHECK WAS REWRITTEN (2026-09-13) — IT WAS SKIPPING FOREVER # ------------------------------------------------------------------ # The original version keyed its ENTIRE existence on `bootstrap/standup.sh` and exited 3 # (SKIP) when that file was absent. Established by measurement, not assumption: # # * `bootstrap/standup.sh` HAS NEVER EXISTED ON `main`. It was not deleted — there is no # deletion commit for it anywhere (`git log --all --diff-filter=D -- bootstrap/standup.sh` # is empty). It lives on unmerged prod-standup branches (`work/prod-standup-campaign`, # `prod/*`, `feat/prod-hardening`) and never landed. `git ls-tree main bootstrap/` lists # only README.md, flux/, gitea/, sops/. docs/BULK-NFS-STORAGE.md §7 already recorded # this, including the consequence: "nothing in the gate currently refuses a # placeholder-shaped overlay value". # * So this check reported SKIP on every gate run since it landed, and SKIPPED IS NOT # GREEN in this repo. An incident-derived test that can only ever skip is the quiet # version of the failure it was written to prevent. # * Worse, the skip was HIDING A RED. Run against the real `standup.sh` from # `work/prod-standup-campaign`, the original check FAILED 2 of 34 assertions — both # false positives from treating the EMPTY STRING as placeholder-shaped for knobs whose # emptiness is deliberately authored (`BACKUP_CREDENTIALS_SUFFIX` and # `VELERO_BACKUP_PREFIX`, both `""` on dev). cluster-vars documents that emptiness as # load-bearing ("THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING … an # EMPTY VALUE IS SAFE UNDER StrictPostBuildSubstitutions"). So the check would have # false-blocked the moment its own input landed. That is fixed below. # # THE HAZARD IS NOT GONE, so this coverage is NOT obsolete and was NOT deleted. The # invariant has two independent halves, and only one of them needs `standup.sh`: # # HALF A — THE AUTHORED VALUES (provable on `main` today, needs NO guard). # No environment overlay may ship a placeholder-shaped effective cluster-vars value. # This is the OUTCOME the 2026-09-05 guard existed to prevent, asserted at the source # instead of at the guard: if prod's `S3_BACKUP_ENDPOINT` is never a placeholder, velero # cannot be pointed at a hostname that does not exist, whether or not a guard catches # it. Its entire input surface is `clusters/`, all of which is on `main`. # HALF B — THE COUPLING (activates automatically when a guard lands). # The real predicate, extracted from the real file and executed, agrees with an # INDEPENDENT detector about every authored value; every consumer of a guarded knob # applies the guard; the guard matches its vocabulary ANYWHERE, not as a prefix. # # THIS CHECK NEVER EXITS 3. Half A always runs, against real files, with an assertion # floor. Half B runs when a guard exists and is reported as NOT PROVEN when one does not — # stated loudly in the output rather than swallowed as a skip. # # THE GUARD IS DISCOVERED, NOT HARDCODED (this is the "re-point" made automatic) # ----------------------------------------------------------------------------- # The original hardcoded `bootstrap/standup.sh`, which is exactly why a file that never # landed could disable the whole check. Instead, every tracked `*.sh` in the tree is # scanned for a `_is_placeholder()` definition: # * exactly one such FILE -> that is the guard; Half B runs against it, wherever it is. # A guard that lands at `bootstrap/standup.sh`, or moves to `bootstrap/lib/placeholder.sh`, # is picked up with no edit to this check. # * none -> Half A only, with the gap named in the output. # * more than one file -> FAIL. Two files defining the placeholder vocabulary is the # 2026-09-05 divergence with a second file's name on it; this check cannot say which # one a standup run would source. # `PLACEHOLDER_GUARD_FILE` overrides discovery (repo-relative) for negative fixtures. # NOTE: discovery is shell-only, because the guard family is shell. If the refusal ever # moves to a non-shell mechanism (a Kyverno policy, a CEL admission rule), discovery finds # nothing and says so — re-point this check deliberately in that change. # # HOW THE REAL GUARD IS EXERCISED (and why not a copy) # --------------------------------------------------- # `standup.sh` is NOT sourceable: at top level it parses argv, hard-exits when # --kubeconfig/--expect-server/ are absent, `cd`s to the repo root, exports # KUBECONFIG, installs an EXIT trap and finally dispatches steps against a live cluster. # So this check EXTRACTS the predicate's own source lines from the file and sources ONLY # those, then calls the real function. That is the real bytes, not a reimplementation: # * every function whose name ends `_is_placeholder` is discovered, not named here; # * a duplicate definition of one is a failure (which of the two would run?); # * the extracted body must be a SELF-CONTAINED predicate — `case`/`return` and nothing # else, no command substitution, no call to another function. If the guard ever grows # a dependency, extraction stops being faithful and THIS CHECK FAILS LOUDLY, naming the # remedy (move the predicate into a small sourceable library) rather than quietly # falling back to testing a copy of the logic; # * a CANARY pins the extraction itself: the extracted function must refuse the empty # string and accept a realistic endpoint. # There is deliberately NO second implementation of the matching logic anywhere in this file. # # THE INDEPENDENT DETECTOR (and the empty-string distinction the old version got wrong) # ------------------------------------------------------------------------------------ # Asking "does the guard agree with the guard?" proves nothing, so placeholder-shaped is # decided by a detector written HERE, whose vocabulary is the UNION of two independent # sources: # * SEED — universal loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER, REPLACE-ME, # CHANGE-ME, FILL-ME, SET-ME, XXXX), plus `` templates. NOT derived from # any guard, so DELETING a guard arm cannot make a value stop being placeholder-shaped. # * the guard's OWN vocabulary (when a guard exists), read out of its case-arms with the # glob anchors STRIPPED. Stripping the anchors is what makes the comparison non-circular # in the dimension that actually broke: the detector matches a word ANYWHERE, # case-insensitively, and the guard must agree. A prefix-anchored guard cannot pass. # # LOUD vs EMPTY — the correction. A LOUD placeholder is a positive statement that a value # was never decided, and it is always a defect in an overlay. The EMPTY STRING is not: # cluster-vars authors `""` deliberately for suffix/prefix knobs, and Flux strict # substitution fails on an UNDEFINED variable, not on one defined as `""`. So emptiness is # NOT treated as a loud placeholder. It is held to a different, real rule instead (A2): an # empty effective value must be STATED EXPLICITLY by that overlay, never inherited in # silence — which is cluster-vars' own documented convention ("Both overlays state this # knob explicitly … so neither inherits it silently"). Emptiness still counts as # placeholder-shaped for a knob a GUARD covers, because a `""` case-arm in the guard is the # guard author's declaration that empty is not a usable target for that knob. # # GIVEN the repository as authored, # THEN HALF A (always): # (A1) VALUES: no environment overlay's EFFECTIVE cluster-vars value is LOUD # placeholder-shaped, for any knob that no guard refuses. (Base alone is exempt: # it is not a bootstrap target — `flux bootstrap --path=clusters/` — and its # loud sentinels exist to be overridden. A base placeholder an overlay forgets to # override IS caught, because it becomes that overlay's effective value.) # (A2) EXPLICITNESS: an empty effective value is stated explicitly by that overlay. # (A3) SENTINEL DISCIPLINE: if base carries a LOUD sentinel for a knob, every overlay # states that knob explicitly — the `TBD-PROD-*` convention only works if no # overlay can inherit the sentinel by omission. # (A4) DETECTOR CANARIES: the detector recognises the exact 2026-09-05 string # MID-STRING, accepts both real endpoints the repo actually ships, and does NOT # call the empty string loud. # (A5) LOAD-BEARING: ci/l0.sh runs the L0 conformance layer on every gate run and # ci/gate.sh sets the merge block when that stage fails. # HALF B (only when a guard file is discovered): # (B1) VALUES: for every knob a guard is responsible for, every authored value — the # base default AND each overlay's value — that the detector calls placeholder- # shaped is REFUSED by the real guard, and every value it calls usable is # ACCEPTED (a guard that refuses a real endpoint leaves velero suspended forever, # the same class of silent failure). # (B2) CONSUMERS: every function that reads a guarded knob (via `cvar`, following one # hop of variable aliasing) applies a placeholder guard to it. This is the "all # three guards died at once" regression stated as a property. # (B3) ANCHORING: for each word in the guard's own vocabulary, the guard refuses a # value carrying that word in the MIDDLE (not at the start) — the precise defect # of 2026-09-05 — while still accepting a realistic endpoint. # (B4) HARNESS: exactly one definition per predicate, one guard FILE in the tree, the # extracted body verbatim and self-contained, the canary holds, no divergent # second copy of the matching logic, and assertion floors keep a pass from # degenerating into a no-op. # # WHAT THIS DOES *NOT* PROVE (honest gaps) # - While no guard exists on `main`, HALF B IS NOT PROVEN AT ALL, and the check says so on # stdout every run. What is proven is the stronger-at-the-source half: the values a # guard would be asked about are not placeholders in the first place. This is a real # narrowing of coverage versus the intent, and it is reported, not hidden. # - Nothing about RUNTIME: not that the endpoint resolves, not that credentials work, not # that a backup exists. `standup.sh s3check` probes reachability from a pod (operator- # run, cluster-gated) and conformance/tests/l3-fault-injection/dr-restore proves restore. # - Half B does not execute the dispatch path, so it cannot prove the guard is REACHED in # a real run — only that, when reached, it refuses what it must. (B2) is a text-level # taint trace over one file, not an interpreter. # - It says nothing about guards keyed on shapes OTHER than placeholders (e.g. the # EXTERNALDNS_TXT_OWNER_ID == 'default' test), nor about placeholder-shaped values # outside cluster-vars (the Ansible inventory's BLOCKED/NOT-A-DEVICE sentinels are # asserted by the node_baseline role itself, at play time). # - A1 reads cluster-vars, not every manifest. A placeholder hardcoded in a manifest # rather than in a knob is out of scope here. # # SELECTION. `ci/touched-requirements.sh` keys `clusters/*` -> R-ENV-02 (Half A's whole # input surface) and `bootstrap/*` -> R-STATE-03 (where a guard lands), so every path that # can break this check attributes a requirement it declares. Deliberately requirement-scoped, # not whole-repo — see chainsaw-test.yaml. # ============================================================================= set -euo pipefail DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root PYTHON="${PYTHON:-python3}" GUARD_OVERRIDE="${PLACEHOLDER_GUARD_FILE:-}" cd "${ROOT}" # The authored half of the coupling is the one thing this check cannot do without. if [ ! -f "${ROOT}/clusters/base/cluster-vars.yaml" ]; then echo "CHECK FAIL (placeholder-guard-coupling): clusters/base/cluster-vars.yaml is absent — the authored values this check audits do not exist. That is not a SKIP: the knobs this invariant is about have been moved or deleted, and this check must be re-pointed in the SAME change." >&2 exit 1 fi WORK="$(mktemp -d)" trap 'rm -rf "${WORK}"' EXIT # --------------------------------------------------------------------------------------- # PHASE 1 (python): discover the guard, derive the corpus from the real files, and run # every assertion that does not need a shell. Writes: # ${WORK}/guards.sh the predicate source lines, verbatim, and nothing else (may be empty) # ${WORK}/corpus.tsv \t\t\t (may be empty) # ${WORK}/findings.txt one line per failed assertion # ${WORK}/counts.txt # ${WORK}/mode.txt the discovered guard's repo-relative path, or empty # --------------------------------------------------------------------------------------- "${PYTHON}" - "${ROOT}" "${WORK}" "${GUARD_OVERRIDE}" <<'PY' import re import sys from pathlib import Path try: import yaml except ImportError: # pragma: no cover print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", file=sys.stderr) raise SystemExit(1) ROOT, WORK, OVERRIDE = Path(sys.argv[1]), Path(sys.argv[2]), sys.argv[3] FAILURES: list[str] = [] CHECKS = 0 def check(cond, msg): global CHECKS CHECKS += 1 if not cond: FAILURES.append(msg) def bail(msg): """A failure that makes the rest of the check meaningless: report and stop.""" print(f"CHECK FAIL (placeholder-guard-coupling): {msg}", file=sys.stderr) (WORK / "findings.txt").write_text(msg + "\n") (WORK / "counts.txt").write_text("0\n") (WORK / "corpus.tsv").write_text("") (WORK / "guards.sh").write_text("") (WORK / "mode.txt").write_text("") raise SystemExit(1) DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{") SELF_DIR = ROOT / "conformance" / "tests" / "l0-static" / "placeholder-guard-coupling" SKIP_DIRS = {".git", ".claude", "_artifacts", "node_modules", ".venv"} # ------------------------------------------------------------------ discover the guard FILE # Hardcoding a path is what let a file that never landed disable this whole check. if OVERRIDE: cand = [ROOT / OVERRIDE] if not cand[0].is_file(): bail(f"PLACEHOLDER_GUARD_FILE={OVERRIDE} does not exist") else: cand = [] for p in sorted(ROOT.rglob("*.sh")): if SKIP_DIRS & set(p.parts) or SELF_DIR in p.parents: continue try: text = p.read_text() except (OSError, UnicodeDecodeError): # pragma: no cover continue if any(DEF_RE.match(line) for line in text.splitlines()): cand.append(p) check(len(cand) <= 1, "more than one file defines a *_is_placeholder predicate (" + ", ".join(str(p.relative_to(ROOT)) for p in cand) + ") — two files owning the placeholder vocabulary is the 2026-09-05 divergence with a " "second file's name on it, and this check cannot say which one a standup run would " "source. Keep exactly one, and source it from the other.") GUARD_REL = str(cand[0].relative_to(ROOT)) if len(cand) == 1 else "" (WORK / "mode.txt").write_text(GUARD_REL + "\n") # --------------------------------------------------------- HALF B setup: extract the guard guard_src: list[str] = [] guard_words: dict[str, list[str]] = {} funcs: dict[str, list[str]] = {} CALL_RE = re.compile(r"\b([a-z0-9_]+_is_placeholder)\s+\"([^\"]*)\"") if GUARD_REL: TEXT = (ROOT / GUARD_REL).read_text() LINES = TEXT.splitlines() defs: dict[str, list[int]] = {} for i, line in enumerate(LINES): m = DEF_RE.match(line) if m: defs.setdefault(m.group(1), []).append(i) ARM_RE = re.compile(r"^(?P[^()]*?)\)\s*return\s+[01]\s*;;$") ALLOWED = ( re.compile(r"^$"), re.compile(r"^#"), re.compile(r"^case\s+\"\$\{?[0-9A-Za-z_]+\}?\"\s+in$"), re.compile(r"^esac$"), re.compile(r"^return\s+[01]$"), ARM_RE, ) for name, at in sorted(defs.items()): check(len(at) == 1, f"{GUARD_REL} defines {name}() {len(at)} times (lines " f"{', '.join(str(a + 1) for a in at)}) — which one runs depends on file order, and " "this check can no longer say which one it extracted. Keep exactly one definition.") if len(at) != 1: continue start = at[0] end = next((j for j in range(start + 1, len(LINES)) if LINES[j] == "}"), None) if end is None: bail(f"{name}() at {GUARD_REL}:{start + 1} has no closing brace at column 0 — the " "extraction boundary this check relies on is gone. Restore the conventional " "layout, or move the predicate into a sourceable library.") body = LINES[start:end + 1] block = "\n".join(body) check(block in TEXT, f"extracted {name}() is not a verbatim substring of {GUARD_REL} — extraction bug; " "the harness would be testing something the script does not contain") words: list[str] = [] for raw in body[1:-1]: s = raw.strip() if not any(rx.match(s) for rx in ALLOWED) or "$(" in s or "`" in s: bail(f"{name}() at {GUARD_REL}:{start + 1} is no longer a SELF-CONTAINED " f"case/return predicate (offending line: {s!r}). This check exercises the " "REAL function by sourcing its own lines; a body that calls out to anything " "else cannot be extracted faithfully, and silently testing a COPY of the " "logic is exactly the failure mode this check exists to prevent. REMEDY: " "move the predicate into a tiny library that the script sources (e.g. " "bootstrap/lib/placeholder.sh) — discovery will find it — do NOT " "reimplement the matching here.") m = ARM_RE.match(s) if not m: continue for pat in m.group("pats").split("|"): pat = pat.strip() if pat in ('""', "''", ""): continue # the empty-value arm: covered by the canary words.append(pat.strip("'\"").strip("*")) check(len(words) >= 1, f"{name}() has no non-empty placeholder pattern left — every arm that recognises a " "loud placeholder string has been removed, so the guard now only refuses the empty " "value and any 'TBD'-style endpoint would sail straight through it") guard_words[name] = words guard_src.append(block) if not guard_words: bail(f"{GUARD_REL} was discovered as the guard file but yielded no usable predicate — " "the definitions are there but none could be extracted. Fix the extraction in the " "SAME change; do not let this degrade to a pass.") # ------------------------------------------- no divergent second copy of the logic # The guard's own vocabulary in EXECUTABLE text outside the predicate means a second, # independently-drifting implementation of the same test (which is how three guards came # to disagree with one value in the first place). guard_line_nums = {j for name, at in defs.items() if len(at) == 1 for j in range(at[0], next(k for k in range(at[0] + 1, len(LINES)) if LINES[k] == "}") + 1)} divergent: list[str] = [] _words = sorted({w.upper() for ws in guard_words.values() for w in ws if w}) for j, line in enumerate(LINES): if j in guard_line_nums or line.strip().startswith("#"): continue code = line.split("#", 1)[0] hit = [w for w in _words if w in code.upper()] if hit: divergent.append(f"{GUARD_REL}:{j + 1} ({', '.join(hit)}): {line.strip()!r}") check(not divergent, "the guard's own placeholder vocabulary appears in EXECUTABLE text outside the " "predicate — a SECOND implementation of the same test, which is precisely how a value " "change disarmed three guards at once. Call the predicate instead. Sites: " + " | ".join(divergent)) # function bodies, for the consumer trace FUNC_RE = re.compile(r"^([a-z0-9_]+)\(\)\s*\{") i = 0 while i < len(LINES): m = FUNC_RE.match(LINES[i]) if m: end = next((j for j in range(i + 1, len(LINES)) if LINES[j] == "}"), len(LINES) - 1) funcs[m.group(1)] = LINES[i + 1:end] i = end + 1 continue i += 1 check(len(funcs) > 5, f"parsed only {len(funcs)} function(s) out of {GUARD_REL} — the " "consumer assertion (B2) would be vacuous") (WORK / "guards.sh").write_text(("\n".join(guard_src) + "\n") if guard_src else "") # ------------------------------------------------------------- cluster-vars, as authored CV = ROOT / "clusters" / "base" / "cluster-vars.yaml" base: dict[str, str] = {} for d in yaml.safe_load_all(CV.read_text()): if isinstance(d, dict) and d.get("kind") == "ConfigMap" \ and (d.get("metadata") or {}).get("name") == "cluster-vars": base = {str(k): "" if v is None else str(v) for k, v in (d.get("data") or {}).items()} check(len(base) > 5, f"clusters/base/cluster-vars.yaml yielded {len(base)} knob(s) — the " "ConfigMap parse failed, so every value assertion below would be vacuous") # Effective values per overlay, and WHICH knobs that overlay states explicitly. The second # half is what lets emptiness be judged as authored-or-forgotten instead of guessed. overlays: dict[str, dict[str, str]] = {} stated: dict[str, set[str]] = {} for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")): env = overlay.parent.name if env == "base": continue eff = dict(base) explicit: set[str] = set() try: k = yaml.safe_load(overlay.read_text()) or {} except Exception as exc: # noqa: BLE001 check(False, f"clusters/{env}/kustomization.yaml does not parse as YAML ({exc}) — this " "check cannot compute that overlay's effective values") continue for p in (k.get("patches") or []): tgt = p.get("target") or {} if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars": continue try: ops = yaml.safe_load(p.get("patch") or "") or [] except Exception: # noqa: BLE001 continue for op in ops if isinstance(ops, list) else []: if not isinstance(op, dict): continue m = re.match(r"^/data/([A-Za-z0-9_]+)$", str(op.get("path", ""))) if m and str(op.get("op", "")) in ("replace", "add"): eff[m.group(1)] = "" if op.get("value") is None else str(op["value"]) explicit.add(m.group(1)) overlays[env] = eff stated[env] = explicit check(len(overlays) >= 1, "no clusters//kustomization.yaml overlay was parsed — assertion (A1) would be vacuous") # ------------------------------------------ which knobs is a guard responsible for? (HALF B) # A one-hop taint trace from `x="$(cvar KEY)"` to a `*_is_placeholder "${x}"` call. Both # halves are read out of the file; nothing is named here. CVAR_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\(\s*cvar\s+([A-Z0-9_]+)\s*\)\"") ALIAS_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\{([A-Za-z_][A-Za-z0-9_]*)[:}]") guarded_keys: set[str] = set() consumers: list[tuple[str, str, bool]] = [] # (func, key, guarded-in-this-func) for fname, body in funcs.items(): text = "\n".join(body) keys_of_var: dict[str, set[str]] = {} for m in CVAR_RE.finditer(text): keys_of_var.setdefault(m.group(1), set()).add(m.group(2)) # one hop of aliasing: ep="${live}" carries live's taint for _ in range(3): for m in ALIAS_RE.finditer(text): src = keys_of_var.get(m.group(2)) if src: keys_of_var.setdefault(m.group(1), set()).update(src) called_on = {m.group(2) for m in CALL_RE.finditer(text)} called_vars = {re.sub(r"^\$\{?|\}?$", "", a) for a in called_on} guarded_here: dict[str, bool] = {} for var, keys in keys_of_var.items(): for key in keys: guarded_here[key] = guarded_here.get(key, False) or (var in called_vars) for key, g in guarded_here.items(): consumers.append((fname, key, g)) if g: guarded_keys.add(key) if GUARD_REL and not guarded_keys: bail(f"no cluster-vars knob in {GUARD_REL} is passed to a *_is_placeholder predicate. " "Either the guards were removed, or the way the script reads cluster-vars changed and " "this check's derivation must be updated in the SAME change — it must never silently " "verify an empty set.") # (B2) CONSUMERS: every reader of a guarded knob must apply the guard to it. for fname, key, guarded in sorted(consumers): if key not in guarded_keys: continue # nothing claims responsibility for this knob at all check(guarded, f"{GUARD_REL}: {fname}() reads cluster-vars knob {key} but never passes it to a " "*_is_placeholder guard, while other functions do. On 2026-09-05 all three consumers " "of S3_BACKUP_ENDPOINT were disarmed at once; a consumer that reads the knob and skips " "the guard is that failure in one function.") # ------------------------------------------------------------------ independent detector # Deliberately NOT derived from the guard alone: SEED is universal loud-placeholder # vocabulary, and the guard's own words are added so that a guard whose ANCHORING is wrong # still has its vocabulary tested against real values. SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", "REPLACE-ME", "REPLACEME", "CHANGE-ME", "CHANGEME", "FILL-ME", "FILLME", "SET-ME", "XXXX"] all_words = sorted({w.upper() for ws in guard_words.values() for w in ws if w}) VOCAB = sorted({w.upper() for w in SEED} | set(all_words)) ANGLE = re.compile(r"<[^>\s]+>") def loud(value: str) -> bool: """A positive statement that the value was never decided. The EMPTY STRING IS NOT LOUD — see the header: cluster-vars authors "" deliberately, and treating it as a placeholder is the false positive that made the original check red against its own input.""" up = value.upper() return any(w in up for w in VOCAB) or bool(ANGLE.search(value)) def guard_shaped(value: str) -> bool: """What a GUARD must refuse: loud, or empty. Empty counts here only because a `""` case-arm is the guard author's declaration that empty is not a usable target.""" return value.strip() == "" or loud(value) # ------------------------------------------------------------------- (A4) DETECTOR CANARIES # The detector is now the thing doing the refusing in HALF A, so it gets the same # no-op interrogation the guard does. These strings are the real ones from the incident. check(loud("https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com"), "the detector does not recognise the exact 2026-09-05 placeholder " "'https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com' — it is prefix-anchored, which is " "the defect this check exists to catch, now in the check itself") check(loud("TBD-PROD-BULK-NFS-SERVER"), "the detector does not recognise the repo's own TBD-PROD-* sentinel convention " "(clusters/README.md) — assertion (A1) would pass over the very values it is about") check(loud(""), "the detector does not recognise an template as a placeholder") check(not loud("https://0123456789abcdef0123456789abcd.r2.cloudflarestorage.com"), "the detector calls a REAL Cloudflare R2 endpoint a placeholder — it would false-block " "every prod backup target") check(not loud("http://192.168.1.9:9000"), "the detector calls a REAL MinIO endpoint a placeholder — it would false-block dev") check(not loud(""), "the detector calls the EMPTY STRING loud. cluster-vars authors \"\" deliberately for " "suffix/prefix knobs and documents it as load-bearing; emptiness is held to the " "explicitness rule (A2) instead. This exact false positive made the original check FAIL " "2 of 34 assertions against the real guard.") # --------------------------------- (A1) no overlay ships a LOUD value nothing refuses a1_rows = 0 for env, eff in sorted(overlays.items()): for key, value in sorted(eff.items()): if key in guarded_keys: continue # a guard owns this knob: HALF B (B1) asserts it refuses it a1_rows += 1 check(not loud(value), f"clusters/{env}: knob {key} resolves to the placeholder-shaped value {value!r}, " "and nothing refuses it" + (f" — no guard in {GUARD_REL} covers this knob." if GUARD_REL else " — and no placeholder guard exists in the tree at all.") + " Either give it a real value before it reaches an overlay, or guard it the way " "the S3 endpoint is guarded. An overlay that ships a placeholder no guard " "recognises is the 2026-09-05 failure with a different knob's name on it: it " "applies clean, reports Ready, and the thing it configures does not exist.") # ------------------------- (A2) an empty effective value must be STATED, never inherited a2_rows = 0 for env, eff in sorted(overlays.items()): for key, value in sorted(eff.items()): if value.strip() != "": continue a2_rows += 1 check(key in stated[env], f"clusters/{env}: knob {key} resolves to the EMPTY STRING by INHERITANCE from " "clusters/base — the overlay never states it. An empty value can be correct (dev's " "BACKUP_CREDENTIALS_SUFFIX and VELERO_BACKUP_PREFIX are authored empty on purpose), " "but only when the overlay SAYS SO: cluster-vars' own convention is that both " "overlays state such a knob explicitly so neither inherits it silently. An empty " "value nobody stated is indistinguishable from a knob somebody forgot.") # ---------------- (A3) a LOUD base sentinel must be overridden EXPLICITLY by every overlay a3_rows = 0 for key, value in sorted(base.items()): if not loud(value): continue for env in sorted(overlays): a3_rows += 1 check(key in stated[env], f"clusters/base/cluster-vars.yaml authors the LOUD sentinel {key}={value!r}, and " f"clusters/{env} does not state that knob — so that overlay INHERITS the sentinel. " "The TBD-PROD-* convention only works if a sentinel cannot be inherited by " "omission; an omitted op is how prod silently ships a value nobody decided.") # --------------------------------------- HALF B corpus: authored values + anchoring + canaries rows: list[tuple[str, str, str, str]] = [] # guard, expect, origin, value # (B1) VALUES: base default and every overlay's value, for every guarded knob. for key in sorted(guarded_keys): seen: list[tuple[str, str]] = [] if key in base: seen.append((f"clusters/base/cluster-vars.yaml:{key}", base[key])) for env, eff in sorted(overlays.items()): seen.append((f"clusters/{env} effective:{key}", eff.get(key, ""))) check(bool(seen), f"guarded knob {key} is not defined in cluster-vars at all — the guard " "protects a knob nothing authors") for origin, value in seen: expect = "placeholder" if guard_shaped(value) else "usable" for name in sorted(guard_words): rows.append((name, expect, origin, value)) # (B3) ANCHORING: the guard's own vocabulary must match ANYWHERE, not just as a prefix, # and a realistic endpoint must still be accepted. for name, words in sorted(guard_words.items()): for w in sorted(set(words)): rows.append((name, "placeholder", f"anchoring:{name}:mid-string({w})", f"https://account-{w}-id.r2.cloudflarestorage.com")) rows.append((name, "placeholder", f"anchoring:{name}:tail({w})", f"https://r2-endpoint.example.invalid/{w}")) rows.append((name, "placeholder", f"canary:{name}:empty", "")) rows.append((name, "usable", f"canary:{name}:real-r2", "https://0123456789abcdef0123456789abcd.r2.cloudflarestorage.com")) rows.append((name, "usable", f"canary:{name}:real-minio", "http://192.168.1.9:9000")) # Every guarded knob must reach the oracle once per authored source (base + each overlay), # so a knob silently dropping out of the corpus is a failure rather than a quiet pass. if GUARD_REL: authored = [r for r in rows if not r[2].startswith(("anchoring:", "canary:"))] expected_rows = len(guarded_keys) * (1 + len(overlays)) * len(guard_words) check(len(authored) == expected_rows, f"{len(authored)} authored value(s) reached the guard, expected {expected_rows} " f"({len(guarded_keys)} guarded knob(s) x base+{len(overlays)} overlay(s) x " f"{len(guard_words)} predicate(s)) — a guarded knob is missing from cluster-vars or " "from an overlay's effective values, and the coupling is only partly exercised") # --------------------------------------------------- (A5) the check is load-bearing L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else "" GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else "" check("--layer l0" in L0, "ci/l0.sh does not run the L0 conformance layer — this check would never execute in the gate") check(bool(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)) and bool(re.search(r"fail\).*BLOCK=1", GATE)), "ci/gate.sh does not set the merge block when the l0 stage fails") # ------------------------------------------------------- non-vacuity floor for HALF A # HALF A is the half that runs unconditionally, so it carries its own floor: if the # authored-value sweep shrinks to nothing, a pass here means nothing. check(a1_rows >= 20, f"only {a1_rows} authored overlay value(s) were audited by (A1) — the sweep over " "clusters/ effective cluster-vars has collapsed, and a pass would be meaningless. " "Expected roughly (knobs x overlays); check the ConfigMap and overlay parse above.") # --------------------------------------------------------------------------- handoff with (WORK / "corpus.tsv").open("w") as fh: for name, expect, origin, value in rows: fh.write(f"{name}\t{expect}\t{origin}\t{value}\n") (WORK / "findings.txt").write_text("".join(f"{f}\n" for f in FAILURES)) (WORK / "counts.txt").write_text(f"{CHECKS}\n") if GUARD_REL: print(f"guard file: {GUARD_REL} predicates: {', '.join(sorted(guard_words))}") print(f"guarded knobs: {', '.join(sorted(guarded_keys))} " f"consumers traced: {len(consumers)}") else: print("guard file: NONE FOUND in the tree — HALF B (guard/value coupling) IS NOT PROVEN " "ON THIS TREE. No *_is_placeholder predicate exists to extract: bootstrap/standup.sh " "has never been on main (no deletion commit exists; it lives on unmerged prod-standup " "branches). HALF A below runs against the real authored values and is what currently " "prevents the 2026-09-05 outcome. When a guard lands ANYWHERE in the tree, discovery " "picks it up and HALF B activates with no edit to this check.") print(f"detector vocabulary: {', '.join(VOCAB)}") print(f"HALF A: overlays {', '.join(sorted(overlays))}; {len(base)} base knob(s); " f"(A1) {a1_rows} authored overlay value(s) screened; (A2) {a2_rows} empty value(s) " f"checked for explicit statement; (A3) {a3_rows} base-sentinel inheritance check(s)") for name, expect, origin, value in rows: if not origin.startswith(("anchoring:", "canary:")): print(f" authored {origin} = {value!r} -> expect {expect}") PY GUARD_REL="$(tr -d '\n' < "${WORK}/mode.txt")" # --------------------------------------------------------------------------------------- # PHASE 2 (bash): source the REAL predicate lines and run the corpus through them. # Empty when no guard exists — HALF A has already run and carries its own floor. # --------------------------------------------------------------------------------------- # shellcheck disable=SC1090,SC1091 source "${WORK}/guards.sh" oracle_failures=0 oracle_rows=0 while IFS=$'\t' read -r guard expect origin value; do [ -n "${guard}" ] || continue oracle_rows=$((oracle_rows + 1)) if "${guard}" "${value}"; then verdict=placeholder; else verdict=usable; fi if [ "${verdict}" != "${expect}" ]; then oracle_failures=$((oracle_failures + 1)) if [ "${expect}" = placeholder ]; then printf '%s\n' "${guard}() in ${GUARD_REL} ACCEPTS '${value}' (from ${origin}) as a usable value, but it is placeholder-shaped. This is the 2026-09-05 failure exactly: the guard is asked about a value it does not recognise, returns 'fine', and the tier it protects is enabled against something that does not exist. Fix the predicate (match the idiom anywhere, not as a prefix) or stop authoring that idiom." >> "${WORK}/findings.txt" else printf '%s\n' "${guard}() in ${GUARD_REL} REFUSES '${value}' (from ${origin}), which is a usable value. An over-matching guard leaves the tier it protects suspended forever and reports it as an operator decision — a silent failure in the other direction." >> "${WORK}/findings.txt" fi fi done < "${WORK}/corpus.tsv" # ------------------------------------------------------------------------------ verdict py_checks="$(cat "${WORK}/counts.txt")" total=$((py_checks + oracle_rows)) findings="$(cat "${WORK}/findings.txt")" # NO-OP GUARDS: this repo has been bitten by a check that passed when perturbed because it # asked a question that was always satisfiable. The floors below are the minimum evidence # for the verdict to mean anything, not a style rule. The oracle floor applies only when a # guard exists — when none does, HALF A's own floor (a1_rows >= 20, asserted in phase 1) is # what stops this from degenerating, and the absence is reported on stdout. if [ -n "${GUARD_REL}" ] && [ "${oracle_rows}" -lt 6 ]; then findings="${findings}only ${oracle_rows} value(s) were put to the real guard in ${GUARD_REL} — the coupling is not actually being exercised " fi if [ "${total}" -lt 20 ]; then findings="${findings}only ${total} assertions ran — the check degenerated to a near-no-op and its pass would be meaningless " fi if [ -n "${findings}" ]; then echo "CHECK FAIL (placeholder-guard-coupling): $(printf '%s' "${findings}" | grep -c . ) of ${total} assertions failed" >&2 printf '%s' "${findings}" | sed 's/^/ - /' >&2 exit 1 fi if [ -n "${GUARD_REL}" ]; then echo "check OK: placeholder-guard-coupling — ${total} assertions. HALF A: no environment overlay ships a placeholder-shaped effective cluster-vars value, every empty value is stated explicitly rather than inherited, and the detector canaries hold (including the exact 2026-09-05 string, matched mid-string). HALF B: ${oracle_rows} value(s) run through the REAL extracted predicate(s) from ${GUARD_REL} (no reimplementation), covering every authored value of every guarded knob in base and each overlay, the guard's vocabulary matched mid-string, and the empty/real-endpoint canaries. Static slice only: refusal, not reachability — see the header." else echo "check OK: placeholder-guard-coupling — ${total} assertions. HALF A ONLY: no environment overlay ships a placeholder-shaped effective cluster-vars value, every empty value is stated explicitly rather than inherited, no LOUD base sentinel is inherited by omission, and the detector canaries hold (including the exact 2026-09-05 string 'https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com', matched mid-string). HALF B (guard/value coupling) IS NOT PROVEN: no *_is_placeholder predicate exists anywhere in the tree to extract — see the stdout note above and the header. This is a REPORTED gap, not a skip: what a guard would be asked about is proven not to be a placeholder in the first place." fi