From a984442cae1884bfc84faa47c4272e6dd918fcaf Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 6 Sep 2026 08:47:08 -0500 Subject: [PATCH] harden: pin node timezone to UTC, and TEST the guard/value coupling that bit us MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two debt items, both about making a latent problem impossible rather than fixed. === 1. Node timezone pinned to Etc/UTC (node_baseline, `system` tag) === The prod nodes disagreed, measured from k3s's own etcd snapshot annotations: prod-1/2 at -05:00 (Central), prod-3 at -04:00 (Eastern). Functionally harmless — Kubernetes and etcd are UTC internally, and in UTC those snapshots are ~4 minutes apart, which is correct staggering — but it makes cross-node log correlation actively misleading, and this cluster has had two unplanned reboots in two days. `Etc/UTC` chosen on repo convention, not just principle: catalog/plex.yaml records "no other service in the repo sets a timezone ... so use the safe default" with TZ: Etc/UTC, and apps/wave-0/plex.yaml matches (platform/crafty is America/Chicago DELIBERATELY — a user-facing game schedule, not a log clock). Decisive extra fact: THE THREE DEV NODES ARE ALREADY Etc/UTC, so this makes prod match dev rather than inventing a convention. UTC also removes the class — no DST discontinuity, node time == platform time. New knob `node_baseline_timezone`, overridable per environment via group_vars (the same seam as node_baseline_backports_kernel) but deliberately NOT restated there: no environment needs to differ, and a second copy is a second thing to disagree. Guarded by an assert in the role's existing BLOCKED style before the module runs. `--check --diff --tags system -e k3s_hosts=k3s_prod` INDEPENDENTLY CONFIRMS the offset evidence, and is the whole verification — the play was never run for real: k3s-prod-1: -"America/Chicago" +"Etc/UTC" changed k3s-prod-2: -"America/Chicago" +"Etc/UTC" changed k3s-prod-3: -"America/New_York" +"Etc/UTC" changed The same --check against DEV reports the task `ok` on all three — idempotence demonstrated against live hosts, in check mode only. `--list-hosts` confirms `-e k3s_hosts=k3s_prod` selects the three prod nodes (the diagnostic plays take their OWN var; the wrong flag silently targets dev). *** AUTHORED, NOT APPLIED. *** Applying it is a live-cluster action and was deliberately left for the operator to schedule: prod's Longhorn had only just finished resyncing after a node reboot, and changing a node's clock config during storage recovery is not an unattended move. `ansible-inventory --list` was NOT used anywhere — it prints DECRYPTED vault values (the passphrase is mounted into the dev container) and has leaked the prod become-password to a transcript once already. === 2. New L0 check `placeholder-guard-coupling` (R-ENV-02, R-STATE-03) === On 2026-09-05 the PM changed a placeholder in clusters/base/cluster-vars.yaml from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID...`. The guards in bootstrap/standup.sh matched the glob `TBD-*` — a PREFIX. All three silently died, and `resume` would have enabled velero on prod against a nonexistent hostname: installs clean, reports healthy, no backups exist. Found by accident. A guard in one file keyed on the SHAPE of a value in another file, with nothing tying them. The check exercises the REAL predicate — it does not reimplement it. standup.sh is not sourceable (top-level argv parsing hard-exits, cd, export, EXIT trap), so the check extracts the predicate's own source lines and sources only those. Faithfulness is pinned by: discovery of every `*_is_placeholder` (not a hardcoded name), exactly-one-definition, "extracted block is a verbatim substring", a self-containment gate, a canary (must refuse "", must accept a real URL), and a scan for a divergent second copy. If the predicate grows a dependency the check FAILS LOUDLY and names the remedy rather than degrading to testing a copy — the failure mode it exists for. Both sides derive from real files: guarded knobs from a taint trace in standup.sh (`x="$(cvar KEY)"` -> alias hop -> `*_is_placeholder "$x"`); values from base cluster-vars plus each overlay's JSON6902 /data/KEY patches. The independent detector is deliberately NOT the guard: seed idioms UNION the guard's own words with glob anchors STRIPPED — so a prefix-anchored guard cannot pass, and deleting an arm cannot make a value stop being placeholder-shaped. Real tree: 32 assertions, 18 values through the extracted predicate, PASS. NON-VACUITY VERIFIED BY ME, twice, on throwaway copies (the real repo and the read-only spec repo untouched). My first perturbation tripped the self-containment rail instead of the anchoring property, so I redid it shape-preserving — substring globs -> prefix globs, nothing else — and it failed on exactly the right property, naming mid-string and tail acceptances with the diagnosis "This is the 2026-09-05 failure exactly". Control: unperturbed copy exits 0. === 3. ci/touched-requirements.sh: bootstrap/ was UNMAPPED (found by the agent) === The path->requirement map had no `bootstrap/*` entry, so a change touching standup.sh ALONGSIDE any already-mapped path scoped the L0 run to that other path's requirements and silently excluded the new check — the one that exists to catch exactly the 2026-09-05 class. A safety net with a hole where the incident was. Mapped to R-STATE-03 R-GITOPS-01; verified the map now resolves bootstrap paths. VERIFIED: full CI gate PASS. `run.sh --layer l0` 8/8 PASS including the new check. Coverage unchanged at 45/51 (the new check tags existing requirements). Refs R-ENV-02, R-STATE-03, R-TEST-01, R-PLAT-01. Co-Authored-By: Claude Opus 5 --- ci/touched-requirements.sh | 9 + .../chainsaw-test.yaml | 68 +++ .../placeholder-guard-coupling/check.sh | 516 ++++++++++++++++++ infra/ansible/collections/requirements.yml | 3 +- .../roles/node_baseline/defaults/main.yml | 39 ++ .../roles/node_baseline/tasks/main.yml | 46 ++ 6 files changed, 680 insertions(+), 1 deletion(-) create mode 100644 conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/placeholder-guard-coupling/check.sh diff --git a/ci/touched-requirements.sh b/ci/touched-requirements.sh index 181c36a..a7f16ea 100755 --- a/ci/touched-requirements.sh +++ b/ci/touched-requirements.sh @@ -183,6 +183,15 @@ path_requirements() { platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;; conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;; clusters/*) echo "R-GITOPS-01 R-ENV-02" ;; + # bootstrap/ was UNMAPPED until 2026-09-06, and the omission was load-bearing: + # bootstrap/standup.sh carries the placeholder guards that decide whether `velero` may be + # enabled against a real backup endpoint. A change touching standup.sh ALONGSIDE any + # already-mapped path scoped the L0 run to that other path's requirements and silently + # excluded conformance/tests/l0-static/placeholder-guard-coupling — the check that exists + # precisely to catch a guard/value mismatch. The 2026-09-05 guard bug (a prefix glob that + # stopped matching when the value it guards changed, in a different file) is exactly the + # class that omission would hide. + bootstrap/*) echo "R-STATE-03 R-GITOPS-01" ;; .gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;; *) : ;; esac diff --git a/conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml b/conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml new file mode 100644 index 0000000..54b6b2e --- /dev/null +++ b/conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml @@ -0,0 +1,68 @@ +# ============================================================================= +# L0 STATIC — a placeholder value in cluster-vars is REFUSED by the guard that owns it +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# THE BUG CLASS THIS CLOSES (it has now bitten three times in one file). A guard in +# `bootstrap/standup.sh` keys on the SHAPE of a value that lives in +# `clusters/base/cluster-vars.yaml`, and nothing tied the two together. On 2026-09-05 the +# S3 placeholder changed from `TBD-PROD-S3-BACKUP-ENDPOINT` to +# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`; the three guards matched the glob +# `TBD-*`, a PREFIX match, and `fnmatch('https://TBD-R2-…','TBD-*')` is False. All three +# guards silently stopped matching, and `standup.sh resume` would have resumed `velero` on +# prod against a nonexistent hostname: installs clean, reports healthy, NO BACKUPS EXIST. +# It was found by accident. The guard is fixed (`s3_is_placeholder`, substring); the +# COUPLING was untested, which is what makes the class reintroducible. +# +# GIVEN the repository as authored, +# WHEN the REAL predicate is extracted from bootstrap/standup.sh and executed by bash +# (never reimplemented — see check.sh "HOW THE REAL GUARD IS EXERCISED"), +# THEN (1) every authored value of a guarded knob that is placeholder-shaped by an +# INDEPENDENT detector is actually refused by that guard, in base and in every +# environment overlay; +# (2) no environment overlay carries a placeholder-shaped value for a knob no guard +# refuses; +# (3) every function in standup.sh that reads a guarded knob applies the guard to it +# (the "three guards died at once" regression); +# (4) the guard matches its own vocabulary ANYWHERE in a value, not just as a prefix +# — the exact anchoring defect above — and still accepts a real endpoint; +# AND (5) the extraction is faithful, and the check fails LOUDLY rather than silently +# testing a copy if the guard stops being a self-contained predicate. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library + +# PyYAML + bash); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test +# wraps the SAME script. +# +# NOT proven: that the endpoint is REACHABLE or that backups exist (runtime facts — +# `standup.sh s3check` probes the former from a pod, l3-fault-injection/dr-restore proves +# the latter); that guards for knobs OTHER than the placeholder-refusing ones are correct; +# and that the guard runs at all in a real standup (the dispatch path is shell, executed +# only against a live cluster). R-STATE-03 is tagged for the narrow static slice only: the +# authored backup target cannot be a placeholder that the guard fails to recognise. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: placeholder-guard-coupling + labels: + conformance.home.k3s/R-ENV-02: "true" + conformance.home.k3s/R-STATE-03: "true" + annotations: + conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Every placeholder-shaped value authored in cluster-vars, for any environment + overlay, is actually recognised as a placeholder by the real bootstrap guard that + is supposed to refuse it — so a value change in one file cannot silently disarm a + guard in another and enable velero against an endpoint that does not exist. +spec: + steps: + - name: couple-authored-placeholders-to-the-real-guard + description: >- + Extract the real placeholder predicate from bootstrap/standup.sh, run it against + every authored cluster-vars value the repo actually carries, and assert the guard + refuses each one an independent detector calls a placeholder. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/placeholder-guard-coupling/check.sh b/conformance/tests/l0-static/placeholder-guard-coupling/check.sh new file mode 100755 index 0000000..d3814e6 --- /dev/null +++ b/conformance/tests/l0-static/placeholder-guard-coupling/check.sh @@ -0,0 +1,516 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — placeholder-guard-coupling +# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)." +# — spec docs/02-environments-and-provisioning.md) +# R-STATE-03 ("Scheduled off-cluster backups + restore (DR)." +# — spec docs/04-storage-and-state.md) +# +# WHY THIS EXISTS — A GUARD KEYED ON A VALUE THAT LIVES IN ANOTHER FILE +# -------------------------------------------------------------------- +# `bootstrap/standup.sh` refuses to resume tiers whose configuration is still a +# placeholder. Three of those guards keyed on the SHAPE of `S3_BACKUP_ENDPOINT`, which is +# authored in `clusters/base/cluster-vars.yaml`. On 2026-09-05 the placeholder changed +# from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` +# (the MinIO -> Cloudflare R2 move). The guards matched the glob `TBD-*` — a PREFIX match — +# and `fnmatch('https://TBD-R2-…','TBD-*')` is FALSE. All three guards silently stopped +# matching. `standup.sh resume` would then have resumed `velero` on prod against a hostname +# that does not exist: the HelmRelease installs clean, the Kustomization reports Ready, and +# NO BACKUPS EXIST. The only reason it was not shipped is that someone noticed by accident. +# +# The predicate is fixed (`s3_is_placeholder`, substring match). The COUPLING is what was +# never tested: a guard in one file keying on a value in another, with nothing tying them +# together. This check is that tie, and it is derived from BOTH real files — no current +# string is hardcoded here, so it does not rot when the placeholder or the guard changes. +# +# HOW THE REAL GUARD IS EXERCISED (and why not a copy) +# --------------------------------------------------- +# `bootstrap/standup.sh` is NOT sourceable: at top level it parses argv, hard-exits when +# --kubeconfig/--expect-server/ are absent, `cd`s to the repo root, exports +# KUBECONFIG, installs an EXIT trap and finally dispatches steps against a live cluster. +# So this check EXTRACTS the predicate's own source lines from the file and sources ONLY +# those, then calls the real function. That is the real bytes, not a reimplementation: +# * every function whose name ends `_is_placeholder` is discovered, not named here; +# * a duplicate definition of one is a failure (which of the two would run?); +# * the extracted body must be a SELF-CONTAINED predicate — `case`/`return` and nothing +# else, no command substitution, no call to another function. If the guard ever grows +# a dependency, extraction stops being faithful and THIS CHECK FAILS LOUDLY, naming the +# remedy (move the predicate into a small sourceable library that standup.sh sources) +# rather than quietly falling back to testing a copy of the logic; +# * a CANARY pins the extraction itself: the extracted function must refuse the empty +# string and accept a realistic endpoint. An extraction that grabbed the wrong lines, +# or a predicate degraded to always-true / always-false, fails there — this repo has +# already been bitten by a check that passed when perturbed because it asked a question +# that was always satisfiable. +# There is deliberately NO second implementation of the matching logic anywhere in this file. +# +# THE INDEPENDENT DETECTOR (the other half of the comparison) +# ---------------------------------------------------------- +# Asking "does the guard agree with the guard?" proves nothing, so the set of +# placeholder-shaped values is decided by a detector written HERE, whose vocabulary is the +# UNION of two independent sources: +# * SEED — universal loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER, REPLACE-ME, +# CHANGE-ME, FILL-ME, SET-ME, XXXX), plus `` templates and the empty +# string. These are NOT derived from the guard, so DELETING a guard arm cannot make a +# value stop being placeholder-shaped — it makes assertion (1) fail, which is the point. +# A value using an idiom the guard does not know is a finding either way: fix it by +# giving the knob a real value, or by teaching the guard the idiom. +# * the guard's OWN vocabulary, read out of its case-arms with the glob anchors STRIPPED. +# Stripping the anchors is what makes the comparison non-circular in the dimension that +# actually broke: the detector matches a word ANYWHERE, case-insensitively, and the +# guard must agree. A prefix-anchored guard cannot pass. +# +# GIVEN the repository as authored, +# WHEN the real predicate(s) are extracted from bootstrap/standup.sh and executed, +# THEN (1) VALUES: for every knob a guard is responsible for, every authored value — the +# base default AND each environment overlay's value — that the independent +# detector calls placeholder-shaped is REFUSED by the real guard, and every value +# it calls usable is ACCEPTED (a guard that refuses a real endpoint leaves velero +# suspended forever, which is the same class of silent failure); +# (2) OVERLAYS: no environment overlay's EFFECTIVE cluster-vars carries a +# placeholder-shaped value for a knob that no guard refuses. (Base alone is +# exempt: it is not a bootstrap target — `flux bootstrap --path=clusters/` — +# and its loud sentinels exist precisely to be overridden. A base placeholder an +# overlay forgets to override IS caught, because it becomes that overlay's +# effective value.) +# (3) CONSUMERS: every function in standup.sh that reads a guarded knob (via `cvar`, +# following one hop of variable aliasing) applies a placeholder guard to it. This +# is the "all three guards died at once" regression stated as a property: the +# guard being correct is worthless if a consumer stops calling it. +# (4) ANCHORING: for each word in the guard's own vocabulary, the guard refuses a +# value carrying that word in the MIDDLE (not at the start) — the precise defect +# of 2026-09-05 — while still accepting a realistic endpoint. +# (5) HARNESS: exactly one definition per predicate, the extracted body is verbatim +# from the file and self-contained, the canary holds, no divergent second copy of +# the matching logic exists elsewhere in standup.sh, and a floor on the number of +# assertions run keeps a pass from degenerating into a no-op. +# AND (6) LOAD-BEARING: ci/l0.sh runs the L0 conformance layer on every gate run and +# ci/gate.sh sets the merge block when that stage fails. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps) +# - Nothing about RUNTIME: not that the endpoint resolves, not that credentials work, not +# that a backup exists. `standup.sh s3check` probes reachability from a pod (operator- +# run, cluster-gated) and conformance/tests/l3-fault-injection/dr-restore proves restore. +# - It does not execute standup.sh's dispatch path, so it cannot prove the guard is +# REACHED in a real run — only that, when reached, it refuses what it must. The +# consumer property (3) is a static approximation of reachability: it is a text-level +# taint trace over one file, not an interpreter, and a guard whose result is computed +# and then ignored would satisfy it. +# - It says nothing about guards keyed on shapes OTHER than placeholders (e.g. the +# EXTERNALDNS_TXT_OWNER_ID == 'default' test), nor about placeholder-shaped values +# outside cluster-vars (the Ansible inventory's BLOCKED/NOT-A-DEVICE sentinels are +# asserted by the node_baseline role itself, at play time). +# - Stage 3b of ci/l0.sh scopes the conformance run to the TOUCHED requirements. An edit +# to bootstrap/* attributes NO requirement in ci/touched-requirements.sh's path map, so +# the touched set is empty and the FULL L0 layer runs (verified: ci/lib.sh +# load_touched_requirements) — this check runs. An edit to clusters/* maps to +# R-ENV-02, which this check carries, so a cluster-vars value change also selects it. +# A change that touches standup.sh ALONGSIDE some unrelated mapped path (say +# platform/cilium/) would scope the run away from this check; closing that needs a +# `bootstrap/*` entry in the path map, which is not this check's file to edit. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" +STANDUP_REL="${STANDUP_REL:-bootstrap/standup.sh}" + +cd "${ROOT}" + +if [ ! -f "${ROOT}/${STANDUP_REL}" ]; then + echo "check SKIPPED (placeholder-guard-coupling): ${STANDUP_REL} is absent — nothing to couple." >&2 + exit 3 +fi + +WORK="$(mktemp -d)" +trap 'rm -rf "${WORK}"' EXIT + +# --------------------------------------------------------------------------------------- +# PHASE 1 (python): extract the real predicate(s), derive the corpus from the real files, +# and run every assertion that does not need a shell. Writes: +# ${WORK}/guards.sh the predicate source lines, verbatim, and nothing else +# ${WORK}/corpus.tsv \t\t\t +# ${WORK}/findings.txt one line per failed assertion +# ${WORK}/counts.txt +# --------------------------------------------------------------------------------------- +"${PYTHON}" - "${ROOT}" "${WORK}" "${STANDUP_REL}" <<'PY' +import re +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT, WORK, STANDUP_REL = Path(sys.argv[1]), Path(sys.argv[2]), sys.argv[3] +FAILURES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def bail(msg): + """A failure that makes the rest of the check meaningless: report and stop.""" + print(f"CHECK FAIL (placeholder-guard-coupling): {msg}", file=sys.stderr) + (WORK / "findings.txt").write_text(msg + "\n") + (WORK / "counts.txt").write_text("0\n") + (WORK / "corpus.tsv").write_text("") + (WORK / "guards.sh").write_text("") + raise SystemExit(1) + + +TEXT = (ROOT / STANDUP_REL).read_text() +LINES = TEXT.splitlines() + +# ------------------------------------------------------------------ (5) extract the guard +DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{") +defs: dict[str, list[int]] = {} +for i, line in enumerate(LINES): + m = DEF_RE.match(line) + if m: + defs.setdefault(m.group(1), []).append(i) + +if not defs: + bail(f"{STANDUP_REL} defines no *_is_placeholder predicate. Either the guard was renamed " + "(update this check's discovery pattern in the SAME change) or it was DELETED, which " + "is the 2026-09-05 failure with the guard removed instead of disarmed.") + +# An "argument-shaped" call: "" +CALL_RE = re.compile(r"\b([a-z0-9_]+_is_placeholder)\s+\"([^\"]*)\"") + +guard_src: list[str] = [] +guard_words: dict[str, list[str]] = {} +ARM_RE = re.compile(r"^(?P[^()]*?)\)\s*return\s+[01]\s*;;$") +ALLOWED = ( + re.compile(r"^$"), + re.compile(r"^#"), + re.compile(r"^case\s+\"\$\{?[0-9A-Za-z_]+\}?\"\s+in$"), + re.compile(r"^esac$"), + re.compile(r"^return\s+[01]$"), + ARM_RE, +) + +for name, at in sorted(defs.items()): + check(len(at) == 1, + f"{STANDUP_REL} defines {name}() {len(at)} times (lines " + f"{', '.join(str(a + 1) for a in at)}) — which one runs depends on file order, and " + "this check can no longer say which one it extracted. Keep exactly one definition.") + if len(at) != 1: + continue + start = at[0] + end = next((j for j in range(start + 1, len(LINES)) if LINES[j] == "}"), None) + if end is None: + bail(f"{name}() at {STANDUP_REL}:{start + 1} has no closing brace at column 0 — the " + "extraction boundary this check relies on is gone. Restore the conventional " + "layout, or move the predicate into a sourceable library and point this check at it.") + body = LINES[start:end + 1] + block = "\n".join(body) + check(block in TEXT, + f"extracted {name}() is not a verbatim substring of {STANDUP_REL} — extraction bug; " + "the harness would be testing something the script does not contain") + + words: list[str] = [] + for raw in body[1:-1]: + s = raw.strip() + if not any(rx.match(s) for rx in ALLOWED) or "$(" in s or "`" in s: + bail(f"{name}() at {STANDUP_REL}:{start + 1} is no longer a SELF-CONTAINED " + f"case/return predicate (offending line: {s!r}). This check exercises the REAL " + "function by sourcing its own lines; a body that calls out to anything else " + "cannot be extracted faithfully, and silently testing a COPY of the logic is " + "exactly the failure mode this check exists to prevent. REMEDY: move the " + "predicate into a tiny library that standup.sh sources (e.g. " + "bootstrap/lib/placeholder.sh) and point STANDUP_REL/this extractor at it — do " + "NOT reimplement the matching here.") + m = ARM_RE.match(s) + if not m: + continue + for pat in m.group("pats").split("|"): + pat = pat.strip() + if pat in ('""', "''", ""): + continue # the empty-value arm: covered by the canary + words.append(pat.strip("'\"").strip("*")) + check(len(words) >= 1, + f"{name}() has no non-empty placeholder pattern left — every arm that recognises a " + "loud placeholder string has been removed, so the guard now only refuses the empty " + "value and any 'TBD'-style endpoint would sail straight through it") + guard_words[name] = words + guard_src.append(block) + +(WORK / "guards.sh").write_text("\n".join(guard_src) + "\n") + +# ------------------------------------------------- no divergent second copy of the logic +# Any of the guard's own vocabulary appearing in EXECUTABLE text outside the predicate means +# a second, independently-drifting implementation of the same test (which is how three +# guards came to disagree with one value in the first place). +guard_line_nums = {j for name, at in defs.items() if len(at) == 1 + for j in range(at[0], next(k for k in range(at[0] + 1, len(LINES)) + if LINES[k] == "}") + 1)} +all_words = sorted({w.upper() for ws in guard_words.values() for w in ws if w}) +divergent: list[str] = [] +for j, line in enumerate(LINES): + if j in guard_line_nums or line.strip().startswith("#"): + continue + code = line.split("#", 1)[0] + hit = [w for w in all_words if w in code.upper()] + if hit: + divergent.append(f"{STANDUP_REL}:{j + 1} ({', '.join(hit)}): {line.strip()!r}") +check(not divergent, + "the guard's own placeholder vocabulary appears in EXECUTABLE text outside the " + f"predicate — a SECOND implementation of the same test, which is precisely how a value " + f"change disarmed three guards at once. Call the predicate instead. Sites: " + + " | ".join(divergent)) + +# ------------------------------------------------------------- cluster-vars, as authored +CV = ROOT / "clusters" / "base" / "cluster-vars.yaml" +if not CV.is_file(): + bail("clusters/base/cluster-vars.yaml is absent — the authored half of the coupling is " + "missing and this check cannot mean anything") + +base: dict[str, str] = {} +for d in yaml.safe_load_all(CV.read_text()): + if isinstance(d, dict) and d.get("kind") == "ConfigMap" \ + and (d.get("metadata") or {}).get("name") == "cluster-vars": + base = {str(k): "" if v is None else str(v) for k, v in (d.get("data") or {}).items()} +check(len(base) > 5, f"clusters/base/cluster-vars.yaml yielded {len(base)} knob(s) — the " + "ConfigMap parse failed, so every value assertion below would be vacuous") + +overlays: dict[str, dict[str, str]] = {} +for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")): + env = overlay.parent.name + if env == "base": + continue + eff = dict(base) + try: + k = yaml.safe_load(overlay.read_text()) or {} + except Exception as exc: # noqa: BLE001 + check(False, f"clusters/{env}/kustomization.yaml does not parse as YAML ({exc}) — this " + "check cannot compute that overlay's effective values") + continue + for p in (k.get("patches") or []): + tgt = p.get("target") or {} + if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars": + continue + try: + ops = yaml.safe_load(p.get("patch") or "") or [] + except Exception: # noqa: BLE001 + continue + for op in ops if isinstance(ops, list) else []: + if not isinstance(op, dict): + continue + m = re.match(r"^/data/([A-Za-z0-9_]+)$", str(op.get("path", ""))) + if m and str(op.get("op", "")) in ("replace", "add"): + eff[m.group(1)] = "" if op.get("value") is None else str(op["value"]) + overlays[env] = eff +check(len(overlays) >= 1, + "no clusters//kustomization.yaml overlay was parsed — assertion (2) would be vacuous") + +# ------------------------------------------------ which knobs is a guard responsible for? +# Function bodies of standup.sh, then a one-hop taint trace from `x="$(cvar KEY)"` to a +# `*_is_placeholder "${x}"` call. Both halves are read out of the file; nothing is named here. +FUNC_RE = re.compile(r"^([a-z0-9_]+)\(\)\s*\{") +funcs: dict[str, list[str]] = {} +i = 0 +while i < len(LINES): + m = FUNC_RE.match(LINES[i]) + if m: + end = next((j for j in range(i + 1, len(LINES)) if LINES[j] == "}"), len(LINES) - 1) + funcs[m.group(1)] = LINES[i + 1:end] + i = end + 1 + continue + i += 1 +check(len(funcs) > 5, f"parsed only {len(funcs)} function(s) out of {STANDUP_REL} — the " + "consumer assertion (3) would be vacuous") + +CVAR_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\(\s*cvar\s+([A-Z0-9_]+)\s*\)\"") +ALIAS_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\{([A-Za-z_][A-Za-z0-9_]*)[:}]") + +guarded_keys: set[str] = set() +consumers: list[tuple[str, str, bool]] = [] # (func, key, guarded-in-this-func) +for fname, body in funcs.items(): + text = "\n".join(body) + keys_of_var: dict[str, set[str]] = {} + for m in CVAR_RE.finditer(text): + keys_of_var.setdefault(m.group(1), set()).add(m.group(2)) + # one hop of aliasing: ep="${live}" carries live's taint + for _ in range(3): + for m in ALIAS_RE.finditer(text): + src = keys_of_var.get(m.group(2)) + if src: + keys_of_var.setdefault(m.group(1), set()).update(src) + called_on = {m.group(2) for m in CALL_RE.finditer(text)} + called_vars = {re.sub(r"^\$\{?|\}?$", "", a) for a in called_on} + # Aggregate per (function, knob): the knob is guarded HERE if ANY variable carrying its + # value is passed to a predicate — `live="$(cvar K)"; ep="${live}"; guard "${ep}"` counts. + guarded_here: dict[str, bool] = {} + for var, keys in keys_of_var.items(): + for key in keys: + guarded_here[key] = guarded_here.get(key, False) or (var in called_vars) + for key, g in guarded_here.items(): + consumers.append((fname, key, g)) + if g: + guarded_keys.add(key) + +if not guarded_keys: + bail(f"no cluster-vars knob in {STANDUP_REL} is passed to a *_is_placeholder predicate. " + "Either the guards were removed, or the way the script reads cluster-vars changed and " + "this check's derivation must be updated in the SAME change — it must never silently " + "verify an empty set.") + +# (3) CONSUMERS: every reader of a guarded knob must apply the guard to it. +for fname, key, guarded in sorted(consumers): + if key not in guarded_keys: + continue # nothing claims responsibility for this knob at all + check(guarded, + f"{STANDUP_REL}: {fname}() reads cluster-vars knob {key} but never passes it to a " + "*_is_placeholder guard, while other functions do. On 2026-09-05 all three consumers " + "of S3_BACKUP_ENDPOINT were disarmed at once; a consumer that reads the knob and skips " + "the guard is that failure in one function.") + +# ---------------------------------------------------------------- independent detector +# Deliberately NOT derived from the guard alone (see the header): SEED is universal +# loud-placeholder vocabulary, and the guard's own words are added so that a guard whose +# ANCHORING is wrong still has its vocabulary tested against real values. +SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", "REPLACE-ME", "REPLACEME", + "CHANGE-ME", "CHANGEME", "FILL-ME", "FILLME", "SET-ME", "XXXX"] +VOCAB = sorted({w.upper() for w in SEED} | set(all_words)) +ANGLE = re.compile(r"<[^>\s]+>") + + +def placeholder_shaped(value: str) -> bool: + if value.strip() == "": + return True # an unset endpoint is not a target + up = value.upper() + return any(w in up for w in VOCAB) or bool(ANGLE.search(value)) + + +rows: list[tuple[str, str, str, str]] = [] # guard, expect, origin, value +GUARD = sorted(guard_words)[0] if len(guard_words) == 1 else None + +# (1) VALUES: base default and every overlay's value, for every guarded knob. +for key in sorted(guarded_keys): + seen: list[tuple[str, str]] = [] + if key in base: + seen.append((f"clusters/base/cluster-vars.yaml:{key}", base[key])) + for env, eff in sorted(overlays.items()): + seen.append((f"clusters/{env} effective:{key}", eff.get(key, ""))) + check(bool(seen), f"guarded knob {key} is not defined in cluster-vars at all — the guard " + "protects a knob nothing authors") + for origin, value in seen: + expect = "placeholder" if placeholder_shaped(value) else "usable" + for name in sorted(guard_words): + rows.append((name, expect, origin, value)) + +# (2) OVERLAYS: a placeholder-shaped effective value for an UNGUARDED knob. +for env, eff in sorted(overlays.items()): + for key, value in sorted(eff.items()): + if key in guarded_keys or not placeholder_shaped(value): + continue + check(False, + f"clusters/{env}: knob {key} resolves to the placeholder-shaped value {value!r}, " + f"and NO guard in {STANDUP_REL} refuses it. Either give it a real value before it " + "reaches an overlay, or guard it the way the S3 endpoint is guarded — an overlay " + "that ships a placeholder no guard recognises is the 2026-09-05 failure with a " + "different knob's name on it.") + +# (4) ANCHORING: the guard's own vocabulary must match ANYWHERE, not just as a prefix, +# and a realistic endpoint must still be accepted. +for name, words in sorted(guard_words.items()): + for w in sorted(set(words)): + rows.append((name, "placeholder", + f"anchoring:{name}:mid-string({w})", + f"https://account-{w}-id.r2.cloudflarestorage.com")) + rows.append((name, "placeholder", f"anchoring:{name}:tail({w})", + f"https://r2-endpoint.example.invalid/{w}")) + rows.append((name, "placeholder", f"canary:{name}:empty", "")) + rows.append((name, "usable", f"canary:{name}:real-r2", + "https://0123456789abcdef0123456789abcd.r2.cloudflarestorage.com")) + rows.append((name, "usable", f"canary:{name}:real-minio", "http://192.168.1.9:9000")) + +# Every guarded knob must reach the oracle once per authored source (base + each overlay), +# so a knob silently dropping out of the corpus is a failure rather than a quiet pass. +authored = [r for r in rows if not r[2].startswith(("anchoring:", "canary:"))] +expected_rows = len(guarded_keys) * (1 + len(overlays)) * len(guard_words) +check(len(authored) == expected_rows, + f"{len(authored)} authored value(s) reached the guard, expected {expected_rows} " + f"({len(guarded_keys)} guarded knob(s) x base+{len(overlays)} overlay(s) x " + f"{len(guard_words)} predicate(s)) — a guarded knob is missing from cluster-vars or from " + "an overlay's effective values, and the coupling is only partly exercised") + +# ------------------------------------------------------- (6) the check is load-bearing +L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else "" +GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else "" +check("--layer l0" in L0, + "ci/l0.sh does not run the L0 conformance layer — this check would never execute in the gate") +check(bool(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)) and bool(re.search(r"fail\).*BLOCK=1", GATE)), + "ci/gate.sh does not set the merge block when the l0 stage fails") + +# --------------------------------------------------------------------------- handoff +with (WORK / "corpus.tsv").open("w") as fh: + for name, expect, origin, value in rows: + fh.write(f"{name}\t{expect}\t{origin}\t{value}\n") +(WORK / "findings.txt").write_text("".join(f"{f}\n" for f in FAILURES)) +(WORK / "counts.txt").write_text(f"{CHECKS}\n") + +print(f"guards: {', '.join(sorted(guard_words))} " + f"vocabulary: {', '.join(VOCAB)}") +print(f"guarded knobs: {', '.join(sorted(guarded_keys))} " + f"consumers traced: {len(consumers)} overlays: {', '.join(sorted(overlays))}") +for name, expect, origin, value in rows: + if not origin.startswith(("anchoring:", "canary:")): + print(f" authored {origin} = {value!r} -> expect {expect}") +PY + +# --------------------------------------------------------------------------------------- +# PHASE 2 (bash): source the REAL predicate lines and run the corpus through them. +# --------------------------------------------------------------------------------------- +# shellcheck disable=SC1090,SC1091 +source "${WORK}/guards.sh" + +oracle_failures=0 +oracle_rows=0 +while IFS=$'\t' read -r guard expect origin value; do + [ -n "${guard}" ] || continue + oracle_rows=$((oracle_rows + 1)) + if "${guard}" "${value}"; then verdict=placeholder; else verdict=usable; fi + if [ "${verdict}" != "${expect}" ]; then + oracle_failures=$((oracle_failures + 1)) + if [ "${expect}" = placeholder ]; then + printf '%s\n' "${guard}() in ${STANDUP_REL} ACCEPTS '${value}' (from ${origin}) as a usable value, but it is placeholder-shaped. This is the 2026-09-05 failure exactly: the guard is asked about a value it does not recognise, returns 'fine', and the tier it protects is enabled against something that does not exist. Fix the predicate (match the idiom anywhere, not as a prefix) or stop authoring that idiom." >> "${WORK}/findings.txt" + else + printf '%s\n' "${guard}() in ${STANDUP_REL} REFUSES '${value}' (from ${origin}), which is a usable value. An over-matching guard leaves the tier it protects suspended forever and reports it as an operator decision — a silent failure in the other direction." >> "${WORK}/findings.txt" + fi + fi +done < "${WORK}/corpus.tsv" + +# ------------------------------------------------------------------------------ verdict +py_checks="$(cat "${WORK}/counts.txt")" +total=$((py_checks + oracle_rows)) +findings="$(cat "${WORK}/findings.txt")" + +# NO-OP GUARD: this repo has been bitten by a check that passed when perturbed because it +# asked a question that was always satisfiable. The floors below are the minimum evidence +# for the verdict to mean anything, not a style rule. +if [ "${oracle_rows}" -lt 6 ]; then + findings="${findings}only ${oracle_rows} value(s) were put to the real guard — the coupling is not actually being exercised +" +fi +if [ "${total}" -lt 20 ]; then + findings="${findings}only ${total} assertions ran — the check degenerated to a near-no-op and its pass would be meaningless +" +fi + +if [ -n "${findings}" ]; then + echo "CHECK FAIL (placeholder-guard-coupling): $(printf '%s' "${findings}" | grep -c . ) of ${total} assertions failed" >&2 + printf '%s' "${findings}" | sed 's/^/ - /' >&2 + exit 1 +fi + +echo "check OK: placeholder-guard-coupling — ${total} assertions; ${oracle_rows} value(s) run through the REAL extracted predicate(s) from ${STANDUP_REL} (no reimplementation), covering every authored cluster-vars value of every guarded knob in base and each overlay, the guard's own vocabulary matched mid-string, and the empty/real-endpoint canaries. Static slice only: refusal, not reachability — see the header." diff --git a/infra/ansible/collections/requirements.yml b/infra/ansible/collections/requirements.yml index 55fcdf1..0ee606b 100644 --- a/infra/ansible/collections/requirements.yml +++ b/infra/ansible/collections/requirements.yml @@ -2,7 +2,8 @@ # Ansible Galaxy collections required by infra/ansible. Install with: # ansible-galaxy collection install -r collections/requirements.yml collections: - # node_baseline role: community.general.modprobe, community.general.filesystem + # node_baseline role: community.general.modprobe, community.general.filesystem, + # community.general.timezone - name: community.general # node_baseline role: ansible.posix.mount - name: ansible.posix diff --git a/infra/ansible/roles/node_baseline/defaults/main.yml b/infra/ansible/roles/node_baseline/defaults/main.yml index c2a438f..059519d 100644 --- a/infra/ansible/roles/node_baseline/defaults/main.yml +++ b/infra/ansible/roles/node_baseline/defaults/main.yml @@ -81,6 +81,45 @@ node_baseline_kernel_modules: - dm_crypt - nfs +# --- NODE CLOCK / TIMEZONE (tag `system`) --------------------------------------------- +# EVERY NODE IS UTC. Measured drift that made this a knob rather than an assumption: the +# three prod nodes disagreed on their local zone (read off k3s's own etcd snapshot +# annotations, 2026-09-05) — +# k3s-prod-1 2026-09-05T16:31:28-05:00 (Central) +# k3s-prod-2 2026-09-05T16:29:07-05:00 (Central) +# k3s-prod-3 2026-09-05T17:27:35-04:00 (Eastern) +# In UTC those three are ~4 minutes apart, which is the CORRECT staggering: nothing was +# functionally wrong, because Kubernetes, etcd and the container runtime are UTC internally. +# What IS wrong is that any timestamp a node renders in LOCAL time — journald's default +# output, `ls -l`, a shell prompt, an etcd snapshot filename — cannot be compared across +# nodes without knowing each node's zone. Correlating three nodes' logs during an incident +# is exactly when nobody has that knowledge to hand, and this cluster took two unplanned +# node reboots in the 24h before this was written. +# +# WHY UTC AND NOT "STANDARDISE ON CENTRAL". Picking one local zone leaves the class of +# problem intact (a fourth node, or a DST boundary, re-creates it) and adds a translation +# step to every log read, because every timestamp the PLATFORM emits — Kubernetes events, +# Prometheus, Loki, Velero, CNPG WAL, the etcd snapshot annotations above — is already UTC. +# UTC removes the class: node-local time and platform time become the same clock, and there +# is no DST discontinuity in which a repeated or skipped hour breaks log ordering. This also +# matches the only timezone convention the repo already had — catalog/plex.yaml records +# "no other service in the repo sets a timezone ... so use the safe default" and both it and +# apps/wave-0/plex.yaml set TZ=Etc/UTC. (platform/crafty sets America/Chicago, deliberately: +# that is a USER-FACING game server's schedule, not a node's log clock.) +# +# WHY A KNOB AT ALL (R-ENV-02). The value is a node-level Ansible fact, so its override seam +# is inventory/group_vars/, NOT clusters/base/cluster-vars.yaml (a Flux postBuild +# ConfigMap no Ansible code reads) — the same seam node_baseline_backports_kernel and +# node_baseline_dns_servers use. NO ENVIRONMENT OVERRIDES IT TODAY, and one should not have +# to: dev and prod both want UTC and there is nothing environment-specific about a node +# clock. It is a variable rather than a literal in the task so a genuinely +# operator-facing one-off host (never a k3s node) can differ without forking the role. +# +# Must be a zoneinfo name present on the node (`timedatectl list-timezones`); the assert in +# tasks/main.yml rejects an empty or obviously-malformed value before the module runs, and +# community.general.timezone itself refuses a name /usr/share/zoneinfo does not have. +node_baseline_timezone: Etc/UTC + # --- DATA DISKS (tasks/disks.yml, tag `disks`) ---------------------------------------- # The SHAPE of a node's storage is DECLARED per environment, never inferred from whether # a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape: diff --git a/infra/ansible/roles/node_baseline/tasks/main.yml b/infra/ansible/roles/node_baseline/tasks/main.yml index b2fb6a7..276ac0e 100644 --- a/infra/ansible/roles/node_baseline/tasks/main.yml +++ b/infra/ansible/roles/node_baseline/tasks/main.yml @@ -262,6 +262,52 @@ vars: newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}" +# ===================================================================================== +# NODE CLOCK — tag `system`: every node renders time in the SAME zone, and that zone is UTC +# ===================================================================================== +# MEASURED DRIFT (2026-09-05), read off k3s's own etcd snapshot annotations on live prod: +# k3s-prod-1 2026-09-05T16:31:28-05:00 Central +# k3s-prod-2 2026-09-05T16:29:07-05:00 Central +# k3s-prod-3 2026-09-05T17:27:35-04:00 Eastern <-- a different UTC offset +# Functionally harmless (in UTC those three are ~4 minutes apart, which is the correct +# staggering — Kubernetes, etcd and containerd are UTC internally). Operationally it is a +# trap: any timestamp a node renders in LOCAL time (journald's default output, `ls -l`, an +# etcd snapshot filename) is not comparable across nodes without knowing each node's zone, +# and cross-node log correlation during an incident is precisely when nobody knows that. +# +# Rationale for UTC rather than standardising on one local zone, and for this being a knob +# whose override seam is inventory/group_vars/: see node_baseline_timezone in +# defaults/main.yml. NOT a cluster-vars knob — no Ansible code reads that ConfigMap. +# +# IDEMPOTENCE / BLAST RADIUS. community.general.timezone is a converge-to-state module: it +# reports `changed` only when /etc/localtime + /etc/timezone do not already name this zone, +# and it does NOT step the clock (no time jump, no NTP interaction) — only the RENDERING of +# an unchanged monotonic clock moves. No reboot is required. Already-running services that +# render local time keep the old zone until they are next restarted (the module restarts +# cron itself, which is the one scheduler on these nodes that cares); k3s, etcd and the +# kubelet are unaffected either way because they emit UTC regardless. +- name: Assert this node's timezone has actually been decided + ansible.builtin.assert: + that: + - node_baseline_timezone is string + - node_baseline_timezone | length > 0 + - "'BLOCKED' not in node_baseline_timezone" + - node_baseline_timezone is match('^[A-Za-z][A-Za-z0-9+_-]*(/[A-Za-z0-9+_.-]+)*$') + fail_msg: >- + node_baseline_timezone is not a usable zoneinfo name (got: + '{{ node_baseline_timezone | default("") }}'). It must be a name + /usr/share/zoneinfo actually carries — `timedatectl list-timezones` on the node — and + the repo default (Etc/UTC) is almost certainly what you want; see the rationale in + roles/node_baseline/defaults/main.yml. Empty is NOT "leave it alone": a node whose zone + nobody has decided is the config drift this task exists to remove. + quiet: true + tags: [system] + +- name: Pin the node timezone (cross-node log correlation; UTC by default) + community.general.timezone: + name: "{{ node_baseline_timezone }}" + tags: [system] + - name: Persist kernel modules for k3s/Cilium/Longhorn ansible.builtin.copy: dest: /etc/modules-load.d/k3s-cilium-longhorn.conf