harden: pin node timezone to UTC, and TEST the guard/value coupling that bit us
Two debt items, both about making a latent problem impossible rather than fixed. === 1. Node timezone pinned to Etc/UTC (node_baseline, `system` tag) === The prod nodes disagreed, measured from k3s's own etcd snapshot annotations: prod-1/2 at -05:00 (Central), prod-3 at -04:00 (Eastern). Functionally harmless — Kubernetes and etcd are UTC internally, and in UTC those snapshots are ~4 minutes apart, which is correct staggering — but it makes cross-node log correlation actively misleading, and this cluster has had two unplanned reboots in two days. `Etc/UTC` chosen on repo convention, not just principle: catalog/plex.yaml records "no other service in the repo sets a timezone ... so use the safe default" with TZ: Etc/UTC, and apps/wave-0/plex.yaml matches (platform/crafty is America/Chicago DELIBERATELY — a user-facing game schedule, not a log clock). Decisive extra fact: THE THREE DEV NODES ARE ALREADY Etc/UTC, so this makes prod match dev rather than inventing a convention. UTC also removes the class — no DST discontinuity, node time == platform time. New knob `node_baseline_timezone`, overridable per environment via group_vars (the same seam as node_baseline_backports_kernel) but deliberately NOT restated there: no environment needs to differ, and a second copy is a second thing to disagree. Guarded by an assert in the role's existing BLOCKED style before the module runs. `--check --diff --tags system -e k3s_hosts=k3s_prod` INDEPENDENTLY CONFIRMS the offset evidence, and is the whole verification — the play was never run for real: k3s-prod-1: -"America/Chicago" +"Etc/UTC" changed k3s-prod-2: -"America/Chicago" +"Etc/UTC" changed k3s-prod-3: -"America/New_York" +"Etc/UTC" changed The same --check against DEV reports the task `ok` on all three — idempotence demonstrated against live hosts, in check mode only. `--list-hosts` confirms `-e k3s_hosts=k3s_prod` selects the three prod nodes (the diagnostic plays take their OWN var; the wrong flag silently targets dev). *** AUTHORED, NOT APPLIED. *** Applying it is a live-cluster action and was deliberately left for the operator to schedule: prod's Longhorn had only just finished resyncing after a node reboot, and changing a node's clock config during storage recovery is not an unattended move. `ansible-inventory --list` was NOT used anywhere — it prints DECRYPTED vault values (the passphrase is mounted into the dev container) and has leaked the prod become-password to a transcript once already. === 2. New L0 check `placeholder-guard-coupling` (R-ENV-02, R-STATE-03) === On 2026-09-05 the PM changed a placeholder in clusters/base/cluster-vars.yaml from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID...`. The guards in bootstrap/standup.sh matched the glob `TBD-*` — a PREFIX. All three silently died, and `resume` would have enabled velero on prod against a nonexistent hostname: installs clean, reports healthy, no backups exist. Found by accident. A guard in one file keyed on the SHAPE of a value in another file, with nothing tying them. The check exercises the REAL predicate — it does not reimplement it. standup.sh is not sourceable (top-level argv parsing hard-exits, cd, export, EXIT trap), so the check extracts the predicate's own source lines and sources only those. Faithfulness is pinned by: discovery of every `*_is_placeholder` (not a hardcoded name), exactly-one-definition, "extracted block is a verbatim substring", a self-containment gate, a canary (must refuse "", must accept a real URL), and a scan for a divergent second copy. If the predicate grows a dependency the check FAILS LOUDLY and names the remedy rather than degrading to testing a copy — the failure mode it exists for. Both sides derive from real files: guarded knobs from a taint trace in standup.sh (`x="$(cvar KEY)"` -> alias hop -> `*_is_placeholder "$x"`); values from base cluster-vars plus each overlay's JSON6902 /data/KEY patches. The independent detector is deliberately NOT the guard: seed idioms UNION the guard's own words with glob anchors STRIPPED — so a prefix-anchored guard cannot pass, and deleting an arm cannot make a value stop being placeholder-shaped. Real tree: 32 assertions, 18 values through the extracted predicate, PASS. NON-VACUITY VERIFIED BY ME, twice, on throwaway copies (the real repo and the read-only spec repo untouched). My first perturbation tripped the self-containment rail instead of the anchoring property, so I redid it shape-preserving — substring globs -> prefix globs, nothing else — and it failed on exactly the right property, naming mid-string and tail acceptances with the diagnosis "This is the 2026-09-05 failure exactly". Control: unperturbed copy exits 0. === 3. ci/touched-requirements.sh: bootstrap/ was UNMAPPED (found by the agent) === The path->requirement map had no `bootstrap/*` entry, so a change touching standup.sh ALONGSIDE any already-mapped path scoped the L0 run to that other path's requirements and silently excluded the new check — the one that exists to catch exactly the 2026-09-05 class. A safety net with a hole where the incident was. Mapped to R-STATE-03 R-GITOPS-01; verified the map now resolves bootstrap paths. VERIFIED: full CI gate PASS. `run.sh --layer l0` 8/8 PASS including the new check. Coverage unchanged at 45/51 (the new check tags existing requirements). Refs R-ENV-02, R-STATE-03, R-TEST-01, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
da3e9cc3f3
commit
a984442cae
6 files changed
+680
-1
No files matched your search
@@ -183,6 +183,15 @@ path_requirements() {
|
||||
platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
|
||||
conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
|
||||
clusters/*) echo "R-GITOPS-01 R-ENV-02" ;;
|
||||
# bootstrap/ was UNMAPPED until 2026-09-06, and the omission was load-bearing:
|
||||
# bootstrap/standup.sh carries the placeholder guards that decide whether `velero` may be
|
||||
# enabled against a real backup endpoint. A change touching standup.sh ALONGSIDE any
|
||||
# already-mapped path scoped the L0 run to that other path's requirements and silently
|
||||
# excluded conformance/tests/l0-static/placeholder-guard-coupling — the check that exists
|
||||
# precisely to catch a guard/value mismatch. The 2026-09-05 guard bug (a prefix glob that
|
||||
# stopped matching when the value it guards changed, in a different file) is exactly the
|
||||
# class that omission would hide.
|
||||
bootstrap/*) echo "R-STATE-03 R-GITOPS-01" ;;
|
||||
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
|
||||
*) : ;;
|
||||
esac
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — a placeholder value in cluster-vars is REFUSED by the guard that owns it
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE BUG CLASS THIS CLOSES (it has now bitten three times in one file). A guard in
|
||||
# `bootstrap/standup.sh` keys on the SHAPE of a value that lives in
|
||||
# `clusters/base/cluster-vars.yaml`, and nothing tied the two together. On 2026-09-05 the
|
||||
# S3 placeholder changed from `TBD-PROD-S3-BACKUP-ENDPOINT` to
|
||||
# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`; the three guards matched the glob
|
||||
# `TBD-*`, a PREFIX match, and `fnmatch('https://TBD-R2-…','TBD-*')` is False. All three
|
||||
# guards silently stopped matching, and `standup.sh resume` would have resumed `velero` on
|
||||
# prod against a nonexistent hostname: installs clean, reports healthy, NO BACKUPS EXIST.
|
||||
# It was found by accident. The guard is fixed (`s3_is_placeholder`, substring); the
|
||||
# COUPLING was untested, which is what makes the class reintroducible.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# WHEN the REAL predicate is extracted from bootstrap/standup.sh and executed by bash
|
||||
# (never reimplemented — see check.sh "HOW THE REAL GUARD IS EXERCISED"),
|
||||
# THEN (1) every authored value of a guarded knob that is placeholder-shaped by an
|
||||
# INDEPENDENT detector is actually refused by that guard, in base and in every
|
||||
# environment overlay;
|
||||
# (2) no environment overlay carries a placeholder-shaped value for a knob no guard
|
||||
# refuses;
|
||||
# (3) every function in standup.sh that reads a guarded knob applies the guard to it
|
||||
# (the "three guards died at once" regression);
|
||||
# (4) the guard matches its own vocabulary ANYWHERE in a value, not just as a prefix
|
||||
# — the exact anchoring defect above — and still accepts a real endpoint;
|
||||
# AND (5) the extraction is faithful, and the check fails LOUDLY rather than silently
|
||||
# testing a copy if the guard stops being a self-contained predicate.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
|
||||
# PyYAML + bash); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test
|
||||
# wraps the SAME script.
|
||||
#
|
||||
# NOT proven: that the endpoint is REACHABLE or that backups exist (runtime facts —
|
||||
# `standup.sh s3check` probes the former from a pod, l3-fault-injection/dr-restore proves
|
||||
# the latter); that guards for knobs OTHER than the placeholder-refusing ones are correct;
|
||||
# and that the guard runs at all in a real standup (the dispatch path is shell, executed
|
||||
# only against a live cluster). R-STATE-03 is tagged for the narrow static slice only: the
|
||||
# authored backup target cannot be a placeholder that the guard fails to recognise.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: placeholder-guard-coupling
|
||||
labels:
|
||||
conformance.home.k3s/R-ENV-02: "true"
|
||||
conformance.home.k3s/R-STATE-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every placeholder-shaped value authored in cluster-vars, for any environment
|
||||
overlay, is actually recognised as a placeholder by the real bootstrap guard that
|
||||
is supposed to refuse it — so a value change in one file cannot silently disarm a
|
||||
guard in another and enable velero against an endpoint that does not exist.
|
||||
spec:
|
||||
steps:
|
||||
- name: couple-authored-placeholders-to-the-real-guard
|
||||
description: >-
|
||||
Extract the real placeholder predicate from bootstrap/standup.sh, run it against
|
||||
every authored cluster-vars value the repo actually carries, and assert the guard
|
||||
refuses each one an independent detector calls a placeholder.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,516 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — placeholder-guard-coupling
|
||||
# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)."
|
||||
# — spec docs/02-environments-and-provisioning.md)
|
||||
# R-STATE-03 ("Scheduled off-cluster backups + restore (DR)."
|
||||
# — spec docs/04-storage-and-state.md)
|
||||
#
|
||||
# WHY THIS EXISTS — A GUARD KEYED ON A VALUE THAT LIVES IN ANOTHER FILE
|
||||
# --------------------------------------------------------------------
|
||||
# `bootstrap/standup.sh` refuses to resume tiers whose configuration is still a
|
||||
# placeholder. Three of those guards keyed on the SHAPE of `S3_BACKUP_ENDPOINT`, which is
|
||||
# authored in `clusters/base/cluster-vars.yaml`. On 2026-09-05 the placeholder changed
|
||||
# from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`
|
||||
# (the MinIO -> Cloudflare R2 move). The guards matched the glob `TBD-*` — a PREFIX match —
|
||||
# and `fnmatch('https://TBD-R2-…','TBD-*')` is FALSE. All three guards silently stopped
|
||||
# matching. `standup.sh resume` would then have resumed `velero` on prod against a hostname
|
||||
# that does not exist: the HelmRelease installs clean, the Kustomization reports Ready, and
|
||||
# NO BACKUPS EXIST. The only reason it was not shipped is that someone noticed by accident.
|
||||
#
|
||||
# The predicate is fixed (`s3_is_placeholder`, substring match). The COUPLING is what was
|
||||
# never tested: a guard in one file keying on a value in another, with nothing tying them
|
||||
# together. This check is that tie, and it is derived from BOTH real files — no current
|
||||
# string is hardcoded here, so it does not rot when the placeholder or the guard changes.
|
||||
#
|
||||
# HOW THE REAL GUARD IS EXERCISED (and why not a copy)
|
||||
# ---------------------------------------------------
|
||||
# `bootstrap/standup.sh` is NOT sourceable: at top level it parses argv, hard-exits when
|
||||
# --kubeconfig/--expect-server/<step> are absent, `cd`s to the repo root, exports
|
||||
# KUBECONFIG, installs an EXIT trap and finally dispatches steps against a live cluster.
|
||||
# So this check EXTRACTS the predicate's own source lines from the file and sources ONLY
|
||||
# those, then calls the real function. That is the real bytes, not a reimplementation:
|
||||
# * every function whose name ends `_is_placeholder` is discovered, not named here;
|
||||
# * a duplicate definition of one is a failure (which of the two would run?);
|
||||
# * the extracted body must be a SELF-CONTAINED predicate — `case`/`return` and nothing
|
||||
# else, no command substitution, no call to another function. If the guard ever grows
|
||||
# a dependency, extraction stops being faithful and THIS CHECK FAILS LOUDLY, naming the
|
||||
# remedy (move the predicate into a small sourceable library that standup.sh sources)
|
||||
# rather than quietly falling back to testing a copy of the logic;
|
||||
# * a CANARY pins the extraction itself: the extracted function must refuse the empty
|
||||
# string and accept a realistic endpoint. An extraction that grabbed the wrong lines,
|
||||
# or a predicate degraded to always-true / always-false, fails there — this repo has
|
||||
# already been bitten by a check that passed when perturbed because it asked a question
|
||||
# that was always satisfiable.
|
||||
# There is deliberately NO second implementation of the matching logic anywhere in this file.
|
||||
#
|
||||
# THE INDEPENDENT DETECTOR (the other half of the comparison)
|
||||
# ----------------------------------------------------------
|
||||
# Asking "does the guard agree with the guard?" proves nothing, so the set of
|
||||
# placeholder-shaped values is decided by a detector written HERE, whose vocabulary is the
|
||||
# UNION of two independent sources:
|
||||
# * SEED — universal loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER, REPLACE-ME,
|
||||
# CHANGE-ME, FILL-ME, SET-ME, XXXX), plus `<angle-bracket>` templates and the empty
|
||||
# string. These are NOT derived from the guard, so DELETING a guard arm cannot make a
|
||||
# value stop being placeholder-shaped — it makes assertion (1) fail, which is the point.
|
||||
# A value using an idiom the guard does not know is a finding either way: fix it by
|
||||
# giving the knob a real value, or by teaching the guard the idiom.
|
||||
# * the guard's OWN vocabulary, read out of its case-arms with the glob anchors STRIPPED.
|
||||
# Stripping the anchors is what makes the comparison non-circular in the dimension that
|
||||
# actually broke: the detector matches a word ANYWHERE, case-insensitively, and the
|
||||
# guard must agree. A prefix-anchored guard cannot pass.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# WHEN the real predicate(s) are extracted from bootstrap/standup.sh and executed,
|
||||
# THEN (1) VALUES: for every knob a guard is responsible for, every authored value — the
|
||||
# base default AND each environment overlay's value — that the independent
|
||||
# detector calls placeholder-shaped is REFUSED by the real guard, and every value
|
||||
# it calls usable is ACCEPTED (a guard that refuses a real endpoint leaves velero
|
||||
# suspended forever, which is the same class of silent failure);
|
||||
# (2) OVERLAYS: no environment overlay's EFFECTIVE cluster-vars carries a
|
||||
# placeholder-shaped value for a knob that no guard refuses. (Base alone is
|
||||
# exempt: it is not a bootstrap target — `flux bootstrap --path=clusters/<env>` —
|
||||
# and its loud sentinels exist precisely to be overridden. A base placeholder an
|
||||
# overlay forgets to override IS caught, because it becomes that overlay's
|
||||
# effective value.)
|
||||
# (3) CONSUMERS: every function in standup.sh that reads a guarded knob (via `cvar`,
|
||||
# following one hop of variable aliasing) applies a placeholder guard to it. This
|
||||
# is the "all three guards died at once" regression stated as a property: the
|
||||
# guard being correct is worthless if a consumer stops calling it.
|
||||
# (4) ANCHORING: for each word in the guard's own vocabulary, the guard refuses a
|
||||
# value carrying that word in the MIDDLE (not at the start) — the precise defect
|
||||
# of 2026-09-05 — while still accepting a realistic endpoint.
|
||||
# (5) HARNESS: exactly one definition per predicate, the extracted body is verbatim
|
||||
# from the file and self-contained, the canary holds, no divergent second copy of
|
||||
# the matching logic exists elsewhere in standup.sh, and a floor on the number of
|
||||
# assertions run keeps a pass from degenerating into a no-op.
|
||||
# AND (6) LOAD-BEARING: ci/l0.sh runs the L0 conformance layer on every gate run and
|
||||
# ci/gate.sh sets the merge block when that stage fails.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps)
|
||||
# - Nothing about RUNTIME: not that the endpoint resolves, not that credentials work, not
|
||||
# that a backup exists. `standup.sh s3check` probes reachability from a pod (operator-
|
||||
# run, cluster-gated) and conformance/tests/l3-fault-injection/dr-restore proves restore.
|
||||
# - It does not execute standup.sh's dispatch path, so it cannot prove the guard is
|
||||
# REACHED in a real run — only that, when reached, it refuses what it must. The
|
||||
# consumer property (3) is a static approximation of reachability: it is a text-level
|
||||
# taint trace over one file, not an interpreter, and a guard whose result is computed
|
||||
# and then ignored would satisfy it.
|
||||
# - It says nothing about guards keyed on shapes OTHER than placeholders (e.g. the
|
||||
# EXTERNALDNS_TXT_OWNER_ID == 'default' test), nor about placeholder-shaped values
|
||||
# outside cluster-vars (the Ansible inventory's BLOCKED/NOT-A-DEVICE sentinels are
|
||||
# asserted by the node_baseline role itself, at play time).
|
||||
# - Stage 3b of ci/l0.sh scopes the conformance run to the TOUCHED requirements. An edit
|
||||
# to bootstrap/* attributes NO requirement in ci/touched-requirements.sh's path map, so
|
||||
# the touched set is empty and the FULL L0 layer runs (verified: ci/lib.sh
|
||||
# load_touched_requirements) — this check runs. An edit to clusters/* maps to
|
||||
# R-ENV-02, which this check carries, so a cluster-vars value change also selects it.
|
||||
# A change that touches standup.sh ALONGSIDE some unrelated mapped path (say
|
||||
# platform/cilium/) would scope the run away from this check; closing that needs a
|
||||
# `bootstrap/*` entry in the path map, which is not this check's file to edit.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
STANDUP_REL="${STANDUP_REL:-bootstrap/standup.sh}"
|
||||
|
||||
cd "${ROOT}"
|
||||
|
||||
if [ ! -f "${ROOT}/${STANDUP_REL}" ]; then
|
||||
echo "check SKIPPED (placeholder-guard-coupling): ${STANDUP_REL} is absent — nothing to couple." >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "${WORK}"' EXIT
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# PHASE 1 (python): extract the real predicate(s), derive the corpus from the real files,
|
||||
# and run every assertion that does not need a shell. Writes:
|
||||
# ${WORK}/guards.sh the predicate source lines, verbatim, and nothing else
|
||||
# ${WORK}/corpus.tsv <guard>\t<expect: placeholder|usable>\t<origin>\t<value>
|
||||
# ${WORK}/findings.txt one line per failed assertion
|
||||
# ${WORK}/counts.txt <assertions-run>
|
||||
# ---------------------------------------------------------------------------------------
|
||||
"${PYTHON}" - "${ROOT}" "${WORK}" "${STANDUP_REL}" <<'PY'
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT, WORK, STANDUP_REL = Path(sys.argv[1]), Path(sys.argv[2]), sys.argv[3]
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def bail(msg):
|
||||
"""A failure that makes the rest of the check meaningless: report and stop."""
|
||||
print(f"CHECK FAIL (placeholder-guard-coupling): {msg}", file=sys.stderr)
|
||||
(WORK / "findings.txt").write_text(msg + "\n")
|
||||
(WORK / "counts.txt").write_text("0\n")
|
||||
(WORK / "corpus.tsv").write_text("")
|
||||
(WORK / "guards.sh").write_text("")
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
TEXT = (ROOT / STANDUP_REL).read_text()
|
||||
LINES = TEXT.splitlines()
|
||||
|
||||
# ------------------------------------------------------------------ (5) extract the guard
|
||||
DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{")
|
||||
defs: dict[str, list[int]] = {}
|
||||
for i, line in enumerate(LINES):
|
||||
m = DEF_RE.match(line)
|
||||
if m:
|
||||
defs.setdefault(m.group(1), []).append(i)
|
||||
|
||||
if not defs:
|
||||
bail(f"{STANDUP_REL} defines no *_is_placeholder predicate. Either the guard was renamed "
|
||||
"(update this check's discovery pattern in the SAME change) or it was DELETED, which "
|
||||
"is the 2026-09-05 failure with the guard removed instead of disarmed.")
|
||||
|
||||
# An "argument-shaped" call: <name> "<something>"
|
||||
CALL_RE = re.compile(r"\b([a-z0-9_]+_is_placeholder)\s+\"([^\"]*)\"")
|
||||
|
||||
guard_src: list[str] = []
|
||||
guard_words: dict[str, list[str]] = {}
|
||||
ARM_RE = re.compile(r"^(?P<pats>[^()]*?)\)\s*return\s+[01]\s*;;$")
|
||||
ALLOWED = (
|
||||
re.compile(r"^$"),
|
||||
re.compile(r"^#"),
|
||||
re.compile(r"^case\s+\"\$\{?[0-9A-Za-z_]+\}?\"\s+in$"),
|
||||
re.compile(r"^esac$"),
|
||||
re.compile(r"^return\s+[01]$"),
|
||||
ARM_RE,
|
||||
)
|
||||
|
||||
for name, at in sorted(defs.items()):
|
||||
check(len(at) == 1,
|
||||
f"{STANDUP_REL} defines {name}() {len(at)} times (lines "
|
||||
f"{', '.join(str(a + 1) for a in at)}) — which one runs depends on file order, and "
|
||||
"this check can no longer say which one it extracted. Keep exactly one definition.")
|
||||
if len(at) != 1:
|
||||
continue
|
||||
start = at[0]
|
||||
end = next((j for j in range(start + 1, len(LINES)) if LINES[j] == "}"), None)
|
||||
if end is None:
|
||||
bail(f"{name}() at {STANDUP_REL}:{start + 1} has no closing brace at column 0 — the "
|
||||
"extraction boundary this check relies on is gone. Restore the conventional "
|
||||
"layout, or move the predicate into a sourceable library and point this check at it.")
|
||||
body = LINES[start:end + 1]
|
||||
block = "\n".join(body)
|
||||
check(block in TEXT,
|
||||
f"extracted {name}() is not a verbatim substring of {STANDUP_REL} — extraction bug; "
|
||||
"the harness would be testing something the script does not contain")
|
||||
|
||||
words: list[str] = []
|
||||
for raw in body[1:-1]:
|
||||
s = raw.strip()
|
||||
if not any(rx.match(s) for rx in ALLOWED) or "$(" in s or "`" in s:
|
||||
bail(f"{name}() at {STANDUP_REL}:{start + 1} is no longer a SELF-CONTAINED "
|
||||
f"case/return predicate (offending line: {s!r}). This check exercises the REAL "
|
||||
"function by sourcing its own lines; a body that calls out to anything else "
|
||||
"cannot be extracted faithfully, and silently testing a COPY of the logic is "
|
||||
"exactly the failure mode this check exists to prevent. REMEDY: move the "
|
||||
"predicate into a tiny library that standup.sh sources (e.g. "
|
||||
"bootstrap/lib/placeholder.sh) and point STANDUP_REL/this extractor at it — do "
|
||||
"NOT reimplement the matching here.")
|
||||
m = ARM_RE.match(s)
|
||||
if not m:
|
||||
continue
|
||||
for pat in m.group("pats").split("|"):
|
||||
pat = pat.strip()
|
||||
if pat in ('""', "''", ""):
|
||||
continue # the empty-value arm: covered by the canary
|
||||
words.append(pat.strip("'\"").strip("*"))
|
||||
check(len(words) >= 1,
|
||||
f"{name}() has no non-empty placeholder pattern left — every arm that recognises a "
|
||||
"loud placeholder string has been removed, so the guard now only refuses the empty "
|
||||
"value and any 'TBD'-style endpoint would sail straight through it")
|
||||
guard_words[name] = words
|
||||
guard_src.append(block)
|
||||
|
||||
(WORK / "guards.sh").write_text("\n".join(guard_src) + "\n")
|
||||
|
||||
# ------------------------------------------------- no divergent second copy of the logic
|
||||
# Any of the guard's own vocabulary appearing in EXECUTABLE text outside the predicate means
|
||||
# a second, independently-drifting implementation of the same test (which is how three
|
||||
# guards came to disagree with one value in the first place).
|
||||
guard_line_nums = {j for name, at in defs.items() if len(at) == 1
|
||||
for j in range(at[0], next(k for k in range(at[0] + 1, len(LINES))
|
||||
if LINES[k] == "}") + 1)}
|
||||
all_words = sorted({w.upper() for ws in guard_words.values() for w in ws if w})
|
||||
divergent: list[str] = []
|
||||
for j, line in enumerate(LINES):
|
||||
if j in guard_line_nums or line.strip().startswith("#"):
|
||||
continue
|
||||
code = line.split("#", 1)[0]
|
||||
hit = [w for w in all_words if w in code.upper()]
|
||||
if hit:
|
||||
divergent.append(f"{STANDUP_REL}:{j + 1} ({', '.join(hit)}): {line.strip()!r}")
|
||||
check(not divergent,
|
||||
"the guard's own placeholder vocabulary appears in EXECUTABLE text outside the "
|
||||
f"predicate — a SECOND implementation of the same test, which is precisely how a value "
|
||||
f"change disarmed three guards at once. Call the predicate instead. Sites: "
|
||||
+ " | ".join(divergent))
|
||||
|
||||
# ------------------------------------------------------------- cluster-vars, as authored
|
||||
CV = ROOT / "clusters" / "base" / "cluster-vars.yaml"
|
||||
if not CV.is_file():
|
||||
bail("clusters/base/cluster-vars.yaml is absent — the authored half of the coupling is "
|
||||
"missing and this check cannot mean anything")
|
||||
|
||||
base: dict[str, str] = {}
|
||||
for d in yaml.safe_load_all(CV.read_text()):
|
||||
if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
|
||||
and (d.get("metadata") or {}).get("name") == "cluster-vars":
|
||||
base = {str(k): "" if v is None else str(v) for k, v in (d.get("data") or {}).items()}
|
||||
check(len(base) > 5, f"clusters/base/cluster-vars.yaml yielded {len(base)} knob(s) — the "
|
||||
"ConfigMap parse failed, so every value assertion below would be vacuous")
|
||||
|
||||
overlays: dict[str, dict[str, str]] = {}
|
||||
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
||||
env = overlay.parent.name
|
||||
if env == "base":
|
||||
continue
|
||||
eff = dict(base)
|
||||
try:
|
||||
k = yaml.safe_load(overlay.read_text()) or {}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
check(False, f"clusters/{env}/kustomization.yaml does not parse as YAML ({exc}) — this "
|
||||
"check cannot compute that overlay's effective values")
|
||||
continue
|
||||
for p in (k.get("patches") or []):
|
||||
tgt = p.get("target") or {}
|
||||
if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
|
||||
continue
|
||||
try:
|
||||
ops = yaml.safe_load(p.get("patch") or "") or []
|
||||
except Exception: # noqa: BLE001
|
||||
continue
|
||||
for op in ops if isinstance(ops, list) else []:
|
||||
if not isinstance(op, dict):
|
||||
continue
|
||||
m = re.match(r"^/data/([A-Za-z0-9_]+)$", str(op.get("path", "")))
|
||||
if m and str(op.get("op", "")) in ("replace", "add"):
|
||||
eff[m.group(1)] = "" if op.get("value") is None else str(op["value"])
|
||||
overlays[env] = eff
|
||||
check(len(overlays) >= 1,
|
||||
"no clusters/<env>/kustomization.yaml overlay was parsed — assertion (2) would be vacuous")
|
||||
|
||||
# ------------------------------------------------ which knobs is a guard responsible for?
|
||||
# Function bodies of standup.sh, then a one-hop taint trace from `x="$(cvar KEY)"` to a
|
||||
# `*_is_placeholder "${x}"` call. Both halves are read out of the file; nothing is named here.
|
||||
FUNC_RE = re.compile(r"^([a-z0-9_]+)\(\)\s*\{")
|
||||
funcs: dict[str, list[str]] = {}
|
||||
i = 0
|
||||
while i < len(LINES):
|
||||
m = FUNC_RE.match(LINES[i])
|
||||
if m:
|
||||
end = next((j for j in range(i + 1, len(LINES)) if LINES[j] == "}"), len(LINES) - 1)
|
||||
funcs[m.group(1)] = LINES[i + 1:end]
|
||||
i = end + 1
|
||||
continue
|
||||
i += 1
|
||||
check(len(funcs) > 5, f"parsed only {len(funcs)} function(s) out of {STANDUP_REL} — the "
|
||||
"consumer assertion (3) would be vacuous")
|
||||
|
||||
CVAR_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\(\s*cvar\s+([A-Z0-9_]+)\s*\)\"")
|
||||
ALIAS_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\{([A-Za-z_][A-Za-z0-9_]*)[:}]")
|
||||
|
||||
guarded_keys: set[str] = set()
|
||||
consumers: list[tuple[str, str, bool]] = [] # (func, key, guarded-in-this-func)
|
||||
for fname, body in funcs.items():
|
||||
text = "\n".join(body)
|
||||
keys_of_var: dict[str, set[str]] = {}
|
||||
for m in CVAR_RE.finditer(text):
|
||||
keys_of_var.setdefault(m.group(1), set()).add(m.group(2))
|
||||
# one hop of aliasing: ep="${live}" carries live's taint
|
||||
for _ in range(3):
|
||||
for m in ALIAS_RE.finditer(text):
|
||||
src = keys_of_var.get(m.group(2))
|
||||
if src:
|
||||
keys_of_var.setdefault(m.group(1), set()).update(src)
|
||||
called_on = {m.group(2) for m in CALL_RE.finditer(text)}
|
||||
called_vars = {re.sub(r"^\$\{?|\}?$", "", a) for a in called_on}
|
||||
# Aggregate per (function, knob): the knob is guarded HERE if ANY variable carrying its
|
||||
# value is passed to a predicate — `live="$(cvar K)"; ep="${live}"; guard "${ep}"` counts.
|
||||
guarded_here: dict[str, bool] = {}
|
||||
for var, keys in keys_of_var.items():
|
||||
for key in keys:
|
||||
guarded_here[key] = guarded_here.get(key, False) or (var in called_vars)
|
||||
for key, g in guarded_here.items():
|
||||
consumers.append((fname, key, g))
|
||||
if g:
|
||||
guarded_keys.add(key)
|
||||
|
||||
if not guarded_keys:
|
||||
bail(f"no cluster-vars knob in {STANDUP_REL} is passed to a *_is_placeholder predicate. "
|
||||
"Either the guards were removed, or the way the script reads cluster-vars changed and "
|
||||
"this check's derivation must be updated in the SAME change — it must never silently "
|
||||
"verify an empty set.")
|
||||
|
||||
# (3) CONSUMERS: every reader of a guarded knob must apply the guard to it.
|
||||
for fname, key, guarded in sorted(consumers):
|
||||
if key not in guarded_keys:
|
||||
continue # nothing claims responsibility for this knob at all
|
||||
check(guarded,
|
||||
f"{STANDUP_REL}: {fname}() reads cluster-vars knob {key} but never passes it to a "
|
||||
"*_is_placeholder guard, while other functions do. On 2026-09-05 all three consumers "
|
||||
"of S3_BACKUP_ENDPOINT were disarmed at once; a consumer that reads the knob and skips "
|
||||
"the guard is that failure in one function.")
|
||||
|
||||
# ---------------------------------------------------------------- independent detector
|
||||
# Deliberately NOT derived from the guard alone (see the header): SEED is universal
|
||||
# loud-placeholder vocabulary, and the guard's own words are added so that a guard whose
|
||||
# ANCHORING is wrong still has its vocabulary tested against real values.
|
||||
SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", "REPLACE-ME", "REPLACEME",
|
||||
"CHANGE-ME", "CHANGEME", "FILL-ME", "FILLME", "SET-ME", "XXXX"]
|
||||
VOCAB = sorted({w.upper() for w in SEED} | set(all_words))
|
||||
ANGLE = re.compile(r"<[^>\s]+>")
|
||||
|
||||
|
||||
def placeholder_shaped(value: str) -> bool:
|
||||
if value.strip() == "":
|
||||
return True # an unset endpoint is not a target
|
||||
up = value.upper()
|
||||
return any(w in up for w in VOCAB) or bool(ANGLE.search(value))
|
||||
|
||||
|
||||
rows: list[tuple[str, str, str, str]] = [] # guard, expect, origin, value
|
||||
GUARD = sorted(guard_words)[0] if len(guard_words) == 1 else None
|
||||
|
||||
# (1) VALUES: base default and every overlay's value, for every guarded knob.
|
||||
for key in sorted(guarded_keys):
|
||||
seen: list[tuple[str, str]] = []
|
||||
if key in base:
|
||||
seen.append((f"clusters/base/cluster-vars.yaml:{key}", base[key]))
|
||||
for env, eff in sorted(overlays.items()):
|
||||
seen.append((f"clusters/{env} effective:{key}", eff.get(key, "")))
|
||||
check(bool(seen), f"guarded knob {key} is not defined in cluster-vars at all — the guard "
|
||||
"protects a knob nothing authors")
|
||||
for origin, value in seen:
|
||||
expect = "placeholder" if placeholder_shaped(value) else "usable"
|
||||
for name in sorted(guard_words):
|
||||
rows.append((name, expect, origin, value))
|
||||
|
||||
# (2) OVERLAYS: a placeholder-shaped effective value for an UNGUARDED knob.
|
||||
for env, eff in sorted(overlays.items()):
|
||||
for key, value in sorted(eff.items()):
|
||||
if key in guarded_keys or not placeholder_shaped(value):
|
||||
continue
|
||||
check(False,
|
||||
f"clusters/{env}: knob {key} resolves to the placeholder-shaped value {value!r}, "
|
||||
f"and NO guard in {STANDUP_REL} refuses it. Either give it a real value before it "
|
||||
"reaches an overlay, or guard it the way the S3 endpoint is guarded — an overlay "
|
||||
"that ships a placeholder no guard recognises is the 2026-09-05 failure with a "
|
||||
"different knob's name on it.")
|
||||
|
||||
# (4) ANCHORING: the guard's own vocabulary must match ANYWHERE, not just as a prefix,
|
||||
# and a realistic endpoint must still be accepted.
|
||||
for name, words in sorted(guard_words.items()):
|
||||
for w in sorted(set(words)):
|
||||
rows.append((name, "placeholder",
|
||||
f"anchoring:{name}:mid-string({w})",
|
||||
f"https://account-{w}-id.r2.cloudflarestorage.com"))
|
||||
rows.append((name, "placeholder", f"anchoring:{name}:tail({w})",
|
||||
f"https://r2-endpoint.example.invalid/{w}"))
|
||||
rows.append((name, "placeholder", f"canary:{name}:empty", ""))
|
||||
rows.append((name, "usable", f"canary:{name}:real-r2",
|
||||
"https://0123456789abcdef0123456789abcd.r2.cloudflarestorage.com"))
|
||||
rows.append((name, "usable", f"canary:{name}:real-minio", "http://192.168.1.9:9000"))
|
||||
|
||||
# Every guarded knob must reach the oracle once per authored source (base + each overlay),
|
||||
# so a knob silently dropping out of the corpus is a failure rather than a quiet pass.
|
||||
authored = [r for r in rows if not r[2].startswith(("anchoring:", "canary:"))]
|
||||
expected_rows = len(guarded_keys) * (1 + len(overlays)) * len(guard_words)
|
||||
check(len(authored) == expected_rows,
|
||||
f"{len(authored)} authored value(s) reached the guard, expected {expected_rows} "
|
||||
f"({len(guarded_keys)} guarded knob(s) x base+{len(overlays)} overlay(s) x "
|
||||
f"{len(guard_words)} predicate(s)) — a guarded knob is missing from cluster-vars or from "
|
||||
"an overlay's effective values, and the coupling is only partly exercised")
|
||||
|
||||
# ------------------------------------------------------- (6) the check is load-bearing
|
||||
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
||||
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
||||
check("--layer l0" in L0,
|
||||
"ci/l0.sh does not run the L0 conformance layer — this check would never execute in the gate")
|
||||
check(bool(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)) and bool(re.search(r"fail\).*BLOCK=1", GATE)),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
# --------------------------------------------------------------------------- handoff
|
||||
with (WORK / "corpus.tsv").open("w") as fh:
|
||||
for name, expect, origin, value in rows:
|
||||
fh.write(f"{name}\t{expect}\t{origin}\t{value}\n")
|
||||
(WORK / "findings.txt").write_text("".join(f"{f}\n" for f in FAILURES))
|
||||
(WORK / "counts.txt").write_text(f"{CHECKS}\n")
|
||||
|
||||
print(f"guards: {', '.join(sorted(guard_words))} "
|
||||
f"vocabulary: {', '.join(VOCAB)}")
|
||||
print(f"guarded knobs: {', '.join(sorted(guarded_keys))} "
|
||||
f"consumers traced: {len(consumers)} overlays: {', '.join(sorted(overlays))}")
|
||||
for name, expect, origin, value in rows:
|
||||
if not origin.startswith(("anchoring:", "canary:")):
|
||||
print(f" authored {origin} = {value!r} -> expect {expect}")
|
||||
PY
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# PHASE 2 (bash): source the REAL predicate lines and run the corpus through them.
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# shellcheck disable=SC1090,SC1091
|
||||
source "${WORK}/guards.sh"
|
||||
|
||||
oracle_failures=0
|
||||
oracle_rows=0
|
||||
while IFS=$'\t' read -r guard expect origin value; do
|
||||
[ -n "${guard}" ] || continue
|
||||
oracle_rows=$((oracle_rows + 1))
|
||||
if "${guard}" "${value}"; then verdict=placeholder; else verdict=usable; fi
|
||||
if [ "${verdict}" != "${expect}" ]; then
|
||||
oracle_failures=$((oracle_failures + 1))
|
||||
if [ "${expect}" = placeholder ]; then
|
||||
printf '%s\n' "${guard}() in ${STANDUP_REL} ACCEPTS '${value}' (from ${origin}) as a usable value, but it is placeholder-shaped. This is the 2026-09-05 failure exactly: the guard is asked about a value it does not recognise, returns 'fine', and the tier it protects is enabled against something that does not exist. Fix the predicate (match the idiom anywhere, not as a prefix) or stop authoring that idiom." >> "${WORK}/findings.txt"
|
||||
else
|
||||
printf '%s\n' "${guard}() in ${STANDUP_REL} REFUSES '${value}' (from ${origin}), which is a usable value. An over-matching guard leaves the tier it protects suspended forever and reports it as an operator decision — a silent failure in the other direction." >> "${WORK}/findings.txt"
|
||||
fi
|
||||
fi
|
||||
done < "${WORK}/corpus.tsv"
|
||||
|
||||
# ------------------------------------------------------------------------------ verdict
|
||||
py_checks="$(cat "${WORK}/counts.txt")"
|
||||
total=$((py_checks + oracle_rows))
|
||||
findings="$(cat "${WORK}/findings.txt")"
|
||||
|
||||
# NO-OP GUARD: this repo has been bitten by a check that passed when perturbed because it
|
||||
# asked a question that was always satisfiable. The floors below are the minimum evidence
|
||||
# for the verdict to mean anything, not a style rule.
|
||||
if [ "${oracle_rows}" -lt 6 ]; then
|
||||
findings="${findings}only ${oracle_rows} value(s) were put to the real guard — the coupling is not actually being exercised
|
||||
"
|
||||
fi
|
||||
if [ "${total}" -lt 20 ]; then
|
||||
findings="${findings}only ${total} assertions ran — the check degenerated to a near-no-op and its pass would be meaningless
|
||||
"
|
||||
fi
|
||||
|
||||
if [ -n "${findings}" ]; then
|
||||
echo "CHECK FAIL (placeholder-guard-coupling): $(printf '%s' "${findings}" | grep -c . ) of ${total} assertions failed" >&2
|
||||
printf '%s' "${findings}" | sed 's/^/ - /' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "check OK: placeholder-guard-coupling — ${total} assertions; ${oracle_rows} value(s) run through the REAL extracted predicate(s) from ${STANDUP_REL} (no reimplementation), covering every authored cluster-vars value of every guarded knob in base and each overlay, the guard's own vocabulary matched mid-string, and the empty/real-endpoint canaries. Static slice only: refusal, not reachability — see the header."
|
||||
@@ -2,7 +2,8 @@
|
||||
# Ansible Galaxy collections required by infra/ansible. Install with:
|
||||
# ansible-galaxy collection install -r collections/requirements.yml
|
||||
collections:
|
||||
# node_baseline role: community.general.modprobe, community.general.filesystem
|
||||
# node_baseline role: community.general.modprobe, community.general.filesystem,
|
||||
# community.general.timezone
|
||||
- name: community.general
|
||||
# node_baseline role: ansible.posix.mount
|
||||
- name: ansible.posix
|
||||
@@ -81,6 +81,45 @@ node_baseline_kernel_modules:
|
||||
- dm_crypt
|
||||
- nfs
|
||||
|
||||
# --- NODE CLOCK / TIMEZONE (tag `system`) ---------------------------------------------
|
||||
# EVERY NODE IS UTC. Measured drift that made this a knob rather than an assumption: the
|
||||
# three prod nodes disagreed on their local zone (read off k3s's own etcd snapshot
|
||||
# annotations, 2026-09-05) —
|
||||
# k3s-prod-1 2026-09-05T16:31:28-05:00 (Central)
|
||||
# k3s-prod-2 2026-09-05T16:29:07-05:00 (Central)
|
||||
# k3s-prod-3 2026-09-05T17:27:35-04:00 (Eastern)
|
||||
# In UTC those three are ~4 minutes apart, which is the CORRECT staggering: nothing was
|
||||
# functionally wrong, because Kubernetes, etcd and the container runtime are UTC internally.
|
||||
# What IS wrong is that any timestamp a node renders in LOCAL time — journald's default
|
||||
# output, `ls -l`, a shell prompt, an etcd snapshot filename — cannot be compared across
|
||||
# nodes without knowing each node's zone. Correlating three nodes' logs during an incident
|
||||
# is exactly when nobody has that knowledge to hand, and this cluster took two unplanned
|
||||
# node reboots in the 24h before this was written.
|
||||
#
|
||||
# WHY UTC AND NOT "STANDARDISE ON CENTRAL". Picking one local zone leaves the class of
|
||||
# problem intact (a fourth node, or a DST boundary, re-creates it) and adds a translation
|
||||
# step to every log read, because every timestamp the PLATFORM emits — Kubernetes events,
|
||||
# Prometheus, Loki, Velero, CNPG WAL, the etcd snapshot annotations above — is already UTC.
|
||||
# UTC removes the class: node-local time and platform time become the same clock, and there
|
||||
# is no DST discontinuity in which a repeated or skipped hour breaks log ordering. This also
|
||||
# matches the only timezone convention the repo already had — catalog/plex.yaml records
|
||||
# "no other service in the repo sets a timezone ... so use the safe default" and both it and
|
||||
# apps/wave-0/plex.yaml set TZ=Etc/UTC. (platform/crafty sets America/Chicago, deliberately:
|
||||
# that is a USER-FACING game server's schedule, not a node's log clock.)
|
||||
#
|
||||
# WHY A KNOB AT ALL (R-ENV-02). The value is a node-level Ansible fact, so its override seam
|
||||
# is inventory/group_vars/<env>, NOT clusters/base/cluster-vars.yaml (a Flux postBuild
|
||||
# ConfigMap no Ansible code reads) — the same seam node_baseline_backports_kernel and
|
||||
# node_baseline_dns_servers use. NO ENVIRONMENT OVERRIDES IT TODAY, and one should not have
|
||||
# to: dev and prod both want UTC and there is nothing environment-specific about a node
|
||||
# clock. It is a variable rather than a literal in the task so a genuinely
|
||||
# operator-facing one-off host (never a k3s node) can differ without forking the role.
|
||||
#
|
||||
# Must be a zoneinfo name present on the node (`timedatectl list-timezones`); the assert in
|
||||
# tasks/main.yml rejects an empty or obviously-malformed value before the module runs, and
|
||||
# community.general.timezone itself refuses a name /usr/share/zoneinfo does not have.
|
||||
node_baseline_timezone: Etc/UTC
|
||||
|
||||
# --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
|
||||
# The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
|
||||
# a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:
|
||||
|
||||
@@ -262,6 +262,52 @@
|
||||
vars:
|
||||
newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
|
||||
|
||||
# =====================================================================================
|
||||
# NODE CLOCK — tag `system`: every node renders time in the SAME zone, and that zone is UTC
|
||||
# =====================================================================================
|
||||
# MEASURED DRIFT (2026-09-05), read off k3s's own etcd snapshot annotations on live prod:
|
||||
# k3s-prod-1 2026-09-05T16:31:28-05:00 Central
|
||||
# k3s-prod-2 2026-09-05T16:29:07-05:00 Central
|
||||
# k3s-prod-3 2026-09-05T17:27:35-04:00 Eastern <-- a different UTC offset
|
||||
# Functionally harmless (in UTC those three are ~4 minutes apart, which is the correct
|
||||
# staggering — Kubernetes, etcd and containerd are UTC internally). Operationally it is a
|
||||
# trap: any timestamp a node renders in LOCAL time (journald's default output, `ls -l`, an
|
||||
# etcd snapshot filename) is not comparable across nodes without knowing each node's zone,
|
||||
# and cross-node log correlation during an incident is precisely when nobody knows that.
|
||||
#
|
||||
# Rationale for UTC rather than standardising on one local zone, and for this being a knob
|
||||
# whose override seam is inventory/group_vars/<env>: see node_baseline_timezone in
|
||||
# defaults/main.yml. NOT a cluster-vars knob — no Ansible code reads that ConfigMap.
|
||||
#
|
||||
# IDEMPOTENCE / BLAST RADIUS. community.general.timezone is a converge-to-state module: it
|
||||
# reports `changed` only when /etc/localtime + /etc/timezone do not already name this zone,
|
||||
# and it does NOT step the clock (no time jump, no NTP interaction) — only the RENDERING of
|
||||
# an unchanged monotonic clock moves. No reboot is required. Already-running services that
|
||||
# render local time keep the old zone until they are next restarted (the module restarts
|
||||
# cron itself, which is the one scheduler on these nodes that cares); k3s, etcd and the
|
||||
# kubelet are unaffected either way because they emit UTC regardless.
|
||||
- name: Assert this node's timezone has actually been decided
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- node_baseline_timezone is string
|
||||
- node_baseline_timezone | length > 0
|
||||
- "'BLOCKED' not in node_baseline_timezone"
|
||||
- node_baseline_timezone is match('^[A-Za-z][A-Za-z0-9+_-]*(/[A-Za-z0-9+_.-]+)*$')
|
||||
fail_msg: >-
|
||||
node_baseline_timezone is not a usable zoneinfo name (got:
|
||||
'{{ node_baseline_timezone | default("<undefined>") }}'). It must be a name
|
||||
/usr/share/zoneinfo actually carries — `timedatectl list-timezones` on the node — and
|
||||
the repo default (Etc/UTC) is almost certainly what you want; see the rationale in
|
||||
roles/node_baseline/defaults/main.yml. Empty is NOT "leave it alone": a node whose zone
|
||||
nobody has decided is the config drift this task exists to remove.
|
||||
quiet: true
|
||||
tags: [system]
|
||||
|
||||
- name: Pin the node timezone (cross-node log correlation; UTC by default)
|
||||
community.general.timezone:
|
||||
name: "{{ node_baseline_timezone }}"
|
||||
tags: [system]
|
||||
|
||||
- name: Persist kernel modules for k3s/Cilium/Longhorn
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
|
||||
|
||||
Reference in new issue
Block a user