jakman85andClaude Opus 5
a984442cae
Two debt items, both about making a latent problem impossible rather than fixed. === 1. Node timezone pinned to Etc/UTC (node_baseline, `system` tag) === The prod nodes disagreed, measured from k3s's own etcd snapshot annotations: prod-1/2 at -05:00 (Central), prod-3 at -04:00 (Eastern). Functionally harmless — Kubernetes and etcd are UTC internally, and in UTC those snapshots are ~4 minutes apart, which is correct staggering — but it makes cross-node log correlation actively misleading, and this cluster has had two unplanned reboots in two days. `Etc/UTC` chosen on repo convention, not just principle: catalog/plex.yaml records "no other service in the repo sets a timezone ... so use the safe default" with TZ: Etc/UTC, and apps/wave-0/plex.yaml matches (platform/crafty is America/Chicago DELIBERATELY — a user-facing game schedule, not a log clock). Decisive extra fact: THE THREE DEV NODES ARE ALREADY Etc/UTC, so this makes prod match dev rather than inventing a convention. UTC also removes the class — no DST discontinuity, node time == platform time. New knob `node_baseline_timezone`, overridable per environment via group_vars (the same seam as node_baseline_backports_kernel) but deliberately NOT restated there: no environment needs to differ, and a second copy is a second thing to disagree. Guarded by an assert in the role's existing BLOCKED style before the module runs. `--check --diff --tags system -e k3s_hosts=k3s_prod` INDEPENDENTLY CONFIRMS the offset evidence, and is the whole verification — the play was never run for real: k3s-prod-1: -"America/Chicago" +"Etc/UTC" changed k3s-prod-2: -"America/Chicago" +"Etc/UTC" changed k3s-prod-3: -"America/New_York" +"Etc/UTC" changed The same --check against DEV reports the task `ok` on all three — idempotence demonstrated against live hosts, in check mode only. `--list-hosts` confirms `-e k3s_hosts=k3s_prod` selects the three prod nodes (the diagnostic plays take their OWN var; the wrong flag silently targets dev). *** AUTHORED, NOT APPLIED. *** Applying it is a live-cluster action and was deliberately left for the operator to schedule: prod's Longhorn had only just finished resyncing after a node reboot, and changing a node's clock config during storage recovery is not an unattended move. `ansible-inventory --list` was NOT used anywhere — it prints DECRYPTED vault values (the passphrase is mounted into the dev container) and has leaked the prod become-password to a transcript once already. === 2. New L0 check `placeholder-guard-coupling` (R-ENV-02, R-STATE-03) === On 2026-09-05 the PM changed a placeholder in clusters/base/cluster-vars.yaml from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID...`. The guards in bootstrap/standup.sh matched the glob `TBD-*` — a PREFIX. All three silently died, and `resume` would have enabled velero on prod against a nonexistent hostname: installs clean, reports healthy, no backups exist. Found by accident. A guard in one file keyed on the SHAPE of a value in another file, with nothing tying them. The check exercises the REAL predicate — it does not reimplement it. standup.sh is not sourceable (top-level argv parsing hard-exits, cd, export, EXIT trap), so the check extracts the predicate's own source lines and sources only those. Faithfulness is pinned by: discovery of every `*_is_placeholder` (not a hardcoded name), exactly-one-definition, "extracted block is a verbatim substring", a self-containment gate, a canary (must refuse "", must accept a real URL), and a scan for a divergent second copy. If the predicate grows a dependency the check FAILS LOUDLY and names the remedy rather than degrading to testing a copy — the failure mode it exists for. Both sides derive from real files: guarded knobs from a taint trace in standup.sh (`x="$(cvar KEY)"` -> alias hop -> `*_is_placeholder "$x"`); values from base cluster-vars plus each overlay's JSON6902 /data/KEY patches. The independent detector is deliberately NOT the guard: seed idioms UNION the guard's own words with glob anchors STRIPPED — so a prefix-anchored guard cannot pass, and deleting an arm cannot make a value stop being placeholder-shaped. Real tree: 32 assertions, 18 values through the extracted predicate, PASS. NON-VACUITY VERIFIED BY ME, twice, on throwaway copies (the real repo and the read-only spec repo untouched). My first perturbation tripped the self-containment rail instead of the anchoring property, so I redid it shape-preserving — substring globs -> prefix globs, nothing else — and it failed on exactly the right property, naming mid-string and tail acceptances with the diagnosis "This is the 2026-09-05 failure exactly". Control: unperturbed copy exits 0. === 3. ci/touched-requirements.sh: bootstrap/ was UNMAPPED (found by the agent) === The path->requirement map had no `bootstrap/*` entry, so a change touching standup.sh ALONGSIDE any already-mapped path scoped the L0 run to that other path's requirements and silently excluded the new check — the one that exists to catch exactly the 2026-09-05 class. A safety net with a hole where the incident was. Mapped to R-STATE-03 R-GITOPS-01; verified the map now resolves bootstrap paths. VERIFIED: full CI gate PASS. `run.sh --layer l0` 8/8 PASS including the new check. Coverage unchanged at 45/51 (the new check tags existing requirements). Refs R-ENV-02, R-STATE-03, R-TEST-01, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]>
69 lines
4.0 KiB
YAML
69 lines
4.0 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — a placeholder value in cluster-vars is REFUSED by the guard that owns it
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
#
|
|
# THE BUG CLASS THIS CLOSES (it has now bitten three times in one file). A guard in
|
|
# `bootstrap/standup.sh` keys on the SHAPE of a value that lives in
|
|
# `clusters/base/cluster-vars.yaml`, and nothing tied the two together. On 2026-09-05 the
|
|
# S3 placeholder changed from `TBD-PROD-S3-BACKUP-ENDPOINT` to
|
|
# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`; the three guards matched the glob
|
|
# `TBD-*`, a PREFIX match, and `fnmatch('https://TBD-R2-…','TBD-*')` is False. All three
|
|
# guards silently stopped matching, and `standup.sh resume` would have resumed `velero` on
|
|
# prod against a nonexistent hostname: installs clean, reports healthy, NO BACKUPS EXIST.
|
|
# It was found by accident. The guard is fixed (`s3_is_placeholder`, substring); the
|
|
# COUPLING was untested, which is what makes the class reintroducible.
|
|
#
|
|
# GIVEN the repository as authored,
|
|
# WHEN the REAL predicate is extracted from bootstrap/standup.sh and executed by bash
|
|
# (never reimplemented — see check.sh "HOW THE REAL GUARD IS EXERCISED"),
|
|
# THEN (1) every authored value of a guarded knob that is placeholder-shaped by an
|
|
# INDEPENDENT detector is actually refused by that guard, in base and in every
|
|
# environment overlay;
|
|
# (2) no environment overlay carries a placeholder-shaped value for a knob no guard
|
|
# refuses;
|
|
# (3) every function in standup.sh that reads a guarded knob applies the guard to it
|
|
# (the "three guards died at once" regression);
|
|
# (4) the guard matches its own vocabulary ANYWHERE in a value, not just as a prefix
|
|
# — the exact anchoring defect above — and still accepts a real endpoint;
|
|
# AND (5) the extraction is faithful, and the check fails LOUDLY rather than silently
|
|
# testing a copy if the guard stops being a self-contained predicate.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
|
|
# PyYAML + bash); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test
|
|
# wraps the SAME script.
|
|
#
|
|
# NOT proven: that the endpoint is REACHABLE or that backups exist (runtime facts —
|
|
# `standup.sh s3check` probes the former from a pod, l3-fault-injection/dr-restore proves
|
|
# the latter); that guards for knobs OTHER than the placeholder-refusing ones are correct;
|
|
# and that the guard runs at all in a real standup (the dispatch path is shell, executed
|
|
# only against a live cluster). R-STATE-03 is tagged for the narrow static slice only: the
|
|
# authored backup target cannot be a placeholder that the guard fails to recognise.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: placeholder-guard-coupling
|
|
labels:
|
|
conformance.home.k3s/R-ENV-02: "true"
|
|
conformance.home.k3s/R-STATE-03: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03"
|
|
conformance.home.k3s/layer: "L0"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Every placeholder-shaped value authored in cluster-vars, for any environment
|
|
overlay, is actually recognised as a placeholder by the real bootstrap guard that
|
|
is supposed to refuse it — so a value change in one file cannot silently disarm a
|
|
guard in another and enable velero against an endpoint that does not exist.
|
|
spec:
|
|
steps:
|
|
- name: couple-authored-placeholders-to-the-real-guard
|
|
description: >-
|
|
Extract the real placeholder predicate from bootstrap/standup.sh, run it against
|
|
every authored cluster-vars value the repo actually carries, and assert the guard
|
|
refuses each one an independent detector calls a placeholder.
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|