Two debt items, both about making a latent problem impossible rather than fixed.
=== 1. Node timezone pinned to Etc/UTC (node_baseline, `system` tag) ===
The prod nodes disagreed, measured from k3s's own etcd snapshot annotations:
prod-1/2 at -05:00 (Central), prod-3 at -04:00 (Eastern). Functionally harmless —
Kubernetes and etcd are UTC internally, and in UTC those snapshots are ~4 minutes
apart, which is correct staggering — but it makes cross-node log correlation
actively misleading, and this cluster has had two unplanned reboots in two days.
`Etc/UTC` chosen on repo convention, not just principle: catalog/plex.yaml records
"no other service in the repo sets a timezone ... so use the safe default" with
TZ: Etc/UTC, and apps/wave-0/plex.yaml matches (platform/crafty is America/Chicago
DELIBERATELY — a user-facing game schedule, not a log clock). Decisive extra fact:
THE THREE DEV NODES ARE ALREADY Etc/UTC, so this makes prod match dev rather than
inventing a convention. UTC also removes the class — no DST discontinuity, node
time == platform time.
New knob `node_baseline_timezone`, overridable per environment via group_vars (the
same seam as node_baseline_backports_kernel) but deliberately NOT restated there:
no environment needs to differ, and a second copy is a second thing to disagree.
Guarded by an assert in the role's existing BLOCKED style before the module runs.
`--check --diff --tags system -e k3s_hosts=k3s_prod` INDEPENDENTLY CONFIRMS the
offset evidence, and is the whole verification — the play was never run for real:
k3s-prod-1: -"America/Chicago" +"Etc/UTC" changed
k3s-prod-2: -"America/Chicago" +"Etc/UTC" changed
k3s-prod-3: -"America/New_York" +"Etc/UTC" changed
The same --check against DEV reports the task `ok` on all three — idempotence
demonstrated against live hosts, in check mode only. `--list-hosts` confirms
`-e k3s_hosts=k3s_prod` selects the three prod nodes (the diagnostic plays take
their OWN var; the wrong flag silently targets dev).
*** AUTHORED, NOT APPLIED. *** Applying it is a live-cluster action and was
deliberately left for the operator to schedule: prod's Longhorn had only just
finished resyncing after a node reboot, and changing a node's clock config during
storage recovery is not an unattended move.
`ansible-inventory --list` was NOT used anywhere — it prints DECRYPTED vault values
(the passphrase is mounted into the dev container) and has leaked the prod
become-password to a transcript once already.
=== 2. New L0 check `placeholder-guard-coupling` (R-ENV-02, R-STATE-03) ===
On 2026-09-05 the PM changed a placeholder in clusters/base/cluster-vars.yaml from
`TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID...`. The guards in
bootstrap/standup.sh matched the glob `TBD-*` — a PREFIX. All three silently died,
and `resume` would have enabled velero on prod against a nonexistent hostname:
installs clean, reports healthy, no backups exist. Found by accident. A guard in
one file keyed on the SHAPE of a value in another file, with nothing tying them.
The check exercises the REAL predicate — it does not reimplement it. standup.sh is
not sourceable (top-level argv parsing hard-exits, cd, export, EXIT trap), so the
check extracts the predicate's own source lines and sources only those. Faithfulness
is pinned by: discovery of every `*_is_placeholder` (not a hardcoded name),
exactly-one-definition, "extracted block is a verbatim substring", a self-containment
gate, a canary (must refuse "", must accept a real URL), and a scan for a divergent
second copy. If the predicate grows a dependency the check FAILS LOUDLY and names
the remedy rather than degrading to testing a copy — the failure mode it exists for.
Both sides derive from real files: guarded knobs from a taint trace in standup.sh
(`x="$(cvar KEY)"` -> alias hop -> `*_is_placeholder "$x"`); values from base
cluster-vars plus each overlay's JSON6902 /data/KEY patches. The independent
detector is deliberately NOT the guard: seed idioms UNION the guard's own words with
glob anchors STRIPPED — so a prefix-anchored guard cannot pass, and deleting an arm
cannot make a value stop being placeholder-shaped.
Real tree: 32 assertions, 18 values through the extracted predicate, PASS.
NON-VACUITY VERIFIED BY ME, twice, on throwaway copies (the real repo and the
read-only spec repo untouched). My first perturbation tripped the self-containment
rail instead of the anchoring property, so I redid it shape-preserving — substring
globs -> prefix globs, nothing else — and it failed on exactly the right property,
naming mid-string and tail acceptances with the diagnosis "This is the 2026-09-05
failure exactly". Control: unperturbed copy exits 0.
=== 3. ci/touched-requirements.sh: bootstrap/ was UNMAPPED (found by the agent) ===
The path->requirement map had no `bootstrap/*` entry, so a change touching
standup.sh ALONGSIDE any already-mapped path scoped the L0 run to that other path's
requirements and silently excluded the new check — the one that exists to catch
exactly the 2026-09-05 class. A safety net with a hole where the incident was.
Mapped to R-STATE-03 R-GITOPS-01; verified the map now resolves bootstrap paths.
VERIFIED: full CI gate PASS. `run.sh --layer l0` 8/8 PASS including the new check.
Coverage unchanged at 45/51 (the new check tags existing requirements).
Refs R-ENV-02, R-STATE-03, R-TEST-01, R-PLAT-01.
Co-Authored-By: Claude Opus 5 <[email protected]>
jakman85andClaude Opus 5
a984442cae