harden: pin node timezone to UTC, and TEST the guard/value coupling that bit us

Two debt items, both about making a latent problem impossible rather than fixed.

=== 1. Node timezone pinned to Etc/UTC (node_baseline, `system` tag) ===

The prod nodes disagreed, measured from k3s's own etcd snapshot annotations:
prod-1/2 at -05:00 (Central), prod-3 at -04:00 (Eastern). Functionally harmless —
Kubernetes and etcd are UTC internally, and in UTC those snapshots are ~4 minutes
apart, which is correct staggering — but it makes cross-node log correlation
actively misleading, and this cluster has had two unplanned reboots in two days.

`Etc/UTC` chosen on repo convention, not just principle: catalog/plex.yaml records
"no other service in the repo sets a timezone ... so use the safe default" with
TZ: Etc/UTC, and apps/wave-0/plex.yaml matches (platform/crafty is America/Chicago
DELIBERATELY — a user-facing game schedule, not a log clock). Decisive extra fact:
THE THREE DEV NODES ARE ALREADY Etc/UTC, so this makes prod match dev rather than
inventing a convention. UTC also removes the class — no DST discontinuity, node
time == platform time.

New knob `node_baseline_timezone`, overridable per environment via group_vars (the
same seam as node_baseline_backports_kernel) but deliberately NOT restated there:
no environment needs to differ, and a second copy is a second thing to disagree.
Guarded by an assert in the role's existing BLOCKED style before the module runs.

`--check --diff --tags system -e k3s_hosts=k3s_prod` INDEPENDENTLY CONFIRMS the
offset evidence, and is the whole verification — the play was never run for real:
  k3s-prod-1: -"America/Chicago"  +"Etc/UTC"  changed
  k3s-prod-2: -"America/Chicago"  +"Etc/UTC"  changed
  k3s-prod-3: -"America/New_York" +"Etc/UTC"  changed
The same --check against DEV reports the task `ok` on all three — idempotence
demonstrated against live hosts, in check mode only. `--list-hosts` confirms
`-e k3s_hosts=k3s_prod` selects the three prod nodes (the diagnostic plays take
their OWN var; the wrong flag silently targets dev).

*** AUTHORED, NOT APPLIED. *** Applying it is a live-cluster action and was
deliberately left for the operator to schedule: prod's Longhorn had only just
finished resyncing after a node reboot, and changing a node's clock config during
storage recovery is not an unattended move.

`ansible-inventory --list` was NOT used anywhere — it prints DECRYPTED vault values
(the passphrase is mounted into the dev container) and has leaked the prod
become-password to a transcript once already.

=== 2. New L0 check `placeholder-guard-coupling` (R-ENV-02, R-STATE-03) ===

On 2026-09-05 the PM changed a placeholder in clusters/base/cluster-vars.yaml from
`TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID...`. The guards in
bootstrap/standup.sh matched the glob `TBD-*` — a PREFIX. All three silently died,
and `resume` would have enabled velero on prod against a nonexistent hostname:
installs clean, reports healthy, no backups exist. Found by accident. A guard in
one file keyed on the SHAPE of a value in another file, with nothing tying them.

The check exercises the REAL predicate — it does not reimplement it. standup.sh is
not sourceable (top-level argv parsing hard-exits, cd, export, EXIT trap), so the
check extracts the predicate's own source lines and sources only those. Faithfulness
is pinned by: discovery of every `*_is_placeholder` (not a hardcoded name),
exactly-one-definition, "extracted block is a verbatim substring", a self-containment
gate, a canary (must refuse "", must accept a real URL), and a scan for a divergent
second copy. If the predicate grows a dependency the check FAILS LOUDLY and names
the remedy rather than degrading to testing a copy — the failure mode it exists for.

Both sides derive from real files: guarded knobs from a taint trace in standup.sh
(`x="$(cvar KEY)"` -> alias hop -> `*_is_placeholder "$x"`); values from base
cluster-vars plus each overlay's JSON6902 /data/KEY patches. The independent
detector is deliberately NOT the guard: seed idioms UNION the guard's own words with
glob anchors STRIPPED — so a prefix-anchored guard cannot pass, and deleting an arm
cannot make a value stop being placeholder-shaped.

Real tree: 32 assertions, 18 values through the extracted predicate, PASS.

NON-VACUITY VERIFIED BY ME, twice, on throwaway copies (the real repo and the
read-only spec repo untouched). My first perturbation tripped the self-containment
rail instead of the anchoring property, so I redid it shape-preserving — substring
globs -> prefix globs, nothing else — and it failed on exactly the right property,
naming mid-string and tail acceptances with the diagnosis "This is the 2026-09-05
failure exactly". Control: unperturbed copy exits 0.

=== 3. ci/touched-requirements.sh: bootstrap/ was UNMAPPED (found by the agent) ===

The path->requirement map had no `bootstrap/*` entry, so a change touching
standup.sh ALONGSIDE any already-mapped path scoped the L0 run to that other path's
requirements and silently excluded the new check — the one that exists to catch
exactly the 2026-09-05 class. A safety net with a hole where the incident was.
Mapped to R-STATE-03 R-GITOPS-01; verified the map now resolves bootstrap paths.

VERIFIED: full CI gate PASS. `run.sh --layer l0` 8/8 PASS including the new check.
Coverage unchanged at 45/51 (the new check tags existing requirements).

Refs R-ENV-02, R-STATE-03, R-TEST-01, R-PLAT-01.

Co-Authored-By: Claude Opus 5 <[email protected]>

infra/ansible/collections/requirements.yml

@@ -2,7 +2,8 @@
 # Ansible Galaxy collections required by infra/ansible. Install with:
 #   ansible-galaxy collection install -r collections/requirements.yml
 collections:
-  # node_baseline role: community.general.modprobe, community.general.filesystem
+  # node_baseline role: community.general.modprobe, community.general.filesystem,
+  #                      community.general.timezone
   - name: community.general
   # node_baseline role: ansible.posix.mount
   - name: ansible.posix

infra/ansible/roles/node_baseline/defaults/main.yml

@@ -81,6 +81,45 @@ node_baseline_kernel_modules:
   - dm_crypt
   - nfs
 
+# --- NODE CLOCK / TIMEZONE (tag `system`) ---------------------------------------------
+# EVERY NODE IS UTC. Measured drift that made this a knob rather than an assumption: the
+# three prod nodes disagreed on their local zone (read off k3s's own etcd snapshot
+# annotations, 2026-09-05) —
+#   k3s-prod-1  2026-09-05T16:31:28-05:00   (Central)
+#   k3s-prod-2  2026-09-05T16:29:07-05:00   (Central)
+#   k3s-prod-3  2026-09-05T17:27:35-04:00   (Eastern)
+# In UTC those three are ~4 minutes apart, which is the CORRECT staggering: nothing was
+# functionally wrong, because Kubernetes, etcd and the container runtime are UTC internally.
+# What IS wrong is that any timestamp a node renders in LOCAL time — journald's default
+# output, `ls -l`, a shell prompt, an etcd snapshot filename — cannot be compared across
+# nodes without knowing each node's zone. Correlating three nodes' logs during an incident
+# is exactly when nobody has that knowledge to hand, and this cluster took two unplanned
+# node reboots in the 24h before this was written.
+#
+# WHY UTC AND NOT "STANDARDISE ON CENTRAL". Picking one local zone leaves the class of
+# problem intact (a fourth node, or a DST boundary, re-creates it) and adds a translation
+# step to every log read, because every timestamp the PLATFORM emits — Kubernetes events,
+# Prometheus, Loki, Velero, CNPG WAL, the etcd snapshot annotations above — is already UTC.
+# UTC removes the class: node-local time and platform time become the same clock, and there
+# is no DST discontinuity in which a repeated or skipped hour breaks log ordering. This also
+# matches the only timezone convention the repo already had — catalog/plex.yaml records
+# "no other service in the repo sets a timezone ... so use the safe default" and both it and
+# apps/wave-0/plex.yaml set TZ=Etc/UTC. (platform/crafty sets America/Chicago, deliberately:
+# that is a USER-FACING game server's schedule, not a node's log clock.)
+#
+# WHY A KNOB AT ALL (R-ENV-02). The value is a node-level Ansible fact, so its override seam
+# is inventory/group_vars/<env>, NOT clusters/base/cluster-vars.yaml (a Flux postBuild
+# ConfigMap no Ansible code reads) — the same seam node_baseline_backports_kernel and
+# node_baseline_dns_servers use. NO ENVIRONMENT OVERRIDES IT TODAY, and one should not have
+# to: dev and prod both want UTC and there is nothing environment-specific about a node
+# clock. It is a variable rather than a literal in the task so a genuinely
+# operator-facing one-off host (never a k3s node) can differ without forking the role.
+#
+# Must be a zoneinfo name present on the node (`timedatectl list-timezones`); the assert in
+# tasks/main.yml rejects an empty or obviously-malformed value before the module runs, and
+# community.general.timezone itself refuses a name /usr/share/zoneinfo does not have.
+node_baseline_timezone: Etc/UTC
+
 # --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
 # The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
 # a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -262,6 +262,52 @@
       vars:
         newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
 
+# =====================================================================================
+# NODE CLOCK — tag `system`: every node renders time in the SAME zone, and that zone is UTC
+# =====================================================================================
+# MEASURED DRIFT (2026-09-05), read off k3s's own etcd snapshot annotations on live prod:
+#   k3s-prod-1  2026-09-05T16:31:28-05:00   Central
+#   k3s-prod-2  2026-09-05T16:29:07-05:00   Central
+#   k3s-prod-3  2026-09-05T17:27:35-04:00   Eastern  <-- a different UTC offset
+# Functionally harmless (in UTC those three are ~4 minutes apart, which is the correct
+# staggering — Kubernetes, etcd and containerd are UTC internally). Operationally it is a
+# trap: any timestamp a node renders in LOCAL time (journald's default output, `ls -l`, an
+# etcd snapshot filename) is not comparable across nodes without knowing each node's zone,
+# and cross-node log correlation during an incident is precisely when nobody knows that.
+#
+# Rationale for UTC rather than standardising on one local zone, and for this being a knob
+# whose override seam is inventory/group_vars/<env>: see node_baseline_timezone in
+# defaults/main.yml. NOT a cluster-vars knob — no Ansible code reads that ConfigMap.
+#
+# IDEMPOTENCE / BLAST RADIUS. community.general.timezone is a converge-to-state module: it
+# reports `changed` only when /etc/localtime + /etc/timezone do not already name this zone,
+# and it does NOT step the clock (no time jump, no NTP interaction) — only the RENDERING of
+# an unchanged monotonic clock moves. No reboot is required. Already-running services that
+# render local time keep the old zone until they are next restarted (the module restarts
+# cron itself, which is the one scheduler on these nodes that cares); k3s, etcd and the
+# kubelet are unaffected either way because they emit UTC regardless.
+- name: Assert this node's timezone has actually been decided
+  ansible.builtin.assert:
+    that:
+      - node_baseline_timezone is string
+      - node_baseline_timezone | length > 0
+      - "'BLOCKED' not in node_baseline_timezone"
+      - node_baseline_timezone is match('^[A-Za-z][A-Za-z0-9+_-]*(/[A-Za-z0-9+_.-]+)*$')
+    fail_msg: >-
+      node_baseline_timezone is not a usable zoneinfo name (got:
+      '{{ node_baseline_timezone | default("<undefined>") }}'). It must be a name
+      /usr/share/zoneinfo actually carries — `timedatectl list-timezones` on the node — and
+      the repo default (Etc/UTC) is almost certainly what you want; see the rationale in
+      roles/node_baseline/defaults/main.yml. Empty is NOT "leave it alone": a node whose zone
+      nobody has decided is the config drift this task exists to remove.
+    quiet: true
+  tags: [system]
+
+- name: Pin the node timezone (cross-node log correlation; UTC by default)
+  community.general.timezone:
+    name: "{{ node_baseline_timezone }}"
+  tags: [system]
+
 - name: Persist kernel modules for k3s/Cilium/Longhorn
   ansible.builtin.copy:
     dest: /etc/modules-load.d/k3s-cilium-longhorn.conf