One historical `-dirty` row reddened this gate permanently, even after a clean
re-run of the same test. status.py already resolves duplicates latest-wins
(latest_by_test), so a superseded row is not evidence of anything — but this check
iterated every row, so the poisoned row from the git_sha bug (fixed in the parent
commit) would have failed the audit forever.
The dirty-tree assertion is now scoped to rows status.py would actually honour, by
importing latest_by_test rather than reimplementing the rule, so the two cannot
disagree about which rows are live claims. Structural assertions — provenance keys,
sentinel, test-still-exists — still apply to EVERY row.
Nothing is erased: the row stays in the ledger and runs/20260913T014437Z-19fc8bc/
run.log stays committed. It simply stops being read as a live claim. Verified the
guard is intact afterwards — the check still REFUSES both forgeries it is built to
catch (a self-skip presented as a pass, and a proven row from the root commit).
A dirty-then-clean sequence is also the normal shape of noticing a dirty tree,
fixing it, and re-running; that should not permanently fail an audit.
Co-Authored-By: Claude Opus 5 <[email protected]>
1. auditability-proofs-producible rejected any proven ledger row without a
sentinel, with no layer exemption — while record_run.py DELIBERATELY records
L0/L1 rows sentinel-less via its own SENTINEL_EXEMPT_LAYERS, because
run_l1_offline.py renders skips as JUnit <skipped/> and those layers have no
pass token to attribute. So ANY `--record` run touching L0 or L1 wrote a row
this gate then rejected. Consequence: a requirement whose verifier set includes
an L1 check could never be both PROVEN and L0-green — R-NET-06 is exactly such
a requirement, and hit it the moment it was re-proven on prod. The check now
honours the same constant instead of restating the policy, so the two halves
cannot drift apart again.
2. git_sha() marked the tree dirty from tracked files, but conformance/runs/
ledger.jsonl IS tracked — so the FIRST `--record` run of a session dirtied the
tree and every LATER run was stamped `-dirty`, which check.sh then refuses to
honour. Recording one test poisoned the next, and that cost a second live
fault-injection run against prod to work around. Its docstring already stated
the intent for the untracked runs/<id>/ half of the same artifact; the ledger is
the tracked half. Now excluded via pathspec, mirroring status.py's
IMPL_EXCLUDE_PREFIXES which already lists this path for the same reason.
Neither change weakens an assertion: (1) applies an exemption that already existed
and was already documented, (2) stops the recorder invalidating its own output.
Co-Authored-By: Claude Opus 5 <[email protected]>
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
jakman85andClaude Opus 5