d8b25a67feeea46f3e1311d4e8fb15aedaaad8e8
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
79c703bc79 |
fix(conformance): whole-repo checks audited nested agent worktrees as source
Six discovery sites walked ROOT without pruning dot-directories, so they descended into .claude/worktrees/agent-*/ — up to 18 full copies of the repo, created for isolated agent runs and gitignored (.gitignore:43). The visible half was false FAILs: four ServiceMonitors reported as selecting no Service, and nine PrometheusRule manifests reported as "under test but not in the repo", all of them worktree paths. The whole-repo L0 gate was red for reasons that had nothing to do with authored surface. The dangerous half is quieter. A check that asks "does anything in the repo declare X" can be satisfied by a worktree copy instead of real source, so the same bug that invents failures can also manufacture a pass. That is the reason to fix all six sites rather than only the ones that happened to be failing. egress-paths-selected-by-rules already had this fix and documents it (it once saw 5 CiliumEgressGatewayPolicy objects instead of the 1 authored); this applies the same dot-directory predicate everywhere else, which skips dot-DIRECTORIES while still matching dot-FILES such as .sops.yaml. platform/monitoring/tests/run.sh is the other side of the same failure: its `--list` reported each manifest once per worktree, and the L0 check compares that listing against the tree. Both find calls now prune .claude alongside .git. `--list` returns 21 real pairs where it previously returned them ~19x over. Verified in the pinned toolbox: L0 --include-scope whole-repo is 21 cases, 0 failures, 0 skips, 0 unwired (was 4 failures, then 1). L1 3/3. The 18 stale worktrees are all on already-merged branches and remain on disk; cleaning them up is a separate change and is no longer load-bearing for the gate. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
87c3d83dff |
fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3 SKIPPED on every gate run since it landed, because its check keyed its whole existence on `bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is no deletion commit for it anywhere; it landed only on unmerged prod-standup branches (`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is the quiet version of the failure it was written to prevent. The skip was also hiding a RED. Run against the real `standup.sh` from `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both documented there as load-bearing). It would have false-blocked the merge landing its own input. NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard: HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE cluster-vars value; an empty effective value is STATED by the overlay rather than inherited in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the 2026-09-05 outcome asserted at the source instead of at the guard — if prod's S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does not exist. Input surface is `clusters/`, all of it on `main`. HALF B (self-activating): the real predicate extracted and executed against those same values, the consumer trace, and the mid-string anchoring property. The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded: any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at `bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When no guard exists the gap is printed loudly every run instead of swallowed as a skip. LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is always a defect in an overlay, while `""` is held to the explicitness rule instead — and still counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard author's declaration that empty is not a usable target. Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present, reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34. L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged — R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps. Also: * `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale comment corrected. It claimed standup.sh "carries the placeholder guards", describing a file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the change that lands a placeholder guard selected by nothing. * `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8) locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather than a silent loss of selection. All three ci/tests pass (12 + 31 + 15). * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a placeholder-shaped overlay value" is no longer true; corrected. * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that can break it attributes a requirement it declares. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
5fa278f61c |
fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
6a8953e288 |
feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision
|