204f0ec28e719bf59ee299f15a5fccfb2ddb1bb1
17
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jakman85
|
5fa893095b |
merge: the placeholder guard test was hiding a RED, not a pass
It had been permanently SKIPPED because it expected bootstrap/standup.sh, which never landed on main — planned, never merged, still living only on the unmerged prod-standup branches along with the 2026-09-05 incident it came from. Dropping the real script in and running the old check produced FAIL 2/34: it would have false-blocked the very merge that landed its own input. Now split into the half that needs no guard (no overlay ships a placeholder-shaped effective cluster-vars value; an empty value must be STATED by the overlay, not inherited in silence; a loud base sentinel must be explicitly overridden) and the half that self-activates. The guard is discovered by CONTENT — any tracked *.sh defining <name>_is_placeholder() — so landing one anywhere activates the second half with no edit here, and two definitions is a failure. The check never exits 3 again. Unscoped L0 goes 13 PASS / 1 SKIP -> 14 PASS / 0 SKIP. |
||
|
jakman85
|
e49b86eb41 |
merge: 35 runbooks, served, with a guard that they resolve
All 46 runbook_url annotations pointed at https://<apex>/runbooks/<name> and every one returned 404 — there was no runbooks/ directory and no route. Every alert that reached a human handed them a dead link at the moment they needed it. Runbooks are now served from their own non-gating tier at the apex prefix, PUBLIC on purpose: gating them behind Keycloak would make keycloak-down unreadable exactly when Keycloak is down, which is a loop with no exit. The new L0 guard checks both directions — a runbook_url with no file, and a file not wired into the generator — and it caught a live one on contact with main: CNPGPrimariesCoLocated had shipped hours earlier pointing at a runbook that did not exist. |
||
 jakman85andClaude Opus 5
|
87c3d83dff |
fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3 SKIPPED on every gate run since it landed, because its check keyed its whole existence on `bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is no deletion commit for it anywhere; it landed only on unmerged prod-standup branches (`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is the quiet version of the failure it was written to prevent. The skip was also hiding a RED. Run against the real `standup.sh` from `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both documented there as load-bearing). It would have false-blocked the merge landing its own input. NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard: HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE cluster-vars value; an empty effective value is STATED by the overlay rather than inherited in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the 2026-09-05 outcome asserted at the source instead of at the guard — if prod's S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does not exist. Input surface is `clusters/`, all of it on `main`. HALF B (self-activating): the real predicate extracted and executed against those same values, the consumer trace, and the mid-string anchoring property. The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded: any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at `bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When no guard exists the gap is printed loudly every run instead of swallowed as a skip. LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is always a defect in an overlay, while `""` is held to the explicitness rule instead — and still counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard author's declaration that empty is not a usable target. Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present, reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34. L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged — R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps. Also: * `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale comment corrected. It claimed standup.sh "carries the placeholder guards", describing a file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the change that lands a placeholder guard selected by nothing. * `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8) locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather than a silent loss of selection. All three ci/tests pass (12 + 31 + 15). * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a placeholder-shaped overlay value" is no longer true; corrected. * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that can break it attributes a requirement it declares. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
78d0f1b0f6 |
feat(ci): promtool in the pinned toolchain, one alert-rule harness, and the gaps declared
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern
|
||
|
jakman85andClaude Opus 5
|
b7341b329b |
fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
44983095e1 |
feat(runbooks): make the 42 alert runbook_url annotations resolve
Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names, across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute serving that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that reached a human handed them a dead link at the exact moment the documentation is read, and the annotation made it look solved — the same green-surface-over-absent-mechanism class as the import Job that reported success while storing placeholder secrets. No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is made to resolve instead. WHAT LANDS - runbooks/<34>.md — one per name, each grounded in this cluster's real topology and, where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG 38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the 2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12 clients held literal placeholder secrets behind an entirely green cluster, the Velero R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap re-enabling itself across reboots via systemd-gpt-auto-generator. Each page states what the alert means HERE, the first diagnostic commands, likely causes ordered by probability with a remediation each, and when to escalate or stop. - Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss. - Every page states that kubectl only works via `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`, because dev.sh defaults its current-context to k3s-dev. - Pages also record which alert groups are INERT today (measured 2026-09-13: no Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules evaluate against nothing) so nobody reads their silence as health. SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier (dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file. - The service catalog cannot express this route: ingressRoutes[].host is pattern-bound to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is no path field. - Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is broken, and Gitea is an app-tier workload whose own dependencies several of these runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier, so it would need 34 hand-maintained ReplaceFullPath rules. - configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md), so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not load anyway. - The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01 emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the https-apex listener's, so nothing needs substituting. AUTH — PUBLIC, deliberately auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make runbooks/keycloak-down unreadable when Keycloak is down and runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode. An operator who cannot log in is this route's primary reader. Content is operational procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute so the route is left OUT of the Gateway ext-authz attachment when forward-auth enforcement lands. PROVEN — in-cluster probe through the Gateway on prod /runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8 /runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8 /runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8 /runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8 /runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8 /runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8 HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the `/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence. GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04) 363 offline assertions. It checks BOTH halves, because the regression is silent in both directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its configMapGenerator entry is in git, reviewed, merged, and still 404s in the cluster. Also asserts a quality floor so a stub cannot make the link look fixed, and that route/listener/nginx prefix agree with the annotations. Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the generator (1 FAIL), and a 31-byte stub (7 FAIL). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
5fa278f61c |
fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
6202d92c27 |
test(conformance): gate the URL desired state, and prove the selftest can fail
L0 (212 assertions, was 175):
* runs ci/render-keycloak-client-urls.py --check, so client-urls.yaml must be exactly
what the realm import renders. A hand edit that makes the two disagree is
unmergeable -- this is what keeps the realm import the single source of truth for
hostnames rather than just the place they were first typed.
* asserts client-urls.yaml and the realm import are rendered by the SAME Flux
Kustomization. postBuild.substituteFrom is per-Kustomization, so a split would let
one ${BASE_DOMAIN} resolve to two different values and the realm would converge on
a hostname the HTTPRoutes do not serve.
* asserts the desired state uses no substitution token the realm definition lacks,
names only clients the import declares, and declares no field outside the
five-field write surface; that the CronJob mounts it readOnly at the path the
script defaults to; and that the script, renderer and check agree on which fields
are writable.
* NEW mutations.py: injects 12 plausible regressions into the EXTRACTED script and
requires --selftest to catch every one. A passing suite proves nothing if it cannot
fail, and "reported success while the realm was broken" is precisely what this
component exists to end -- so the same standard applies to its own test. ~20s.
L2: the audit Job now mounts keycloak-client-urls and asserts
KEYCLOAK_CLIENT_URLS_AUDIT_OK. Without that it would pass on a realm still holding
pre-migration redirect URIs -- the exact state prod was measured in.
Alerts: the existing Failing/Stale pair is EXTENDED to name redirect-URI drift, not
duplicated. One CronJob gets one pair; a second pair on the same job_name regex would
fire twice for one cause. No new critical, no new permanently-firing rule.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
fa4deac877 |
fix(conformance): the dirty-tree assertion must judge HONOURED evidence
One historical `-dirty` row reddened this gate permanently, even after a clean re-run of the same test. status.py already resolves duplicates latest-wins (latest_by_test), so a superseded row is not evidence of anything — but this check iterated every row, so the poisoned row from the git_sha bug (fixed in the parent commit) would have failed the audit forever. The dirty-tree assertion is now scoped to rows status.py would actually honour, by importing latest_by_test rather than reimplementing the rule, so the two cannot disagree about which rows are live claims. Structural assertions — provenance keys, sentinel, test-still-exists — still apply to EVERY row. Nothing is erased: the row stays in the ledger and runs/20260913T014437Z-19fc8bc/ run.log stays committed. It simply stops being read as a live claim. Verified the guard is intact afterwards — the check still REFUSES both forgeries it is built to catch (a self-skip presented as a pass, and a proven row from the root commit). A dirty-then-clean sequence is also the normal shape of noticing a dirty tree, fixing it, and re-running; that should not permanently fail an audit. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c66a7e29e9 |
fix(conformance): the harness contradicted itself in two places
1. auditability-proofs-producible rejected any proven ledger row without a sentinel, with no layer exemption — while record_run.py DELIBERATELY records L0/L1 rows sentinel-less via its own SENTINEL_EXEMPT_LAYERS, because run_l1_offline.py renders skips as JUnit <skipped/> and those layers have no pass token to attribute. So ANY `--record` run touching L0 or L1 wrote a row this gate then rejected. Consequence: a requirement whose verifier set includes an L1 check could never be both PROVEN and L0-green — R-NET-06 is exactly such a requirement, and hit it the moment it was re-proven on prod. The check now honours the same constant instead of restating the policy, so the two halves cannot drift apart again. 2. git_sha() marked the tree dirty from tracked files, but conformance/runs/ ledger.jsonl IS tracked — so the FIRST `--record` run of a session dirtied the tree and every LATER run was stamped `-dirty`, which check.sh then refuses to honour. Recording one test poisoned the next, and that cost a second live fault-injection run against prod to work around. Its docstring already stated the intent for the untracked runs/<id>/ half of the same artifact; the ledger is the tracked half. Now excluded via pathspec, mirroring status.py's IMPL_EXCLUDE_PREFIXES which already lists this path for the same reason. Neither change weakens an assertion: (1) applies an exemption that already existed and was already documented, (2) stops the recorder invalidating its own output. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
da8a0eac04 |
test(l0): R-STATE-04 gets a real check — bulk storage provably is not the replicated tier
The gate BLOCKED this branch, correctly, and the block was the interesting part.
Attaching the media library selects R-STATE-04 as a touched requirement ("bulk media storage has
a capacity strategy distinct from the replicated tier"), and R-STATE-04 had NO scaffolded test, so
the coverage REQUIRE stage failed both l0 and l1. There were two ways out. One was to keep
avoiding the token — the existing discipline in catalog/plex.yaml and catalog/qbittorrent.yaml,
which say in as many words that naming the requirement would block the merge. That made sense while
the tier was a stub. It does not make sense in the change that DELIVERS the tier: it would mean
shipping the requirement and claiming nothing. So: the check.
WHAT IT ASSERTS, and why each one is a real failure mode rather than a shape
The `bulk` StorageClass sat as `server: REPLACE-ME` for six roadmap phases, and the only thing
keeping a multi-terabyte library off the 3-replica `longhorn` class was four hand-written comments.
That is the class of failure being closed: a bulk volume on the replicated tier does not fail
loudly — Longhorn accepts the PVC and starts replicating 3x of data three mini-PCs cannot hold.
(1) DISTINCTNESS — no bulk PV/PVC on any class the repo declares with the Longhorn provisioner
(derived from platform/longhorn/, so adding a Longhorn class cannot create a blind spot),
and the class is explicitly EMPTY.
(2) CAPACITY STRATEGY as arithmetic — a bulk volume at least an order of magnitude above the
same service's replicated volume. The requirement is about two magnitudes of data; a "bulk"
volume sized like a config PVC means the tier was named, not applied. The 10x floor is
CALIBRATED against the real corpus, not guessed: the media library is ~460x its config
volume and the Nextcloud archive ~13x its 20Gi, so a higher floor would fire on a legitimate
volume and teach the next reader to weaken the check. My first draft used 100x and did
exactly that — the corpus caught it.
(3) PRE-BOUND FROM BOTH ENDS (claimRef + volumeName, agreeing). Guards the most deceptive
failure available in this tier: a dynamic bulk class hands out a FRESH EMPTY directory, the
pod mounts, the PVC Binds, everything is green, and the 8 TiB was never attached.
(4) Retain, always. (5) mountOptions carry `hard` and exactly one nfsvers=4.x, and never `soft`
— a soft mount hands a server hiccup to the app as a short read, which for a media library
is silent corruption; and v4 is what keeps the firewall surface at one port.
(6) ONE EXPORT, AT MOST ONE MOUNT PER WORKLOAD. This is the assertion I most wanted. Splitting
one export into two subPath mounts looks TIGHTER and breaks link() with EXDEV; Sonarr and
Radarr then degrade to copy-then-delete, rewriting every import over NFS and destroying the
seeding copy, surfacing weeks later as "imports are slow and my ratio is gone". No reviewer
sees that in a diff.
(7) READ-ONLY COUPLING — nfs.readOnly, the volumeMount flag and ReadOnlyMany must agree, so
dropping one still leaves the others; and every bulk mode is a `...Many`.
(8) The server is a BARE cluster-vars token, that knob has a base default, and EVERY overlay
supplies it — the seam that lets one authored definition serve two clusters that reach this
host by different addresses.
(9) LOAD-BEARING and NON-VACUOUS. Wired unconditionally into ci/l0.sh as Stage 3e for the same
reason as 3c/3d: Stage 3b scopes to touched requirements, and the path map keys NEITHER
apps/* NOR platform/longhorn/* to this requirement — so the change most likely to break
this could attribute nothing and be gated by nothing. An EMPTY bulk corpus FAILS rather
than passing trivially, and a floor on the assertion count keeps a pass from degenerating.
It reads the COMMITTED apps/ goldens rather than calling the renderer. Asking the renderer would
only prove it agrees with itself; the goldens are what Flux reconciles, and render==golden is
already its own gate stage.
PERTURBED, because a check that has never failed is not evidence. Each of these was applied to the
real tree and each fired with the intended message: bulk PVC moved onto `longhorn`; Retain -> Delete;
`hard` -> `soft`; the token replaced by the literal 192.168.1.9; sonarr's single mount split into
two subPath mounts (assertion 6); and the prod overlay's knob deleted.
Also hardens render.py: `size` is structurally required of a bulk entry (a PV with no capacity is
rejected by the API server) but the JSON Schema only enforces `export`, so omitting it raised a bare
KeyError three frames down. It now raises a sentence naming the service, the volume and the remedy.
Found by the same perturbation pass.
Coverage: 48/51 -> 49/51 (96.1%), 2 gaps, 0 safety-critical.
BASE_REF=main make -C ci gate -> MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
89cf59ffb7 | docs(conformance): the GIVEN/WHEN/THEN header states the exception it now allows | ||
|
jakman85
|
730d12ddc4 |
fix(conformance): R-ENV-03 shared-path rule tests the harm, not the filename
The blanket regex around RENDERED_DIRS failed any tracked file whose name
carries a delimited `prod` token. That caught the four per-environment backup
credential halves (BACKUP_CREDENTIALS_SUFFIX) even though BOTH halves are
applied to BOTH clusters as base resources and the overlay selects only which
one the shared consumer REFERENCES — i.e. exactly the R-ENV-02 contract the
regex exists to protect. Renaming the files to dodge the regex was rejected as
gaming the guard, so the rule now encodes the property instead.
The fix is not an allowlist of those four paths. A prod-token file in a shared
path is excused only when the tree proves all of:
(A) it declares nothing but Secrets whose names end in the environment suffix
(1) its own directory's kustomization applies it, and EVERY environment value
of the selector resolves to a Secret that some resource of that same
kustomization declares, in the same namespace
(2) nothing names the suffixed Secret — or the suffixed file — literally;
every reference is `<base-name>${VAR}`
(3) that VAR is defined for the base and for every overlay, and prod's value
is the one that selects this file
References are read from parsed YAML values, never raw text: the mechanism is
spelled out by name in comments beside the resources, and a comment references
nothing.
No other assertion is weakened; the check still runs 96 assertions, and the
pass output now notes each excused file and why.
|
||
|
jakman85andClaude Opus 5
|
dc2d1ce17e |
fix(conformance): the R-NET-04 scan descended into nested agent worktrees
TRACKED_YAML rglob'd from the repo root excluding only .git and _artifacts, so it also walked .claude/worktrees/agent-*/ — the isolated worktrees created for agent runs. The check counted 5 CiliumEgressGatewayPolicy objects where exactly 1 is authored, failing R-NET-04 for a reason unrelated to egress steering. Now skips dot-DIRECTORIES (covering .git, .claude, .cache) while still matching dot-FILES such as .sops.yaml. .claude/ is gitignored, so nothing beneath it is authored surface. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6a8953e288 |
feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision
|
||
|
jakman85andClaude Opus 5
|
8a2be824f6 |
feat(ci): substitution-safety gate + escape 6 comment hazards; strict mode is now SAFE
Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only
L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would
overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a
path renders" has exactly one definition. It asserts: every ${...} on a substituted path
is defined-or-escaped (comments included); no default-operator form on an undefined name;
no unparseable expansion; the rendering path actually substitutes cluster-vars; every
overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing.
WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to
touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 —
so an edit to a substituted manifest under apps/ or platform/ would have filtered this
check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh.
Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form,
missing substituteFrom, un-wired gate); each fails as intended, baseline green.
STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted
paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst
--strict` exits 0 against cluster-vars base defaults. Negative control confirms the
incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict
exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped
here — that is a live-cluster change and the operator's call.
Escapes the six comment hazards that stood in the way (apps/vms/unifi x4,
platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own
contract is that $${...} emits a literal ${...}, escaping them is both precise and
self-consistent. kustomize strips these particular comments today, but comment survival is
an implementation detail — a body comment carrying $ WAS observed surviving into built
apps/wave-1 — and a survivor would kill the whole Kustomization.
DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an
indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the
literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place.
Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and
R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the
trigger for removing them.
Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV,
CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are
documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames
are hardcoded elsewhere) and are worth a follow-up.
Gate: MERGE GATE: PASS. 102 assertions, 0 hazards.
Serves: R-ENV-02, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
6e77150ae1 |
fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on ['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored; the coverage rule was NOT narrowed. === THE FALSE GREEN (root cause) === `make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated nothing, and EVERY way of failing to find a base degraded silently to an empty touched set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on `main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's --require gate never runs at all. Green by vacuum. Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base / failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It also now prints the diff base it used. A stale local main warns (widens, never falsely greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites. Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8 throwaway-repo scenarios including the exact false-green case, an already-merged branch, no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and origin/main-preferred-over-stale-main. === NEW L0 STATIC LAYER (conformance/tests/l0-static/) === Where docs/14 actually puts these invariants. Each check.sh really executes offline via `run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds. R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...] values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists — a .example must be referenced by no kustomization and hold only placeholders, and the Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against 7 injected violations, including deleting the negative row and weakening Enforce->Audit. R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to cluster-admin or a wildcard role; the conformance runner's own test identity is scoped and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class policies Enforce and each carry a negative row. Proven against 6 injected violations. R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1, unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green), unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is load-bearing. The real spec repo is never written to. Each check documents what it does NOT cover (runtime decryption, git history, runtime least-privilege, branch protection) rather than overclaiming. status.py now reads check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation. Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 — no sixth scaffold was added. Serves: R-SEC-01, R-SEC-02, R-TEST-01..06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |