jakman85
4865734c5f
VeleroNoRecentBackup could not see the failure that actually happened:
velero-daily sat paused having never produced a successful backup, so
velero_backup_last_successful_timestamp{schedule=...} did not exist and
time() - <absent series> yields no series. Prod had no volume backups and
nothing alerted, further masked by a resources-only schedule succeeding
daily. No velero_* metric describes a Schedule OBJECT, proven by scraping a
throwaway paused Schedule and finding zero lines, so this exposes the CRs
via kube-state-metrics customresource_state and keys on
defaultVolumesToFsBackup — the property, not a name list. Adds the repo's
first prometheus rule-unit-test harness, mutation-checked both ways.
151 lines
9.3 KiB
Markdown
151 lines
9.3 KiB
Markdown
# `ci/` — portable merge gate (runnable today, without Gitea)
|
|
|
|
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
|
|
|
|
The same merge gate the Gitea Actions workflow runs
|
|
([`.gitea/workflows/`](../.gitea/workflows/README.md)), as plain scripts + a Makefile so the gates are
|
|
runnable **now**, before any Gitea host or cluster exists. CI and a laptop call the *identical*
|
|
scripts, so `make -C ci gate` == the CI merge gate.
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
make -C ci gate # the single command: run every stage + the R-TEST-03 decision
|
|
make -C ci help # list targets
|
|
make -C ci tools # which pinned tools are installed locally
|
|
make -C ci touched # print the touched-requirement subset for this branch vs main
|
|
```
|
|
|
|
Individual stages:
|
|
|
|
```bash
|
|
make -C ci lint # yaml/lint (no cluster; live now)
|
|
make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now)
|
|
make -C ci l1 # L1 rendering checks (no cluster; live now)
|
|
make -C ci prom-rules # promtool rule-unit tests on alerts (no cluster; live now)
|
|
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
|
|
make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh)
|
|
```
|
|
|
|
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
|
|
(`kyverno test`), and **`conformance/run.sh --layer l0 --include-scope whole-repo`** — the whole-repo
|
|
L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file,
|
|
least-privilege RBAC, spec-immutability enforcement). See `conformance/tests/README.md`.
|
|
|
|
### `--include-scope whole-repo`: why the L0 sub-stage is not purely requirement-scoped
|
|
|
|
R-TEST-03 runs the conformance subset a change **touches**. That has one blind spot, and it is the
|
|
worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the
|
|
coarse path→requirement map keys to *nothing* (`apps/`, `infra/`, most of `platform/`), cannot be
|
|
selected *by* the requirements a change touches — because **the change most likely to break it
|
|
attributes no requirement at all**. Requirement-scoped selection then filters the check out of
|
|
exactly that change.
|
|
|
|
This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety,
|
|
3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded
|
|
`check.sh` paths, three copies of the same `rc 0/3/*` case block — and, worse, a mechanism that
|
|
failed **silently**, because whoever added the next whole-repo invariant had to *remember* to write
|
|
a stage. Those stages are gone. A test now **declares** `conformance.home.k3s/scope: "whole-repo"`
|
|
and the runner unions every declaring test into the run. Adding the annotation is the only step.
|
|
|
|
`rc 3` is preserved exactly as those stages had it: a **SKIP** of a scope-declared check means the
|
|
tree it audits has vanished, so the runner exits `3`, `ci/l0.sh` sets the gate's own `rc=3`, and the
|
|
gate reports SKIPPED — **never a pass**. The mechanism is itself proven, end-to-end and falsifiably,
|
|
by `conformance/tests/l0-static/whole-repo-invariants-wired`; `conformance/lib/common.sh` carries the
|
|
full rationale and the incidents behind it.
|
|
|
|
## Exit-code contract
|
|
|
|
Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIPPED** (cluster-gated
|
|
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
|
|
the gate stays honest in Phase 0 (no fake green).
|
|
|
|
### The touched-requirement subset can also be *undeterminable* (exit 3)
|
|
|
|
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
|
|
order, **verifying each candidate exists** before using it:
|
|
|
|
```
|
|
BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main
|
|
```
|
|
|
|
If no candidate resolves, if there is no merge base, if the diff command fails, or if **HEAD is not
|
|
ahead of the base** (you are sitting on `main`, or the branch is already merged), it prints a loud
|
|
banner, writes **nothing** to stdout, and exits **3** — plus a machine-readable reason code
|
|
(`on-base` / `no-base` / `diff-failed`) to `$TOUCHED_REASON_FILE` when that variable names a path.
|
|
|
|
**Why this is not cosmetic.** Every one of those conditions used to degrade *silently* to an empty
|
|
touched set and exit 0. An empty set means `conformance/run.sh` is invoked with no `--requirements`,
|
|
which means its coverage `--require` gate never runs — so on 2026-08-31 the local gate printed
|
|
`MERGE GATE: PASS` having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
|
|
on real coverage debt. There is no silent narrowing left: the `HEAD~1` last resort and the two-dot
|
|
diff fallback are both gone. Locked down by `ci/tests/touched-requirements-base.test.sh`.
|
|
|
|
`gate.sh` turns exit 3 into an **UNDECIDED** merge decision, never a pass:
|
|
|
|
| reason | headline | exit |
|
|
|---|---|---|
|
|
| `on-base` | `MERGE GATE: NOT APPLICABLE — no branch under gate` | 0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green") |
|
|
| `no-base`, `diff-failed` | `MERGE GATE: BLOCKED` | 1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
|
|
|
|
A stage that needs the subset (`l0.sh`, `l1.sh`) still runs its FULL layer as a superset sanity net
|
|
in that case — nothing goes untested — and says plainly that the *scoped* check did not happen.
|
|
|
|
## Reproducibility (R-TEST-02)
|
|
|
|
- **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of
|
|
truth shared by the local scripts and the CI installer. Bumping a version is a normal
|
|
feature-branch → main change.
|
|
- **Single command, environment-agnostic:** `make -C ci gate` on any machine with the pinned tools.
|
|
- **Deterministic:** no cluster state is assumed; L2/L3 SKIP unless a cluster is explicitly declared.
|
|
|
|
## Files
|
|
|
|
```
|
|
ci/
|
|
Makefile targets: lint · l0 · l1 · prom-rules · conformance · gate · touched · test · tools · versions
|
|
versions.env PINNED tool versions (R-TEST-02) — single source of truth
|
|
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
|
|
yamllint.yaml lint config for the yaml/lint stage
|
|
lint.sh Stage: yaml/lint
|
|
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
|
|
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
|
|
prom-rules.sh Stage: `promtool test rules` over platform/monitoring's PrometheusRules
|
|
(harness: platform/monitoring/tests/). Catches the two alerting bugs no
|
|
other stage can see — permanently-firing and structurally INERT rules.
|
|
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
|
|
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
|
|
exits 3 LOUDLY rather than guessing when the diff base is unreliable
|
|
gate.sh run all stages + the merge decision (the single command)
|
|
tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against
|
|
throwaway git repos via the REPO_ROOT override — no network, no cluster
|
|
README.md this file
|
|
```
|
|
|
|
## Integration seams — entrypoints owned by OTHER agents
|
|
|
|
This gate is an **orchestrator**. It does not implement schema validation, Kyverno policies, or the
|
|
conformance runner — those are other workstreams' deliverables. Each seam is **env-overridable** so
|
|
the owning agent / PM can reconcile the real entrypoint **without editing these scripts**. Defaults
|
|
below are this workstream's **assumptions** — flagged for the PM to verify at integration.
|
|
|
|
| Seam | Owner / task | Env override | Assumed default | Assumed interface |
|
|
|---|---|---|---|---|
|
|
| ServiceDefinition schema validator | WS-GITOPS · **T-0.3** (`schemas/`) | `SCHEMA_VALIDATE_CMD` | `schemas/validate.sh` | `validate.sh catalog/*.yaml` → exit 0/1; `--version` supported |
|
|
| Kyverno L0 policies + tests | WS-TEST · **T-0.5** (`platform/kyverno/`) | `KYVERNO_POLICY_DIR` | `platform/kyverno` | contains `kyverno-test.yaml` files discoverable by `kyverno test <dir>` |
|
|
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l0\|l1\|l2,l3> [--requirements R-A,R-B]`; artifacts land in `conformance/_artifacts/` |
|
|
| Catalog (ServiceDefinitions) | WS-APPS | `CATALOG_DIR` | `catalog` | `*.yaml` ServiceDefinition files |
|
|
|
|
Artifact locations (git-ignored): JUnit → `results/junit/*.xml`; coverage → `reports/coverage/**`.
|
|
Override with `JUNIT_DIR` / `COVERAGE_DIR` if the runner writes elsewhere.
|
|
|
|
**A seam that is missing or not executable reports SKIPPED (exit 3), not a false pass** — so during
|
|
parallel Phase 0 development the gate is green-honest, and each seam goes live automatically the moment
|
|
its owner delivers a runnable entrypoint.
|
|
|
|
## Cluster gating
|
|
|
|
`CLUSTER_AVAILABLE=true` (env locally, `vars.CLUSTER_AVAILABLE` in Gitea) makes the L2/L3 stage run and
|
|
be **required**; anything else makes it **SKIP** (Phase 0). `REQUIRE_CLUSTER_TESTS=true` (Phase 1+)
|
|
turns any unverified cluster-needing touched requirement into a hard block.
|