2 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	78d0f1b0f6	feat(ci): promtool in the pinned toolchain, one alert-rule harness, and the gaps declared ... The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was missing. Hours later a second, independent harness landed for the Velero rules (platform/monitoring/tests/) with its own fallback — a `podman run` of quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and coverage that looked present and was enforced nowhere: the exact failure mode that had just produced a permanently-firing CNPG alert and a structurally inert Velero alert. PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the Prometheus prod actually runs — verified against the live cluster, not assumed: kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}' -> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless) ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml` beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's manifests (monitoring). It writes the flattened rules under whatever name each test file's own `rule_files:` declares, so no test file needs to know the harness exists, and it `promtool check rules` first — a rules file that does not parse would make every unit test pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim placeholders, with the trade-off recorded there — the hazard it guarded against is addressed by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and the rule under test. NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container image. Not because a pinned download is unusual here (the Containerfile downloads every tool it bakes) but because those two decided the version, the network and the machine SILENTLY, at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no promtool" into a green run, which is how the toolchain stayed without one. Missing promtool is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2. ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The `prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted: it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the declarative replacement gives what a hand-written stage cannot — selection by `conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence, and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3. The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule would otherwise be gated by nothing. The check asserts the pins and their lockstep, that the image bakes and records promtool, that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting a synthetic rule through it twice with one expectation mutated. MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of 58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only register, printed with counts every run (13 alert files nothing tests at all; 35 named alerts inside covered files that nothing asserts on), and it cannot grow: a new PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one genuine environment gap, promtool being absent. PROVEN, in the rebuilt image (promtool 3.14.0): * both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS. * the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files. * NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it red (pending expected at 25m/29m, got firing), reverted. Adding an untested PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet, removed. The harness's own mutation self-test runs on every invocation. * the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99 --include-scope whole-repo --print-selection` lists alert-rules-unit-tested. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 00:02:27 -05:00
jakman85andClaude Opus 5	505083fded	feat(cnpg): alert when >1 CNPG primary shares a node — the drift CNPG cannot prevent ... `instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries (keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it meant three simultaneous failovers. There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0, clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware; spec.replica.primary names a Cluster object in a distributed topology, not a node; primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an imperative status patch by construction and there is nothing to put in git. The obvious workaround is worse than nothing: pod anti-affinity is only ever ...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with 3 instances on 3 nodes it would leave the third instance unschedulable. So this commits the part that IS durable — visibility — and rebalances live. CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw Service selects only the primary) against the node: kube_endpointslice_endpoints carries endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here (no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `*-rw` Service from triggering it, and uses the one metric present for EVERY instance rather than the cnpg_pg_stat_* family that is missing for nextcloud-db. A Ready-node gate stops this becoming another permanently-firing alert: with three clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule only speaks when the spread is actually achievable, and re-arms when the node returns. Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after, with the inner join still resolving all three primaries (quiet because false, not blind); loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the exact version prod runs) and is mutation-checked — dropping the join, dropping the gate or shortening `for:` each makes a test FAIL. Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db -> k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one contiguous burst each. All three back to readyInstances: 3 with synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum. The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or rolling update can re-concentrate the primaries, and now that gets reported instead of discovered by hand. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:32:03 -05:00