jakman85andClaude Opus 5
505083fded
`instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved
the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries
(keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it
meant three simultaneous failovers.
There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0,
clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and
spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware;
spec.replica.primary names a Cluster object in a distributed topology, not a node;
primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never
where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an
imperative status patch by construction and there is nothing to put in git. The obvious
workaround is worse than nothing: pod anti-affinity is only ever
...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with
3 instances on 3 nodes it would leave the third instance unschedulable.
So this commits the part that IS durable — visibility — and rebalances live.
CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw
Service selects only the primary) against the node: kube_endpointslice_endpoints carries
endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither
family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here
(no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at
all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `*-rw`
Service from triggering it, and uses the one metric present for EVERY instance rather than
the cnpg_pg_stat_* family that is missing for nextcloud-db.
A Ready-node gate stops this becoming another permanently-firing alert: with three
clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule
only speaks when the spread is actually achievable, and re-arms when the node returns.
Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after,
with the inner join still resolving all three primaries (quiet because false, not blind);
loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and
firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the
exact version prod runs) and is mutation-checked — dropping the join, dropping the gate
or shortening `for:` each makes a test FAIL.
Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db ->
k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node
has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage
measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one
contiguous burst each. All three back to readyInstances: 3 with
synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum.
The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or
rolling update can re-concentrate the primaries, and now that gets reported instead of
discovered by hand.
Co-Authored-By: Claude Opus 5 <[email protected]>
61 lines
3.1 KiB
Bash
Executable File
61 lines
3.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
###############################################################################
|
|
# platform/cnpg/run-alert-tests.sh — run the promtool unit tests for alerts.yaml.
|
|
#
|
|
# WHY A SCRIPT AND NOT JUST `promtool test rules`. Two impedance mismatches:
|
|
# 1. alerts.yaml is a Kubernetes PrometheusRule CR; promtool wants a bare Prometheus
|
|
# rules file (`groups:` at the top level). So `.spec` is extracted first.
|
|
# 2. runbook_url carries `${BASE_DOMAIN}`, a Flux postBuild substitution that is not
|
|
# resolved in git. promtool does not care what the string says, but leaving a raw
|
|
# `${...}` in a rendered rules file is exactly the kind of thing that gets copied
|
|
# somewhere it DOES matter, so it is substituted with a placeholder here.
|
|
# Both derived files go to a temp dir — nothing generated is left in the worktree, because
|
|
# a generated file sitting next to its source is how a stale copy becomes the thing under
|
|
# test.
|
|
#
|
|
# promtool is NOT in the baked devcontainer toolchain (infra/devcontainer/versions.env).
|
|
# If it is not on PATH this fetches the version that matches the Prometheus actually
|
|
# running on prod, so the unit tests are evaluated by the same engine that will evaluate
|
|
# the rules for real. Override with PROMTOOL=/path/to/promtool.
|
|
#
|
|
# ./run-alert-tests.sh
|
|
###############################################################################
|
|
set -euo pipefail
|
|
|
|
# Match the Prometheus running on prod: quay.io/prometheus/prometheus:v3.14.0-distroless
|
|
# (`kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`).
|
|
PROMETHEUS_VERSION="${PROMETHEUS_VERSION:-3.14.0}"
|
|
|
|
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
tmp="$(mktemp -d)"
|
|
trap 'rm -rf "$tmp"' EXIT
|
|
|
|
promtool="${PROMTOOL:-$(command -v promtool || true)}"
|
|
if [[ -z "$promtool" ]]; then
|
|
echo "==> promtool not on PATH; fetching v${PROMETHEUS_VERSION} (matches prod Prometheus)"
|
|
tarball="prometheus-${PROMETHEUS_VERSION}.linux-amd64"
|
|
curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/${tarball}.tar.gz" \
|
|
| tar -xz -C "$tmp" --strip-components=1 "${tarball}/promtool"
|
|
promtool="$tmp/promtool"
|
|
fi
|
|
# Absolutised because step 3 runs promtool from inside $tmp — a relative PROMTOOL= would
|
|
# break there, and it did during development.
|
|
promtool="$(cd "$(dirname "$promtool")" && pwd)/$(basename "$promtool")"
|
|
echo "==> $("$promtool" --version 2>&1 | head -1)"
|
|
|
|
# 1. PrometheusRule CR -> bare Prometheus rules file, with ${BASE_DOMAIN} resolved.
|
|
sed 's/\${BASE_DOMAIN}/base-domain.invalid/g' "$here/alerts.yaml" \
|
|
| python3 -c 'import sys,yaml; yaml.safe_dump(yaml.safe_load(sys.stdin)["spec"], sys.stdout, sort_keys=False, width=10**6)' \
|
|
> "$tmp/alerts.rules.yaml"
|
|
cp "$here/alerts.promtool-test.yaml" "$tmp/"
|
|
|
|
# 2. Syntax gate first — a rules file that does not parse would make every unit test
|
|
# "pass" vacuously by finding no rules at all.
|
|
echo "==> promtool check rules"
|
|
"$promtool" check rules "$tmp/alerts.rules.yaml"
|
|
|
|
# 3. The unit tests.
|
|
echo "==> promtool test rules"
|
|
cd "$tmp"
|
|
"$promtool" test rules alerts.promtool-test.yaml
|