8 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	78d0f1b0f6	feat(ci): promtool in the pinned toolchain, one alert-rule harness, and the gaps declared ... The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was missing. Hours later a second, independent harness landed for the Velero rules (platform/monitoring/tests/) with its own fallback — a `podman run` of quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and coverage that looked present and was enforced nowhere: the exact failure mode that had just produced a permanently-firing CNPG alert and a structurally inert Velero alert. PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the Prometheus prod actually runs — verified against the live cluster, not assumed: kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}' -> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless) ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml` beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's manifests (monitoring). It writes the flattened rules under whatever name each test file's own `rule_files:` declares, so no test file needs to know the harness exists, and it `promtool check rules` first — a rules file that does not parse would make every unit test pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim placeholders, with the trade-off recorded there — the hazard it guarded against is addressed by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and the rule under test. NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container image. Not because a pinned download is unusual here (the Containerfile downloads every tool it bakes) but because those two decided the version, the network and the machine SILENTLY, at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no promtool" into a green run, which is how the toolchain stayed without one. Missing promtool is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2. ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The `prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted: it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the declarative replacement gives what a hand-written stage cannot — selection by `conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence, and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3. The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule would otherwise be gated by nothing. The check asserts the pins and their lockstep, that the image bakes and records promtool, that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting a synthetic rule through it twice with one expectation mutated. MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of 58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only register, printed with counts every run (13 alert files nothing tests at all; 35 named alerts inside covered files that nothing asserts on), and it cannot grow: a new PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one genuine environment gap, promtool being absent. PROVEN, in the rebuilt image (promtool 3.14.0): * both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS. * the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files. * NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it red (pending expected at 25m/29m, got firing), reverted. Adding an untested PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet, removed. The harness's own mutation self-test runs on every invocation. * the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99 --include-scope whole-repo --print-selection` lists alert-rules-unit-tested. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 00:02:27 -05:00
jakman85andClaude Opus 5	505083fded	feat(cnpg): alert when >1 CNPG primary shares a node — the drift CNPG cannot prevent ... `instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries (keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it meant three simultaneous failovers. There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0, clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware; spec.replica.primary names a Cluster object in a distributed topology, not a node; primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an imperative status patch by construction and there is nothing to put in git. The obvious workaround is worse than nothing: pod anti-affinity is only ever ...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with 3 instances on 3 nodes it would leave the third instance unschedulable. So this commits the part that IS durable — visibility — and rebalances live. CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw Service selects only the primary) against the node: kube_endpointslice_endpoints carries endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here (no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `*-rw` Service from triggering it, and uses the one metric present for EVERY instance rather than the cnpg_pg_stat_* family that is missing for nextcloud-db. A Ready-node gate stops this becoming another permanently-firing alert: with three clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule only speaks when the spread is actually achievable, and re-arms when the node returns. Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after, with the inner join still resolving all three primaries (quiet because false, not blind); loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the exact version prod runs) and is mutation-checked — dropping the join, dropping the gate or shortening `for:` each makes a test FAIL. Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db -> k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one contiguous burst each. All three back to readyInstances: 3 with synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum. The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or rolling update can re-concentrate the primaries, and now that gets reported instead of discovered by hand. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:32:03 -05:00
jakman85andClaude Opus 5	bacb4c61ee	fix(cnpg): the alert told the on-call to run the command that lies ... CNPGNoRecentBaseBackup's own remediation hint said `kubectl get backup`. On this cluster longhorn.io, postgresql.cnpg.io and velero.io each register kind Backup, so the bare short name resolves to Longhorn's empty set and prints 'No resources found' while CNPG backups exist. An on-call following the alert that fires BECAUSE backups look stale would have been told, with authority, that no backups exist at all — and might re-bootstrap a database over good ones. Fully qualify it. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:28:12 -05:00
jakman85andClaude Opus 5	21491f901b	fix(cnpg): scope DR alerts to the CLUSTER, not the pod — stop a permanent false positive ... CNPGNoRecoverabilityPoint has been firing continuously on prod against six pods — keycloak-db-1/-3, nextcloud-db-1/-3, semaphore-db-1/-3 — i.e. exactly the REPLICAS of all three Clusters, while the databases themselves are fine: every Cluster reports firstRecoverabilityPoint=2026-09-07T07:0x, a lastSuccessfulBackup within the last few hours, ContinuousArchiving=True and LastBackupSucceeded=True. Root cause is aggregation scope, not archiving. `cnpg_collector_first_recoverability_point` reads the backup CATALOGUE, which only the primary owns; a replica reports a hard 0 for as long as it is a replica. The metric carries no cnpg.io/cluster label (its label set on prod is container/endpoint/instance/job/namespace/pod), so `max by (namespace, pod)` evaluated each replica as its own subject and an `== 0` comparison then held forever. The rule went from correct to permanently wrong the moment the Clusters went to `instances: 3`; nothing about the expression changed. WHY THIS IS WORSE THAN HAVING NO ALERT. An alert that is always firing has zero information content: its value is identical in the healthy state and in the failure state, so it cannot distinguish them. It is worse than silence in two specific ways. First, it trains the operator to delete the notification on sight, so the day a base backup genuinely stops the seventh identical email is discarded with the other six — the alert actively MASKS the failure it was written to catch, which for a DR alarm means discovering the loss at restore time. Second, it is indistinguishable from a broken alert, so nobody can tell whether the absence of new alerts means health or means the rule rotted. This file exists because a real 38-hour archiving failure told nobody; a rule that cries wolf about six healthy replicas rebuilds exactly the blindness it was supposed to remove. This repo has been here before — see the permanently-firing critical VPNTunnelDown post-mortem in platform/monitoring/alerts.yaml. Fix: group by (namespace, job) and aggregate `pod` away. The CNPG operator creates one PodMonitor per Cluster, named after the Cluster, in the Cluster's namespace, so prometheus-operator derives job="<namespace>/<cluster>" — keycloak/keycloak-db, nextcloud-db/nextcloud-db, semaphore-db/semaphore-db — which identifies the CNPG Cluster uniquely. `job` rather than bare `namespace` deliberately: namespace alone works today (one Cluster per namespace, verified live) but would let two Clusters in one namespace mask each other, since the healthy one's non-zero value would satisfy `max` for both. The alert now states what its summary always claimed — this CLUSTER has no recoverability point — and fires only when no instance of that Cluster reports one. Proven live against prod Prometheus, read-only (promtool query instant): OLD max by (namespace, pod) (...) == 0 -> 6 series, and 6 ALERTS firing NEW max by (namespace, job) (...) == 0 -> empty The new expression is not vacuous: forcing the metric to 0 for one cluster's primary only yields exactly one series, {job="keycloak/keycloak-db"}, so a genuine single-cluster loss of DR still alerts, per cluster. `promtool check rules` passes on all 5 rules. CNPGNoRecentBaseBackup carried the same defect and is fixed the same way. `cnpg_collector_last_available_backup_timestamp` is also primary-only: on prod the primaries read 3.1h/2.1h/0.1h old while all six replicas read 0, i.e. time()-0 = ~56.7 YEARS. It was not firing only because the `> 0` guard was load-bearing — a single clause was all that stood between this rule and six more permanent false positives, too thin a margin for a DR alarm. Aggregating `pod` away removes the dependence: `max` takes the primary's timestamp as the cluster's answer, and the guard now means "this cluster has had at least one backup". Verdict unchanged live (old empty, new empty). The other three rules were audited and deliberately left on `by (namespace, pod)`: * CNPGWALArchiveBacklog — every instance reports archive_status and the shape is `> 0`, so a replica at 0 never fires and no pod can mask another; `pod` names the instance whose archiver is stuck. * CNPGWALVolumeNotRecycling — per-pod is REQUIRED, not merely safe: aggregating it away would compare one instance's WAL count against another's max_wal_size. * CNPGMetricsCollectorErroring — unchanged in meaning AND grouping. Its grouping is already correct: cnpg_last_error is genuinely per-instance, and prod proves it discriminates rather than tracking role — all three nextcloud-db pods report 1 while all six keycloak/semaphore pods report 0. It is a true positive (the nextcloud 42501 CONNECT grant) owned by separate work; nothing here silences or rescopes it. The general rule, recorded in the file: `== 0`/staleness shapes must aggregate `pod` away or every structurally-silent pod fires forever; `> 0` shapes are safe per-pod and keep the instance name. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:09:53 -05:00
jakman85andClaude Opus 5	6a8953e288	feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing ... Minimal prod delivery, authored directly on main (= prod's live revision d1a6628) rather than by merging work/prod-standup-campaign. That branch carries the DNS apex inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun rewriting apex records dev owns while the non-k3s prod still serves 443 there. Operator sequencing is "dev migrates first". Nothing under clusters/ is touched here at all, which is what makes this safe to land now. Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of platform/velero/ and the CNPG destinationPath changes. Those land as feat/r2-backup-prod together with the endpoint and bucket knobs. WHAT THIS FIXES 1. Nothing was watching CNPG. All three prod databases have run since bootstrap with ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed anywhere. Because archive_mode=on and the archive command fails, Postgres refuses to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within the hour, and evaluates to 3 firing series on prod today. Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because nextcloud-db's exporter cannot log in to its own database (permission denied for database "nextcloud", both clusters) so the pg_stat_archiver family does not exist for it — the obvious metric would have been blind to one database in three. 2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to job="kubelet" (the apiserver copy of the same series reports namespace=default), and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report available=0 forever and would otherwise have fired critical permanently. 3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment a node carries the egress-gateway label, with dev's behaviour unchanged today. 4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored LITERALLY and the import still reports success, then sits at Done=True and never re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was green. The CronJob is the level-triggered correction; two alerts make its silence observable. Repaired live already — this stops it recurring. Also: the R-NET-04 conformance scan descended into nested agent worktrees under .claude/, counting 4 egress policies where 1 is authored. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 00:59:22 -05:00
jakman85andClaude Opus 4.8	cbc2301b30	docs: reconcile docs/comments with post-upgrade cluster state ... Text-only cleanup after the upgrade campaign. Updates stale version strings, prose, and comments to the verified current facts; no Kubernetes behavior or version pins changed except the kubectl toolchain bump (below). Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie, Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB, dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18, Grafana 13.2.0. - Version strings across infra/ and platform/ READMEs + comments - kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew) - Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh) - COMPLETED/historical banners on the three executed plan docs - Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges - Appended dated DECISIONS-LOG entry (2026-08-30) superseding the 2026-08-26 LIVE STATE line Left as revalidation items (not blind-edited): external-snapshotter v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1, conformance ImageVolume feature-gate assertions under 1.36. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-30 15:13:12 -05:00
jakman85andClaude Opus 4.8	19a368b237	Component catch-up batch 1: external-dns 1.21, oauth2-proxy 10.x, Alloy 1.x, CNPG in-place flag ... Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-30 11:21:55 -05:00
jakman85	ce83c6af3b	WIP keycloak stack (gate probe)	2026-08-24 19:52:25 -05:00