Compare commits
4
Commits
9b4744d450
...
78d76fa089
| Author | SHA1 | Date | |
|---|---|---|---|
 jakman85
|
78d76fa089 | ||
|
jakman85
|
9e7d919d12 | ||
 jakman85andClaude Opus 5
|
78bc53afc7 | ||
|
jakman85andClaude Opus 5
|
505083fded |
No files matched your search
@@ -2561,3 +2561,130 @@ own status conditions and find out why the list is lying before acting.
|
||||
**Verified same session:** BSL `default` Available (validated 04:13Z); `velero-daily` unpaused and a
|
||||
backup kicked off immediately; two Failed backups from 2026-09-12 (`prod-proof-20260912`,
|
||||
`prod-vp-probe-20260912`) with orphaned kopia blobs still awaiting owner consent to delete.
|
||||
## 2026-09-13 — CNPG primaries spread across nodes, and the honest answer: **there is no declarative fix** (branch `fix/cnpg-primary-spread`)
|
||||
|
||||
The original ask was *"fix the database primaries being solely on prod 2"*. The instance half of
|
||||
that was already done (`instances: 3` + `affinity.podAntiAffinityType: required` +
|
||||
`primaryUpdateMethod: switchover`, commits `f338e91`/`e8d8601`/`cac413e`): every Cluster now has
|
||||
exactly one pod per node and a synchronous standby, so no single node loss can take a database
|
||||
down. The primary half was **not** done — and worse, that roll had *moved* the concentration
|
||||
rather than removed it:
|
||||
|
||||
```
|
||||
BEFORE (2026-09-13 04:19 UTC) AFTER (04:26 UTC)
|
||||
keycloak-db-2 -> k3s-prod-1 primary keycloak-db-1 -> k3s-prod-2 primary
|
||||
nextcloud-db-2 -> k3s-prod-1 primary nextcloud-db-1 -> k3s-prod-3 primary
|
||||
semaphore-db-2 -> k3s-prod-1 primary semaphore-db-2 -> k3s-prod-1 primary
|
||||
```
|
||||
|
||||
All three primaries had landed on `k3s-prod-1` — the node whose failing drive was just replaced and
|
||||
whose etcd WAL shares a 2011 SSD with Longhorn. Losing it meant three simultaneous failovers.
|
||||
|
||||
### The judgment call, made BEFORE touching anything
|
||||
|
||||
**CNPG 1.30.0 has no primary-placement primitive, and the obvious workaround is worse than
|
||||
nothing.** Established by walking the openAPIV3Schema of the CRD *actually installed on prod*
|
||||
(`clusters.postgresql.cnpg.io/v1`, controller-gen v0.21.0) for
|
||||
`/primar|promot|spread|topolog|prefer.*node|switchover/`, not from memory:
|
||||
|
||||
- `spec.affinity.{nodeSelector,nodeAffinity,tolerations,podAntiAffinityType,topologyKey,
|
||||
additionalPodAffinity,additionalPodAntiAffinity}` — every one is documented as applying to *"all
|
||||
the cluster's pods"*. None is role-aware.
|
||||
- `spec.topologySpreadConstraints` — spreads a Cluster's **own** pods; says nothing about how two
|
||||
independent Clusters' primaries relate.
|
||||
- `spec.replica.primary` — names which **Cluster object** is primary in a distributed
|
||||
(`externalClusters`) topology. Not a node.
|
||||
- `spec.primaryUpdateMethod` / `primaryUpdateStrategy` / `switchoverDelay` / `primaryLease` /
|
||||
`postgresql.promotionTimeout` — govern **how and when** a promotion happens, never **where**.
|
||||
- No other CNPG CRD is about scheduling (`backups`, `clusterimagecatalogs`, `databaseroles`,
|
||||
`databases`, `failoverquorums`, `imagecatalogs`, `poolers`, `publications`, `scheduledbackups`,
|
||||
`subscriptions`).
|
||||
- **`targetPrimary` exists only under `status`, never `spec`** (confirmed against the live schema).
|
||||
So `cnpg promote` is an imperative status patch by construction — there is nothing to write in git.
|
||||
|
||||
And `additionalPodAntiAffinity` against `cnpg.io/instanceRole=primary` fails **twice**: (1) Kubernetes
|
||||
pod anti-affinity is only ever `...IgnoredDuringExecution` — there is no `RequiredDuringExecution`
|
||||
variant — while CNPG promotes by **relabelling an already-running pod**, so a scheduling-time
|
||||
constraint cannot bind a promotion at all; (2) even at scheduling time, with 3 instances on 3 nodes
|
||||
and `required` intra-cluster anti-affinity, excluding the node holding another primary leaves the
|
||||
third instance **unschedulable** — trading blast radius for a permanent `readyInstances: 2`.
|
||||
|
||||
**So option (a) — "accept it, every cluster already survives node loss" — was rejected as
|
||||
incomplete, and a git-only "fix" was rejected as impossible.** What was done instead: reduce the
|
||||
blast radius today, and make the drift **visible** so it is a known cheap runbook step rather than
|
||||
something found by hand months later. The placement below is **current, not durable** — say so.
|
||||
|
||||
### Why the drift is not hypothetical
|
||||
|
||||
It already happened, from a routine declarative change: the `required` anti-affinity roll is
|
||||
precisely what re-promoted all three clusters onto one node. `keycloak-db` reported
|
||||
`Primary promotion time: 2026-09-08` — a re-promotion nobody asked for. No prod outage was staged to
|
||||
"prove" non-durability; the repo's own history is the evidence.
|
||||
|
||||
### The rebalance: two switchovers, not three
|
||||
|
||||
`semaphore-db` was **deliberately left** with its primary on `k3s-prod-1`. One primary per node is
|
||||
the goal, and semaphore is by far the lightest writer (its WAL backlog sat at 3 segments because it
|
||||
writes almost nothing), so it is the right tenant for the node with the known etcd/Longhorn I/O
|
||||
contention. Two switchovers instead of three = one less write interruption.
|
||||
|
||||
**Disruption, measured — not estimated.** A `select txid_current()` probe (a real write: it assigns
|
||||
an xid) ran against each `-rw` Service from a pod that stayed a replica throughout, password fed via
|
||||
stdin so it never reached argv:
|
||||
|
||||
| cluster | target | write outage | detail |
|
||||
|---|---|---|---|
|
||||
| `keycloak-db` (SSO) | `keycloak-db-1` on k3s-prod-2 | **~9.4 s** | last OK 04:20:24.634 → fail burst to 04:20:33.731; healthy at t+30s |
|
||||
| `nextcloud-db` | `nextcloud-db-1` on k3s-prod-3 | **9.43 s** | 1402 samples @44 ms; ONE burst 04:25:46.854 → 04:25:56.287; healthy at t+25s |
|
||||
| `semaphore-db` | *(none — kept on k3s-prod-1)* | **0 s** | stayed `readyInstances: 3` throughout |
|
||||
|
||||
One contiguous failure burst each, no flapping, no second dip. Note the 30 s scrape interval showed
|
||||
**no gap at all** for keycloak-db — the `-rw` endpoint was `ready` at every sample. A ~9 s write
|
||||
outage is invisible to Prometheus at this resolution; only the sub-second probe sees it.
|
||||
|
||||
After: all three `readyInstances: 3`, `Cluster in healthy state`, synchronous replication intact —
|
||||
`synchronous_standby_names = ANY 1 (...)` in force on each primary with both standbys
|
||||
`state=streaming sync_state=quorum`.
|
||||
|
||||
### The durable part: `CNPGPrimariesCoLocated`
|
||||
|
||||
Added to `platform/cnpg/alerts.yaml` (already wired via `platform-config/kustomization.yaml`, so
|
||||
Flux reconciles it on merge — no hand-apply).
|
||||
|
||||
**Neither metric family can answer the question alone**, which is the whole difficulty:
|
||||
`cnpg_*` metrics carry `namespace`/`pod` but **no node**; and `kube_pod_labels` is **empty** on this
|
||||
cluster (kube-state-metrics has no `--metric-labels-allowlist`), so `cnpg.io/instanceRole` is not in
|
||||
Prometheus at all. The rule instead reads **CNPG's own definition of "who is primary"** — the
|
||||
`<cluster>-rw` Service selects the primary and nothing else, and
|
||||
`kube_endpointslice_endpoints` carries both `endpoint_nodename` and `targetref_name`. Two
|
||||
`label_replace`s lift those into `node`/`pod`; `* on (namespace, pod) group_left () cnpg_collector_up`
|
||||
restricts the count to real CNPG instances so an unrelated `*-rw` Service cannot trigger it.
|
||||
`cnpg_collector_up` is the right-hand side deliberately — it exists for **every** instance, unlike
|
||||
the `cnpg_pg_stat_*` family which is missing entirely for `nextcloud-db`.
|
||||
|
||||
**A Ready-node gate keeps it from becoming the next permanently-firing alert.** With three clusters
|
||||
on three nodes, losing a node *necessarily* co-locates two primaries — correct behaviour, not a
|
||||
defect — so the rule only speaks while every node is `Ready`, and re-arms itself when the node
|
||||
returns. `for: 30m` rides out a switchover (seconds) and a full rolling update (minutes).
|
||||
`severity: warning`, like everything else in that file: a resilience posture, never a page.
|
||||
|
||||
**Proven in both directions, because this repo has been burned by both.** Live: the expression
|
||||
returned `{node="k3s-prod-1"} => 3` before, and an empty vector after (while the inner join still
|
||||
resolved all three primaries — quiet because the condition is false, not because the query went
|
||||
blind). Loaded into prod's Prometheus as a throwaway twin (`for: 1m`), it was observed going
|
||||
`pending` at +70 s and `firing` at +130 s, then deleted. And `platform/cnpg/alerts.promtool-test.yaml`
|
||||
(`./run-alert-tests.sh`) pins all four behaviours under promtool 3.14.0 — the exact Prometheus
|
||||
version on prod. **The tests were mutation-checked:** removing the `cnpg_collector_up` join,
|
||||
removing the readiness gate, or shortening `for:` each makes a test *fail*, so they are not vacuous.
|
||||
|
||||
### Debt / follow-ups
|
||||
|
||||
- **Placement is current, not durable.** The next failover, node reboot or CNPG rolling update can
|
||||
re-concentrate the primaries. That is now *reported*, not *prevented*. Rebalancing is a ~9 s
|
||||
write interruption per cluster.
|
||||
- **`promtool` is not in the baked toolchain** (`infra/devcontainer/versions.env`), so
|
||||
`run-alert-tests.sh` fetches it on demand and no CI stage runs the alert unit tests. Adding
|
||||
`PROMTOOL_VERSION` to `ci/versions.env` + the Containerfile would let `make -C ci gate` run them.
|
||||
- **No `runbooks/` tree exists** anywhere in the repo, yet 20+ alerts carry `runbook_url`.
|
||||
`cnpg-primary-spread` follows that existing (aspirational) convention; the rule's `description`
|
||||
carries the full actionable procedure so the missing page is not load-bearing.
|
||||
@@ -35,6 +35,33 @@ spec:
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# PersistentVolume qbittorrent-downloads - static external NFS (bulk tier)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: qbittorrent-downloads
|
||||
labels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
spec:
|
||||
capacity:
|
||||
storage: 9Ti
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: ''
|
||||
mountOptions:
|
||||
- nfsvers=4.1
|
||||
- hard
|
||||
- noatime
|
||||
- timeo=600
|
||||
- retrans=2
|
||||
nfs:
|
||||
server: ${BULK_NFS_SERVER}
|
||||
path: /mnt/Storage/plex
|
||||
claimRef:
|
||||
namespace: qbittorrent
|
||||
name: qbittorrent-downloads
|
||||
---
|
||||
# PVC qbittorrent-config (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
@@ -51,6 +78,23 @@ spec:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
# PVC qbittorrent-downloads (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: qbittorrent-downloads
|
||||
namespace: qbittorrent
|
||||
labels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
storageClassName: ''
|
||||
volumeName: qbittorrent-downloads
|
||||
resources:
|
||||
requests:
|
||||
storage: 9Ti
|
||||
---
|
||||
# Deployment (R-PLAT-02)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
@@ -81,9 +125,17 @@ spec:
|
||||
image: lscr.io/linuxserver/qbittorrent:5.1.2
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
env:
|
||||
- name: PUID
|
||||
value: '1000'
|
||||
- name: PGID
|
||||
value: '3000'
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: downloads
|
||||
mountPath: /downloads
|
||||
subPath: downloads
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
@@ -149,10 +201,15 @@ spec:
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
securityContext:
|
||||
fsGroup: 3000
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
claimName: qbittorrent-config
|
||||
- name: downloads
|
||||
persistentVolumeClaim:
|
||||
claimName: qbittorrent-downloads
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
|
||||
+95
-14
@@ -4,19 +4,18 @@
|
||||
# internal download client | egress VPN + kill-switch | config on Longhorn | auth forward-auth
|
||||
# (bare web UI, no native OIDC).
|
||||
#
|
||||
# BULK DOWNLOADS TIER — OPEN PREREQUISITE, intentionally NOT declared here (mirrors plex + sonarr/radarr):
|
||||
# docs/13 row 13 places qBittorrent's downloads/staging volume on the `bulk` tier (large, non-3x-
|
||||
# replicated media handoff to the arr stack). The `bulk` StorageClass exists
|
||||
# (platform/longhorn/storageclasses.yaml) and both the schema and renderer support `class: bulk`, BUT
|
||||
# it is a NON-PROVISIONING STUB pending OQ-2 (its NFS/SMB backing is REPLACE-ME and the NFS CSI driver
|
||||
# is not installed) — a `bulk` PVC would never bind and would leave the pod Pending. Declaring it now
|
||||
# would also pull the bulk-tier requirement into the change's "touched" set, whose in-cluster
|
||||
# conformance test is not yet scaffolded, blocking the merge gate. So the downloads volume is DEFERRED
|
||||
# and this onboarding ships config-only. Once OQ-2 is resolved and `bulk` provisions, attach downloads
|
||||
# with a one-line add under storage:
|
||||
# - name: downloads
|
||||
# class: bulk # external mount; size omitted (sized outside the cluster)
|
||||
# then re-render. See the WS-APPS report for the full bulk-downloads finding.
|
||||
# BULK DOWNLOADS TIER — DELIVERED 2026-09-12 (the deferral below is CLOSED; kept for the record).
|
||||
# This entry used to ship config-only, with a comment promising to attach the downloads volume "once
|
||||
# OQ-2 is resolved". OQ-2 was resolved for STATIC pre-existing datasets (schema 1.5.0 / renderer
|
||||
# `class: bulk` + `export` -> a static, pre-bound NFS PV), and plex/sonarr/radarr/nextcloud all took
|
||||
# their media mounts — but qBittorrent's was never added, so the DOWNLOAD CLIENT was the one member
|
||||
# of the chain with no media volume at all. Measured on prod 2026-09-12:
|
||||
# Session\DefaultSavePath=/downloads/ Session\TempPath=/downloads/incomplete/
|
||||
# Downloads\SavePath=/downloads/ `ls /downloads` -> No such file or directory
|
||||
# i.e. the app was configured to write a path that DID NOT EXIST, in a container with no volume there.
|
||||
# Anything it did manage to create would have been invisible to sonarr/radarr, lost on every pod
|
||||
# recreation, and filling the NODE's ephemeral disk in the meantime. The `downloads` entry under
|
||||
# storage: below closes that. See docs/BULK-NFS-STORAGE.md and the bulk-tier L0 conformance test.
|
||||
#
|
||||
# VPN-class (docs/13 row 13; docs/03 "VPN-class set"): ALL of this pod's public-internet egress
|
||||
# (peer/tracker/DHT traffic) MUST leave via the node PIA tunnel, never the WAN. `egress: vpn` renders
|
||||
@@ -32,6 +31,27 @@ name: qbittorrent
|
||||
kind: container
|
||||
source:
|
||||
image: lscr.io/linuxserver/qbittorrent:5.1.2 # pinned stable tag (R-UPD-01); update.range governs advancement
|
||||
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
|
||||
# LOAD-BEARING FOR THE BULK MOUNT, not cosmetic — exactly as on sonarr/radarr. Measured on prod
|
||||
# 2026-09-12 BEFORE this change: `id` inside the container reported the qbittorrent-nox process as
|
||||
# uid=911 gid=911, and /mnt/Storage/plex/downloads is 0755 owned 1000:1000. 0755 grants write to the
|
||||
# OWNER ONLY, so uid 911 would have mounted the export cleanly and then failed every write with
|
||||
# EACCES — the "mounts fine, cannot write" failure the sonarr entry warns about at length.
|
||||
env:
|
||||
- name: PUID
|
||||
value: "1000" # owner uid of /mnt/Storage/plex/downloads (and of every dir under the export)
|
||||
- name: PGID
|
||||
value: "3000" # the export's dominant group, matching sonarr/radarr: files this writes stay
|
||||
# readable to the EXISTING non-k3s consumers of the same dataset, and land
|
||||
# group-owned the same way the arrs' own writes do
|
||||
podSecurityContext:
|
||||
# For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
|
||||
# on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
|
||||
# the kubelet applies it, so /config becomes group-writable by the now-uid-1000 process without
|
||||
# depending on the image chowning it. (This image DOES chown: init-qbittorrent-config runs
|
||||
# `lsiown -R abc:abc /config`. Belt and braces — the live /config was 911:911 before this change,
|
||||
# and losing write access to it would lose the WebUI password hash and the session state.)
|
||||
fsGroup: 3000
|
||||
exposure: internal # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
|
||||
egress: vpn # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
|
||||
storage:
|
||||
@@ -49,7 +69,68 @@ storage:
|
||||
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
|
||||
# The live /config contents were copied onto each PVC before this landed,
|
||||
# so API keys and credentials other apps reference were preserved.
|
||||
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
|
||||
# THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
|
||||
# (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
|
||||
# /downloads, which is precisely where this app is ALREADY configured to write.
|
||||
#
|
||||
# WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
|
||||
# * LEAST PRIVILEGE, AND IT COSTS NOTHING HERE. A download client has no business writing the
|
||||
# LIBRARY. The arrs need the root mount because THEY hardlink downloads/ -> tv/ or movies/ and
|
||||
# link() is a same-MOUNT operation (EXDEV otherwise) — their smallest common ancestor genuinely
|
||||
# IS the export root. qBittorrent performs no such cross-directory link: it writes into the
|
||||
# downloads tree and nothing else, so narrowing to that subtree removes movies/, tv/ and photos/
|
||||
# from this pod's mount namespace without removing any operation it actually performs.
|
||||
# * HARDLINKS STILL WORK, because they are not done here. The arrs still see ONE mount containing
|
||||
# both /media/downloads and /media/tv, so their link() stays intra-mount. Proven live 2026-09-12:
|
||||
# a file created from this pod under /downloads was hardlinked from the sonarr pod between
|
||||
# /media/downloads/... and /media/tv/... with st_nlink 1 -> 2 and no EXDEV.
|
||||
# * IT IS ONE MOUNT, so the L0 test `bulk-tier-distinct-from-replicated` assertion (6) — no
|
||||
# workload mounts one export twice — holds. That assertion is about TWO mounts inside ONE pod;
|
||||
# a single subPath mount is not the EXDEV trap, two are.
|
||||
# * The narrowing is a CLIENT/kubelet guarantee, stated honestly: the NFS server still exports the
|
||||
# whole dataset to this node, so the containment is the mount namespace, not the server's ACLs —
|
||||
# the same class of caveat docs/BULK-NFS-STORAGE.md §5 records for plex's readOnly.
|
||||
#
|
||||
# NOT readOnly — this is the one workload whose whole job is WRITING here.
|
||||
#
|
||||
# KNOWN, DELIBERATE SIDE EFFECT ON SHARED STATE: the image's init-qbittorrent-config runs
|
||||
# `lsiown abc:abc /downloads` (non-recursive) whenever /downloads is a mount, so starting this pod
|
||||
# chgrps /mnt/Storage/plex/downloads from 1000:1000 to 1000:3000 on the SERVER. Benign and verified:
|
||||
# the directory is mode 0755, whose group bits (r-x) are identical to its other bits, so no consumer
|
||||
# — k3s or legacy — gains or loses any access. Recorded because it is a write to state shared with
|
||||
# non-cluster consumers, which should never be a surprise.
|
||||
#
|
||||
# PATH AGREEMENT WITH THE ARRS IS *NOT* CLOSED BY THIS CHANGE — see the report/DECISIONS-LOG. This
|
||||
# pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
|
||||
# /media/downloads/<name>. Same inode, different absolute path, so Completed Download Handling needs
|
||||
# ONE Remote Path Mapping per arr (remote host `qbittorrent`, remote path /downloads/, local path
|
||||
# /media/downloads/). That lives in the arrs' own databases (app state on their config PVCs), not in
|
||||
# git, so it is an operator step and is called out rather than silently assumed. The alternative —
|
||||
# repointing qBittorrent's SavePath at /media/downloads and mounting this subPath there instead —
|
||||
# buys path agreement at the cost of editing THIS app's config PVC, which is also app state; it was
|
||||
# not chosen because it changes a running download client's save path, which the deferral above
|
||||
# already shows is easy to get wrong.
|
||||
- name: downloads
|
||||
class: bulk
|
||||
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER} — the SAME export
|
||||
# plex (ro) and sonarr/radarr (rw) mount; one dataset, one set of inodes
|
||||
subPath: downloads # the handoff subtree ONLY: no movies/, no tv/, no photos/
|
||||
mountPath: /downloads # matches the app's EXISTING Session\DefaultSavePath / TempPath, so no
|
||||
# change to config-PVC app state is needed to make this take effect
|
||||
size: 9Ti # ADVISORY ONLY on a static NFS PV (the ZFS pool is the real limit): the
|
||||
# dataset's 8.4 T used rounded up, stated the same way plex/sonarr/radarr
|
||||
# state it so `kubectl get pv` reports something true
|
||||
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02).
|
||||
# STILL EMPTY WITH THE NFS MOUNT ABOVE, AND THAT IS CORRECT, NOT AN
|
||||
# OVERSIGHT. An NFS PersistentVolume is mounted by the NODE KERNEL in the
|
||||
# HOST netns, not by this pod: there is no pod-side socket to 2049, so no
|
||||
# CiliumNetworkPolicy can permit or deny it and none is needed. Verified by
|
||||
# construction — sonarr and radarr have mounted this same export read-write
|
||||
# for weeks and neither has any CNP naming the NFS server or port 2049
|
||||
# (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
|
||||
# egress gateway off this traffic (its selector is a podSelector), and
|
||||
# 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
|
||||
# platform/vpn-egress/egress-steering.yaml, "BULK NFS".
|
||||
ingressRoutes:
|
||||
- host: qbittorrent.${BASE_DOMAIN}
|
||||
port: 8080 # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
|
||||
|
||||
@@ -0,0 +1,324 @@
|
||||
---
|
||||
# promtool unit tests for the CNPGPrimariesCoLocated rule in platform/cnpg/alerts.yaml.
|
||||
#
|
||||
# WHY THIS FILE EXISTS. This repo has been burned in both directions by alerts nobody tested:
|
||||
# a permanently-firing `critical` (the VPNTunnelDown post-mortem in platform/monitoring/alerts.yaml)
|
||||
# and, worse, silence that looked like health (the 2026-09-06 CNPG archiving incident — 38h of
|
||||
# ContinuousArchiving=False with no alert of any kind). CNPGPrimariesCoLocated is a JOIN across two
|
||||
# metric families with a label_replace on each side and a node-readiness gate; "it looks right" is
|
||||
# not evidence that it can fire, and it is certainly not evidence that it can stop. These tests pin
|
||||
# BOTH directions plus the two ways the rule is meant to stay quiet.
|
||||
#
|
||||
# HOW TO RUN. `./run-alert-tests.sh` in this directory — it extracts `.spec` from the PrometheusRule
|
||||
# (promtool wants a bare Prometheus rules file, not a Kubernetes CR), substitutes ${BASE_DOMAIN},
|
||||
# and runs promtool against both files in a temp dir. promtool is NOT in the baked devcontainer
|
||||
# toolchain (infra/devcontainer/versions.env); the script downloads the version that matches the
|
||||
# Prometheus actually running on prod (v3.14.0) if it is not already on PATH.
|
||||
#
|
||||
# WHY THE ASSERTIONS USE `ALERTS` AND NOT `exp_alerts`. promtool's alert_rule_test compares
|
||||
# annotations VERBATIM, which would mean pasting this rule's ~1200-character runbook description in
|
||||
# here and re-pasting it on every wording change. Querying the synthetic `ALERTS` series instead
|
||||
# proves exactly what matters — the alert reaches `pending` and then `firing` across the `for: 30m`
|
||||
# boundary, with the right labels — and stays decoupled from annotation prose.
|
||||
#
|
||||
# THE SERIES BELOW ARE REAL SHAPES, not invented ones: every label on
|
||||
# kube_endpointslice_endpoints / cnpg_collector_up / kube_node_status_condition / kube_node_info was
|
||||
# read off live prod Prometheus on 2026-09-12 before being copied here.
|
||||
rule_files:
|
||||
- alerts.rules.yaml
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# 1. THE BAD STATE THAT PROMPTED THIS RULE, exactly as prod looked on 2026-09-12: all three
|
||||
# primaries (keycloak-db-2, nextcloud-db-2, semaphore-db-2) on k3s-prod-1, all nodes Ready.
|
||||
# Asserts the rule CAN fire, that it counts 3 (not 1, which a broken join would give), and
|
||||
# that `for: 30m` is honoured — pending at 29m, firing at 31m.
|
||||
- interval: 1m
|
||||
name: three primaries on one node fires after 30m
|
||||
input_series:
|
||||
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.142",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-2",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.126",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="semaphore-db",endpointslice="semaphore-db-rw-2xl8g",targetref_kind="Pod",targetref_name="semaphore-db-2",targetref_namespace="semaphore-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",address="10.0.0.201",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
# All nine instances publish cnpg_collector_up — the join must pick out only the three that
|
||||
# the -rw EndpointSlices point at, so the replicas are present here on purpose.
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-1",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-3",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-1",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-2",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-3",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-1",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-2",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-3",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# The join resolves each -rw endpoint to (namespace, primary pod, node) — the drill-down an
|
||||
# operator actually wants. All three land on k3s-prod-1.
|
||||
- expr: |
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{namespace="keycloak", node="k3s-prod-1", pod="keycloak-db-2"}'
|
||||
value: 1
|
||||
- labels: '{namespace="nextcloud-db", node="k3s-prod-1", pod="nextcloud-db-2"}'
|
||||
value: 1
|
||||
- labels: '{namespace="semaphore-db", node="k3s-prod-1", pod="semaphore-db-2"}'
|
||||
value: 1
|
||||
# ... and the alert's own expression counts 3 of them on that one node.
|
||||
- expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
) > 1
|
||||
and on ()
|
||||
count(kube_node_status_condition{condition="Ready", status="true"} == 1) == count(kube_node_info)
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 3
|
||||
# NOT PERMANENTLY FIRING, part 1: `for: 30m` really is waited out.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 29m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="pending",node="k3s-prod-1",severity="warning"}'
|
||||
value: 1
|
||||
# NOT PERMANENTLY INERT: it does reach `firing`.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 31m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="firing",node="k3s-prod-1",severity="warning"}'
|
||||
value: 1
|
||||
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# 2. THE GOOD STATE — one primary per node, which is what the rebalancing switchovers on
|
||||
# 2026-09-12 produced. The rule must be SILENT here, and silent for the right reason: the
|
||||
# first assertion proves the join still resolves all three primaries (an alert that is quiet
|
||||
# because its query returns nothing is the failure mode this repo keeps hitting), the second
|
||||
# proves the >1 test then finds no over-loaded node.
|
||||
- interval: 1m
|
||||
name: one primary per node is silent
|
||||
input_series:
|
||||
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-1",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-2",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-1",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-3",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="semaphore-db",endpointslice="semaphore-db-rw-2xl8g",targetref_kind="Pod",targetref_name="semaphore-db-2",targetref_namespace="semaphore-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-1",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-1",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="semaphore-db",pod="semaphore-db-2",cluster="semaphore-db",container="postgres",job="semaphore-db/semaphore-db"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# Quiet for the RIGHT reason: the join is alive and finds all three primaries, one per node.
|
||||
- expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
)
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 1
|
||||
- labels: '{node="k3s-prod-2"}'
|
||||
value: 1
|
||||
- labels: '{node="k3s-prod-3"}'
|
||||
value: 1
|
||||
# ... and no alert is ever raised, at any point in the hour.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 31m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 59m
|
||||
exp_samples: []
|
||||
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# 3. A NODE IS DOWN, so co-location is UNAVOIDABLE and correct — the rule must not nag during
|
||||
# an outage. k3s-prod-3 is NotReady for the first 10 minutes and nextcloud-db has failed over
|
||||
# onto k3s-prod-1, which now holds two primaries. Then the node comes back at t=11m and the
|
||||
# co-location is NOT cleaned up, which is precisely the drift worth reporting: the rule must
|
||||
# RE-ARM by itself and fire 30m later. This is the test that keeps the readiness gate from
|
||||
# turning into a permanent mute.
|
||||
- interval: 1m
|
||||
name: node down suppresses, then re-arms when the node returns
|
||||
input_series:
|
||||
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="nextcloud-db",endpointslice="nextcloud-db-rw-dxhx7",targetref_kind="Pod",targetref_name="nextcloud-db-2",targetref_namespace="nextcloud-db",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x90'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x90'
|
||||
- series: 'cnpg_collector_up{namespace="nextcloud-db",pod="nextcloud-db-2",cluster="nextcloud-db",container="postgres",job="nextcloud-db/nextcloud-db"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
||||
values: '1x90'
|
||||
# k3s-prod-3: NotReady (0) for t=0..10, Ready (1) from t=11 onwards.
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
||||
values: '0x10 1x79'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x90'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x90'
|
||||
promql_expr_test:
|
||||
# Two primaries genuinely do share k3s-prod-1 at t=5m...
|
||||
- expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
) > 1
|
||||
eval_time: 5m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-1"}'
|
||||
value: 2
|
||||
# ... but the readiness gate suppresses the alert entirely while a node is down: not even pending.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 5m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 10m
|
||||
exp_samples: []
|
||||
# Node back at t=11m; the `for` timer starts from scratch, so still only pending at t=25m.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 25m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="pending",node="k3s-prod-1",severity="warning"}'
|
||||
value: 1
|
||||
# RE-ARMED: 30m of full-cluster health with the co-location still there -> firing.
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 45m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CNPGPrimariesCoLocated",alertstate="firing",node="k3s-prod-1",severity="warning"}'
|
||||
value: 1
|
||||
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# 4. THE FALSE-POSITIVE GUARD. `endpointslice=~".+-rw-.+"` on its own is a loose match — any
|
||||
# Service in the cluster named `something-rw` produces an EndpointSlice that matches it. Two
|
||||
# such endpoints on one node must NOT be counted as CNPG primaries, which is what the
|
||||
# `* on (namespace, pod) group_left () cnpg_collector_up` factor is for. Without that factor
|
||||
# this test fires, so it is the thing that pins the join instead of a bare regex.
|
||||
- interval: 1m
|
||||
name: non-CNPG rw services are not counted
|
||||
input_series:
|
||||
- series: 'kube_endpointslice_endpoints{namespace="default",endpointslice="myapp-rw-aaaaa",targetref_kind="Pod",targetref_name="myapp-0",targetref_namespace="default",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_endpointslice_endpoints{namespace="default",endpointslice="other-rw-bbbbb",targetref_kind="Pod",targetref_name="other-0",targetref_namespace="default",endpoint_nodename="k3s-prod-1",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
# One real CNPG primary, on a different node — present so the test cannot pass merely because
|
||||
# the whole query is empty.
|
||||
- series: 'kube_endpointslice_endpoints{namespace="keycloak",endpointslice="keycloak-db-rw-k9phd",targetref_kind="Pod",targetref_name="keycloak-db-2",targetref_namespace="keycloak",endpoint_nodename="k3s-prod-2",ready="true",serving="true",terminating="false",pod="kube-prometheus-stack-kube-state-metrics-0",job="kube-state-metrics"}'
|
||||
values: '1x60'
|
||||
- series: 'cnpg_collector_up{namespace="keycloak",pod="keycloak-db-2",cluster="keycloak-db",container="postgres",job="keycloak/keycloak-db"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-1",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-2",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_status_condition{node="k3s-prod-3",condition="Ready",status="true"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-2"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_node_info{node="k3s-prod-3"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
# Only the CNPG primary survives the join — the two decoy `*-rw` endpoints are dropped.
|
||||
- expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
)
|
||||
eval_time: 10m
|
||||
exp_samples:
|
||||
- labels: '{node="k3s-prod-2"}'
|
||||
value: 1
|
||||
- expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
|
||||
eval_time: 45m
|
||||
exp_samples: []
|
||||
@@ -265,3 +265,116 @@ spec:
|
||||
summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}"
|
||||
description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter"
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# BLAST RADIUS: MORE THAN ONE CNPG PRIMARY ON A SINGLE NODE.
|
||||
#
|
||||
# WHY THIS IS AN ALERT AND NOT A SPEC FIELD (investigated against the live CRD, 2026-09-12).
|
||||
# The operator's original ask was "the database primaries are all on one node". `instances: 3`
|
||||
# + `affinity.podAntiAffinityType: required` fixed the INSTANCE spread — every Cluster now has
|
||||
# exactly one pod per node, so no node loss can take a whole database down. It did NOT fix the
|
||||
# PRIMARY spread: after that roll all three primaries (keycloak-db-2, nextcloud-db-2,
|
||||
# semaphore-db-2) were on k3s-prod-1 at once, so losing that one node meant three simultaneous
|
||||
# failovers — survivable, but the worst single-node blast radius the cluster can have.
|
||||
#
|
||||
# THERE IS NO DECLARATIVE FIX. Enumerated exhaustively from the CRD actually installed here
|
||||
# (cloudnative-pg 1.30.0, clusters.postgresql.cnpg.io/v1, controller-gen v0.21.0) by walking
|
||||
# the whole openAPIV3Schema for /primar|promot|spread|topolog|prefer.*node|switchover/:
|
||||
# * spec.affinity.{nodeSelector,nodeAffinity,tolerations,podAntiAffinityType,topologyKey,
|
||||
# additionalPodAffinity,additionalPodAntiAffinity} — all documented as applying to "all
|
||||
# the cluster's pods". None is role-aware: there is no way to say "this constraint is for
|
||||
# whichever pod is currently primary".
|
||||
# * spec.topologySpreadConstraints — spreads a Cluster's OWN pods across domains. Within one
|
||||
# Cluster that is the job podAntiAffinityType already does; it says nothing about how two
|
||||
# INDEPENDENT Clusters' primaries relate.
|
||||
# * spec.replica.primary — names which CLUSTER OBJECT is primary in a distributed topology
|
||||
# (externalClusters / cross-k8s replica clusters). Not a node, not applicable here.
|
||||
# * spec.primaryUpdateMethod / primaryUpdateStrategy / switchoverDelay / primaryLease /
|
||||
# postgresql.promotionTimeout — govern HOW and WHEN a promotion happens, never WHERE the
|
||||
# primary lands.
|
||||
# Nothing else in the v1 schema mentions primary placement, and no other CNPG CRD is about
|
||||
# scheduling (backups, clusterimagecatalogs, databaseroles, databases, failoverquorums,
|
||||
# imagecatalogs, poolers, publications, scheduledbackups, subscriptions).
|
||||
#
|
||||
# AND THE OBVIOUS HACK IS WORSE THAN NOTHING. "additionalPodAntiAffinity against pods labelled
|
||||
# cnpg.io/instanceRole=primary in other namespaces" fails twice over. (1) Kubernetes pod
|
||||
# (anti-)affinity is only ever *DuringSchedulingIgnoredDuringExecution* — the schema has no
|
||||
# RequiredDuringExecution variant — and CNPG promotes by relabelling an ALREADY-RUNNING pod
|
||||
# (`cnpg.io/instanceRole: replica` -> `primary`), with no rescheduling. A scheduling-time
|
||||
# constraint therefore cannot bind a promotion at all. (2) Even at scheduling time it would be
|
||||
# actively harmful: with 3 instances, 3 nodes and `required` intra-cluster anti-affinity,
|
||||
# excluding the node that holds another cluster's primary leaves the third instance
|
||||
# UNSCHEDULABLE — trading a blast-radius concern for a permanently degraded readyInstances: 2.
|
||||
#
|
||||
# SO PRIMARY PLACEMENT IS OPERATIONAL, NOT DECLARATIVE. `cnpg promote` (which patches
|
||||
# status.targetPrimary — a STATUS field; the CRD has no spec.targetPrimary, confirmed live)
|
||||
# rebalances today and drifts again on the next failover, node reboot or rolling update.
|
||||
# A switchover that silently un-does itself is not a fix, so what is committed here is the
|
||||
# part that IS durable: the condition is now VISIBLE every time it recurs, and rebalancing is
|
||||
# a known, cheap runbook step instead of something discovered by hand months later.
|
||||
#
|
||||
# WHY THIS PROMQL. Neither family alone can answer the question:
|
||||
# * cnpg_* metrics carry namespace/pod/cluster but NO node label (verified live).
|
||||
# * kube_pod_labels is EMPTY on this cluster — kube-state-metrics has no
|
||||
# --metric-labels-allowlist, so `cnpg.io/instanceRole` is not in Prometheus at all.
|
||||
# The join below instead reads CNPG's OWN definition of "who is primary": the `<cluster>-rw`
|
||||
# Service selects the primary and nothing else, so its EndpointSlice has exactly one endpoint
|
||||
# and kube_endpointslice_endpoints carries both `endpoint_nodename` (the node) and
|
||||
# `targetref_name` (the pod). label_replace lifts those into `node`/`pod`, and the
|
||||
# `* on (namespace, pod) group_left () cnpg_collector_up` factor restricts the count to pods
|
||||
# that really are CNPG instances — so an unrelated Service that happens to be named `*-rw`
|
||||
# cannot trigger this. cnpg_collector_up is the right-hand side deliberately: it is emitted by
|
||||
# the instance manager and exists for EVERY instance, unlike the cnpg_pg_stat_* family which
|
||||
# is missing entirely for nextcloud-db (see CNPGMetricsCollectorErroring above).
|
||||
#
|
||||
# THE Ready-NODE GATE is what keeps this from becoming another permanently-firing alert. With
|
||||
# three clusters on three nodes, losing a node NECESSARILY co-locates two primaries, and that
|
||||
# is correct behaviour, not a defect — nagging about it during an outage is pure noise. So the
|
||||
# rule only speaks when every node is Ready, i.e. when the spread is actually achievable and
|
||||
# the co-location is therefore avoidable drift. It re-arms by itself once the node returns.
|
||||
#
|
||||
# for: 30m — a switchover is seconds and a full rolling update of one Cluster is minutes, so
|
||||
# 30m rides out every legitimate transient (the condition must hold at EVERY evaluation) while
|
||||
# still reporting real drift the same hour. severity: warning, like every rule in this file:
|
||||
# this is a resilience posture to correct at leisure, never a page.
|
||||
#
|
||||
# PROVEN IN BOTH DIRECTIONS (2026-09-13 UTC), which is the whole point — an alert that cannot
|
||||
# fire is decoration and one that cannot stop is fatigue:
|
||||
# * FIRES, END TO END, ON LIVE PROD DATA. Before the rebalance the expression returned
|
||||
# {node="k3s-prod-1"} => 3. The rule was then loaded into prod's Prometheus as a
|
||||
# throwaway twin (same expr, `for: 1m` so the proof did not need a 30-minute wait) and
|
||||
# observed going `pending` at +70s and `firing` at +130s with node="k3s-prod-1". The twin
|
||||
# was deleted afterwards; it was never committed.
|
||||
# * GOES QUIET once rebalanced. After the two switchovers the same expression returned an
|
||||
# empty vector, while the inner join still resolved all three primaries — semaphore-db-2
|
||||
# on k3s-prod-1, keycloak-db-1 on k3s-prod-2, nextcloud-db-1 on k3s-prod-3. Quiet because
|
||||
# the condition is false, NOT because the query went blind.
|
||||
# * UNIT-TESTED, AND THE TESTS CAN FAIL. alerts.promtool-test.yaml (run it with
|
||||
# ./run-alert-tests.sh) covers four cases under promtool 3.14.0, the exact Prometheus
|
||||
# version running here: co-located fires after 30m, one-per-node is silent, a NotReady
|
||||
# node suppresses it and then re-arms when the node returns, and a non-CNPG `*-rw`
|
||||
# Service is ignored. Mutation-checked — deleting the cnpg_collector_up join, the
|
||||
# readiness gate, or shortening `for:` each makes a test FAIL rather than pass quietly.
|
||||
- alert: CNPGPrimariesCoLocated
|
||||
expr: |
|
||||
count by (node) (
|
||||
max by (namespace, pod, node) (
|
||||
label_replace(
|
||||
label_replace(
|
||||
kube_endpointslice_endpoints{endpointslice=~".+-rw-.+", targetref_kind="Pod", ready="true"},
|
||||
"pod", "$1", "targetref_name", "(.+)"
|
||||
),
|
||||
"node", "$1", "endpoint_nodename", "(.+)"
|
||||
)
|
||||
)
|
||||
* on (namespace, pod) group_left ()
|
||||
max by (namespace, pod) (cnpg_collector_up)
|
||||
) > 1
|
||||
and on ()
|
||||
count(kube_node_status_condition{condition="Ready", status="true"} == 1) == count(kube_node_info)
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $value }} CNPG primaries share node {{ $labels.node }}"
|
||||
description: "{{ $value }} CloudNativePG PRIMARY instances have been running on {{ $labels.node }} for 30m while every node is Ready, so losing that single node means {{ $value }} simultaneous PostgreSQL failovers at once. Each Cluster survives it (instances: 3, one pod per node, one synchronous standby), so this is BLAST RADIUS, not an outage — but it is avoidable drift: CNPG has no declarative primary-placement field (see the comment above this rule), so the primary lands wherever the last promotion put it. List the primaries with `kubectl get pods -A -l cnpg.io/instanceRole=primary -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName`, then move one onto an idle node with `kubectl cnpg promote -n <ns> <cluster> <instance-on-idle-node>` (the cnpg plugin is NOT in the devcontainer image — fetch the kubectl-cnpg release matching the operator version). Do ONE CLUSTER AT A TIME and wait for readyInstances: 3 before the next: a switchover interrupts writes to that database for about 9 seconds, measured on both clusters moved on 2026-09-13. This alert re-arms itself, so the drift will be reported again rather than silently persisting."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-primary-spread"
|
||||
Executable
+60
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# platform/cnpg/run-alert-tests.sh — run the promtool unit tests for alerts.yaml.
|
||||
#
|
||||
# WHY A SCRIPT AND NOT JUST `promtool test rules`. Two impedance mismatches:
|
||||
# 1. alerts.yaml is a Kubernetes PrometheusRule CR; promtool wants a bare Prometheus
|
||||
# rules file (`groups:` at the top level). So `.spec` is extracted first.
|
||||
# 2. runbook_url carries `${BASE_DOMAIN}`, a Flux postBuild substitution that is not
|
||||
# resolved in git. promtool does not care what the string says, but leaving a raw
|
||||
# `${...}` in a rendered rules file is exactly the kind of thing that gets copied
|
||||
# somewhere it DOES matter, so it is substituted with a placeholder here.
|
||||
# Both derived files go to a temp dir — nothing generated is left in the worktree, because
|
||||
# a generated file sitting next to its source is how a stale copy becomes the thing under
|
||||
# test.
|
||||
#
|
||||
# promtool is NOT in the baked devcontainer toolchain (infra/devcontainer/versions.env).
|
||||
# If it is not on PATH this fetches the version that matches the Prometheus actually
|
||||
# running on prod, so the unit tests are evaluated by the same engine that will evaluate
|
||||
# the rules for real. Override with PROMTOOL=/path/to/promtool.
|
||||
#
|
||||
# ./run-alert-tests.sh
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
# Match the Prometheus running on prod: quay.io/prometheus/prometheus:v3.14.0-distroless
|
||||
# (`kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`).
|
||||
PROMETHEUS_VERSION="${PROMETHEUS_VERSION:-3.14.0}"
|
||||
|
||||
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
tmp="$(mktemp -d)"
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
|
||||
promtool="${PROMTOOL:-$(command -v promtool || true)}"
|
||||
if [[ -z "$promtool" ]]; then
|
||||
echo "==> promtool not on PATH; fetching v${PROMETHEUS_VERSION} (matches prod Prometheus)"
|
||||
tarball="prometheus-${PROMETHEUS_VERSION}.linux-amd64"
|
||||
curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/${tarball}.tar.gz" \
|
||||
| tar -xz -C "$tmp" --strip-components=1 "${tarball}/promtool"
|
||||
promtool="$tmp/promtool"
|
||||
fi
|
||||
# Absolutised because step 3 runs promtool from inside $tmp — a relative PROMTOOL= would
|
||||
# break there, and it did during development.
|
||||
promtool="$(cd "$(dirname "$promtool")" && pwd)/$(basename "$promtool")"
|
||||
echo "==> $("$promtool" --version 2>&1 | head -1)"
|
||||
|
||||
# 1. PrometheusRule CR -> bare Prometheus rules file, with ${BASE_DOMAIN} resolved.
|
||||
sed 's/\${BASE_DOMAIN}/base-domain.invalid/g' "$here/alerts.yaml" \
|
||||
| python3 -c 'import sys,yaml; yaml.safe_dump(yaml.safe_load(sys.stdin)["spec"], sys.stdout, sort_keys=False, width=10**6)' \
|
||||
> "$tmp/alerts.rules.yaml"
|
||||
cp "$here/alerts.promtool-test.yaml" "$tmp/"
|
||||
|
||||
# 2. Syntax gate first — a rules file that does not parse would make every unit test
|
||||
# "pass" vacuously by finding no rules at all.
|
||||
echo "==> promtool check rules"
|
||||
"$promtool" check rules "$tmp/alerts.rules.yaml"
|
||||
|
||||
# 3. The unit tests.
|
||||
echo "==> promtool test rules"
|
||||
cd "$tmp"
|
||||
"$promtool" test rules alerts.promtool-test.yaml
|
||||
Reference in new issue
Block a user