--- # Alert rules for the kured reboot automation. # # WHY THESE THREE, AND WHY THEY ARE NOT OPTIONAL. Reboot automation's characteristic failure is # SILENCE, not noise. Every way this component can break leaves a cluster that looks perfectly # healthy: a sentinel nothing writes, a Service the chart did not create, a NetworkPolicy that drops # the scrape, a taint that excludes a node, a drain wedged on a PodDisruptionBudget, a cluster lock # held forever by a node that hung on warm reboot. In none of those cases does anything turn red on # its own. kured itself stays 1/1 Running throughout. # # kured EXPOSES EXACTLY ONE METRIC. Read from cmd/kured/main.go:92-96 and :113 on 2026-10-17 — the # only `prometheus.MustRegister` in the tree is a GaugeVec `Subsystem: "kured"`, `Name: # "reboot_required"`, labelled `{node}`. So the full vocabulary available is # `kured_reboot_required{node="..."}`, 1 when the sentinel exists and 0 when it does not. Each rule # below is built from that one series plus `kube_node_info` (kube-state-metrics), and nothing else. # There is deliberately no rule over an invented metric name: this repo has shipped a # structurally-inert alert before (VeleroNoRecentBackup over an absent series, which is why # platform/monitoring/tests/run.sh exists), and an alert that cannot fire is worse than no alert # because it also removes the suspicion that would have caught it. # # THESE RULES LIVE HERE, NOT IN platform/monitoring/alerts.yaml, for two reasons. (1) The unit-test # harness treats a `.promtool-test.yaml` COMPANION as covering just its sibling, whereas a # `tests/*.test.yaml` SUITE covers every PrometheusRule in its directory — so shipping them beside # the component keeps the test obligation local and legible (the platform/cnpg precedent). (2) # platform/monitoring/alerts.yaml is concurrently edited by other open branches; adding 3 rules to # its tail would create a merge conflict for no benefit. apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: kured namespace: kured labels: app.kubernetes.io/name: kured spec: groups: - name: platform-health.node-reboots rules: # ── 1. IS THE MECHANISM ALIVE AT ALL? ──────────────────────────────────────────────── - alert: KuredNotReporting # `absent()` yields a series only when the ENTIRE vector is empty, so this fires when no # node at all is publishing — kured gone, or its Service/ServiceMonitor/NetworkPolicy # regressed so nothing is scraped. It stays quiet while a single node is mid-reboot # (the other two still report), which is what makes it specific to a wiring failure # rather than to a reboot in progress. # # THIS IS THE RULE THAT GUARDS THE OTHER TWO. Both of them read `kured_reboot_required`, # so if that series stops existing they go quiet and look like good news. This one turns # the absence itself into the signal. The specific regression it is aimed at: the kured # chart defaults `service.create` and `metrics.create` to FALSE, and Longhorn has already # demonstrated on this cluster (see the companion note in # conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh) that a # NetworkPolicy can let targets be DISCOVERED while every scrape times out and the TSDB # stays empty. expr: absent(kured_reboot_required) # 30m absorbs a HelmRelease upgrade rolling the DaemonSet and a Prometheus restart, # without tolerating a real wiring break for more than one quiet night's worth of time. for: 30m labels: severity: warning annotations: summary: kured is publishing no reboot_required series on any node description: >- No node is reporting kured_reboot_required, so the reboot automation is either not running or not being scraped. While this is true, OS reboots are not happening and the two alerts that would say so (KuredRebootPending, KuredNodeCoverageGap) are themselves inert. Nodes will silently accumulate pending kernel upgrades. runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation" # ── 2. IS IT ALIVE ON *EVERY* NODE? ────────────────────────────────────────────────── - alert: KuredNodeCoverageGap # A per-node agent that silently covers only some nodes is the subtler half of the # liveness question, and `absent()` cannot see it. kured publishes one series per node it # runs on; kube-state-metrics publishes one `kube_node_info` per node that exists # (measured 2026-10-17: 3, matching the 3 nodes). A shortfall means at least one node has # no reboot automation — most likely a new or re-provisioned node carrying a taint this # DaemonSet does not tolerate (it declares none, because no prod node is tainted today). # That node would then never reboot, with nothing else to indicate it. expr: count(kured_reboot_required) < count(kube_node_info) # 2h, chosen to sit comfortably ABOVE the duration of a healthy reboot. A node being # rebooted stops publishing for a few minutes and legitimately creates a shortfall; a # node that is still missing after two hours is either genuinely uncovered or has hung on # reboot, and both of those want a human. for: 2h labels: severity: warning annotations: summary: kured is not running on every node description: >- Fewer nodes are reporting kured_reboot_required than exist in the cluster, so at least one node has no reboot automation and will never be rebooted by it. Most likely a node carries a taint this DaemonSet does not tolerate, or has hung during a reboot. runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation" # ── 3. IS A REBOOT STUCK? ──────────────────────────────────────────────────────────── - alert: KuredRebootPending # A node has wanted a reboot for longer than a full maintenance window plus margin, so # the reboot was OFFERED and did not happen. The causes are all real and all invisible # otherwise: the Prometheus alert gate is blocking on a degraded Longhorn volume that is # not recovering; the drain hit `drainTimeout` against a 0-disruption PDB; another node # holds the cluster lock and never released it (`lockTtl: 0` means a lock held by a hung # node is held FOREVER — that is deliberate, see helmrelease.yaml, and this is the alert # that makes the resulting stall visible). expr: max by (node) (kured_reboot_required) == 1 # 36h, and the window being DAILY is what makes this number meaningful — see the window # block in helmrelease.yaml. With a 03:00-06:00 window every day, a pending reboot waits # at most ~24h legitimately, so 36h means at least one full window elapsed without the # reboot completing. On a weekly window this threshold would have to exceed 7 days, which # is longer than this cluster's 7d Prometheus retention and is reset by any Prometheus # restart past the 1h for-outage-tolerance — i.e. unreliable in the one direction that # matters. Note this alert can still be LATE for that reason, but it cannot be spurious. for: 36h labels: severity: warning annotations: summary: Node {{ $labels.node }} has needed a reboot for over 36h description: >- kured reports a pending reboot on {{ $labels.node }} that has survived at least one full 03:00-06:00 maintenance window. The reboot is being blocked, not merely waiting: likely a degraded Longhorn volume holding the Prometheus alert gate, a drain timing out against a PodDisruptionBudget, or a stale cluster lock held by a node that did not come back. The kernel or security update that needed the reboot is installed but NOT running. runbook_url: "https://${BASE_DOMAIN}/runbooks/node-reboot-automation"