home-network-k3s-impl/platform/kured/alerts.promtool-test.yaml

---
# promtool unit tests for the three rules in platform/kured/alerts.yaml.
#
# WHY THIS FILE EXISTS. The rules it covers are the ONLY thing that can tell an operator that the
# reboot automation has stopped working, and every failure mode they describe is silent — kured stays
# 1/1 Running while rebooting nothing. An alert that cannot fire is therefore not a cosmetic defect
# here, it is the removal of the only signal. This repo has shipped that bug in both directions
# already (a permanently-firing VPNTunnelDown; a structurally inert VeleroNoRecentBackup over an
# absent series), which is why the harness exists at all. Each rule below is pinned BOTH ways: quiet
# on the healthy state, firing on the constructed bad one, with the `for:` boundary asserted rather
# than assumed.
#
# HOW TO RUN. `platform/monitoring/tests/run.sh` — the one repo-wide rule-unit harness. It finds this
# file by the COMPANION convention (`<manifest>.promtool-test.yaml` beside `<manifest>.yaml`), so the
# manifest under test is exactly platform/kured/alerts.yaml and nothing else. Run just this file with
#   platform/monitoring/tests/run.sh platform/kured/alerts.promtool-test.yaml
# The harness flattens the PrometheusRule CR's `.spec.groups` into the bare rules file promtool wants,
# `promtool check rules` it first so a non-parsing file cannot pass vacuously, and runs it all in a
# temp dir. It leaves Flux's `${BASE_DOMAIN}` in the runbook_url annotations verbatim, which is safe
# because the assertions below query the `ALERTS` series and never compare annotations.
#
# WHY THE ASSERTIONS QUERY `ALERTS` RATHER THAN USING `exp_alerts`. promtool's alert_rule_test
# compares annotations VERBATIM, which would mean pasting each rule's description prose in here and
# re-pasting it on every wording change. Querying the synthetic `ALERTS` series proves what actually
# matters — the alert reaches `pending` and then `firing` across its `for:` boundary, with the right
# labels — and stays decoupled from annotation text. Same choice as
# platform/cnpg/alerts.promtool-test.yaml.
#
# THE SERIES SHAPES ARE REAL. `kured_reboot_required{node=...}` is a GaugeVec with exactly one label,
# `node` — read from kured's cmd/kured/main.go:92-96 (the only registered metric in the project).
# `kube_node_info{node=...}` was measured on prod Prometheus on 2026-10-17 returning
# `count(kube_node_info) == 3`, matching the three nodes. The extra kube_node_info labels prod
# carries are omitted deliberately: every rule aggregates with `count()`, so they cannot affect the
# result, and inventing plausible-but-wrong values would be worse than leaving them out.
#
# evaluation_interval is 5m, not the usual 1m: KuredRebootPending has `for: 36h`, and at a 1m
# interval promtool would evaluate that test over 2000+ steps for no added coverage.
rule_files:
  - alerts.rules.yaml
evaluation_interval: 5m

tests:
  # ─────────────────────────────────────────────────────────────────────────────────────────────
  # 1. THE HEALTHY STEADY STATE. Three nodes, kured running on all three, nothing pending.
  #    ALL THREE RULES MUST BE SILENT. This is the case that a badly-written `absent()` or a
  #    `count() <` comparison with an off-by-one gets wrong, and it is also the state the cluster
  #    is in ~100% of the time, so a false positive here would train the operator to ignore the
  #    whole group.
  # ─────────────────────────────────────────────────────────────────────────────────────────────
  - interval: 5m
    name: healthy cluster - no reboots pending, kured on every node, all three rules quiet
    input_series:
      - series: 'kured_reboot_required{node="k3s-prod-1"}'
        values: '0x60'
      - series: 'kured_reboot_required{node="k3s-prod-2"}'
        values: '0x60'
      - series: 'kured_reboot_required{node="k3s-prod-3"}'
        values: '0x60'
      - series: 'kube_node_info{node="k3s-prod-1"}'
        values: '1x60'
      - series: 'kube_node_info{node="k3s-prod-2"}'
        values: '1x60'
      - series: 'kube_node_info{node="k3s-prod-3"}'
        values: '1x60'
    promql_expr_test:
      # Nothing from this group may be pending or firing, at any point in the window.
      - expr: 'count(ALERTS{alertname=~"Kured.*"})'
        eval_time: 4h
        exp_samples: []
      # And the underlying expressions must each be genuinely false, not merely unevaluated:
      # absent() yields no series because the vector is populated...
      - expr: 'absent(kured_reboot_required)'
        eval_time: 4h
        exp_samples: []
      # ...coverage is exact (3 == 3), so the `<` comparison drops the sample...
      - expr: 'count(kured_reboot_required) < count(kube_node_info)'
        eval_time: 4h
        exp_samples: []
      # ...and no node reports a pending reboot.
      - expr: 'max by (node) (kured_reboot_required) == 1'
        eval_time: 4h
        exp_samples: []

  # ─────────────────────────────────────────────────────────────────────────────────────────────
  # 2. KuredNotReporting — THE WHOLE PIPELINE IS DEAD. No kured series at all, which is what the
  #    cluster looks like if the chart's `service.create`/`metrics.create` regress to their FALSE
  #    defaults, if the ServiceMonitor is dropped, or if a NetworkPolicy silently times out the
  #    scrape (the Longhorn precedent). The nodes still exist, so kube_node_info is present —
  #    proving the rule keys on kured's absence and not on a dead cluster.
  # ─────────────────────────────────────────────────────────────────────────────────────────────
  - interval: 5m
    name: no kured series anywhere - KuredNotReporting fires after 30m
    input_series:
      - series: 'kube_node_info{node="k3s-prod-1"}'
        values: '1x60'
      - series: 'kube_node_info{node="k3s-prod-2"}'
        values: '1x60'
      - series: 'kube_node_info{node="k3s-prod-3"}'
        values: '1x60'
    promql_expr_test:
      # The expression itself yields 1 immediately.
      - expr: 'absent(kured_reboot_required)'
        eval_time: 10m
        exp_samples:
          - labels: '{}'
            value: 1
      # `for: 30m` is honoured — still only pending at 25m.
      - expr: 'ALERTS{alertname="KuredNotReporting"}'
        eval_time: 25m
        exp_samples:
          - labels: 'ALERTS{alertname="KuredNotReporting", alertstate="pending", severity="warning"}'
            value: 1
      # ...and firing once past it.
      - expr: 'ALERTS{alertname="KuredNotReporting", alertstate="firing"}'
        eval_time: 35m
        exp_samples:
          - labels: 'ALERTS{alertname="KuredNotReporting", alertstate="firing", severity="warning"}'
            value: 1
      # THE GUARD CLAUSE: with kured absent, the other two rules are inert — exactly the reason
      # this rule has to exist. KuredRebootPending cannot fire (no series to be 1) and
      # KuredNodeCoverageGap cannot either, because `count()` over an empty vector returns NO
      # SAMPLE rather than 0, so the `<` comparison has nothing to compare. Asserting that here
      # pins the dependency: if a future edit made coverage-gap fire in this state, the overlap
      # would be deliberate rather than accidental.
      - expr: 'ALERTS{alertname="KuredRebootPending"}'
        eval_time: 35m
        exp_samples: []
      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
        eval_time: 35m
        exp_samples: []
      - expr: 'count(kured_reboot_required) < count(kube_node_info)'
        eval_time: 35m
        exp_samples: []

  # ─────────────────────────────────────────────────────────────────────────────────────────────
  # 3. KuredNodeCoverageGap — kured COVERS ONLY TWO OF THREE NODES. k3s-prod-3's series stops at
  #    1h and never returns (a node taint this DaemonSet does not tolerate, a node that never came
  #    back from a reboot, a DaemonSet nodeSelector edit). This also pins the `for: 2h` choice,
  #    which exists SO THAT AN ORDINARY REBOOT DOES NOT TRIP IT: the gap is real from ~1h05 but
  #    the alert is only pending an hour later, and a healthy reboot is minutes.
  # ─────────────────────────────────────────────────────────────────────────────────────────────
  - interval: 5m
    name: kured missing on one node - KuredNodeCoverageGap fires after 2h, not before
    input_series:
      - series: 'kured_reboot_required{node="k3s-prod-1"}'
        values: '0x60'
      - series: 'kured_reboot_required{node="k3s-prod-2"}'
        values: '0x60'
      # Ends after 1h (13 samples at 5m) and is stale thereafter.
      - series: 'kured_reboot_required{node="k3s-prod-3"}'
        values: '0x12'
      - series: 'kube_node_info{node="k3s-prod-1"}'
        values: '1x60'
      - series: 'kube_node_info{node="k3s-prod-2"}'
        values: '1x60'
      - series: 'kube_node_info{node="k3s-prod-3"}'
        values: '1x60'
    promql_expr_test:
      # Before the gap opens, coverage is exact and nothing is pending.
      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
        eval_time: 30m
        exp_samples: []
      # The shortfall is real well before the alert fires — 2 < 3.
      - expr: 'count(kured_reboot_required)'
        eval_time: 2h
        exp_samples:
          - labels: '{}'
            value: 2
      # A SHORT OUTAGE DOES NOT ALERT: still only pending at 2h, an hour after the gap opened.
      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
        eval_time: 2h
        exp_samples:
          - labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="pending", severity="warning"}'
            value: 1
      # Firing once the gap has persisted past `for: 2h`.
      - expr: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing"}'
        eval_time: 3h15m
        exp_samples:
          - labels: 'ALERTS{alertname="KuredNodeCoverageGap", alertstate="firing", severity="warning"}'
            value: 1
      # KuredNotReporting must stay QUIET: two nodes are still publishing, so the vector is not
      # absent. This is the distinction between "kured is gone" and "kured is incomplete", and
      # getting it wrong in either direction would collapse two different diagnoses into one.
      - expr: 'ALERTS{alertname="KuredNotReporting"}'
        eval_time: 3h15m
        exp_samples: []

  # ─────────────────────────────────────────────────────────────────────────────────────────────
  # 4. KuredRebootPending — A REBOOT IS BLOCKED. k3s-prod-1 reports a pending reboot continuously:
  #    the sentinel was written (an operator kernel upgrade) and the reboot has not happened across
  #    more than one 03:00-06:00 window. Asserts the 36h boundary in both directions, which is the
  #    assertion that makes the DAILY-window decision load-bearing rather than cosmetic: on a
  #    weekly window this threshold could not distinguish "blocked" from "waiting for Sunday".
  #    Series run to 38h20m (456 samples at 5m) so the post-boundary evaluation has real data.
  # ─────────────────────────────────────────────────────────────────────────────────────────────
  - interval: 5m
    name: reboot pending on one node for over 36h - KuredRebootPending fires, per-node label kept
    input_series:
      - series: 'kured_reboot_required{node="k3s-prod-1"}'
        values: '1x456'
      - series: 'kured_reboot_required{node="k3s-prod-2"}'
        values: '0x456'
      - series: 'kured_reboot_required{node="k3s-prod-3"}'
        values: '0x456'
      - series: 'kube_node_info{node="k3s-prod-1"}'
        values: '1x456'
      - series: 'kube_node_info{node="k3s-prod-2"}'
        values: '1x456'
      - series: 'kube_node_info{node="k3s-prod-3"}'
        values: '1x456'
    promql_expr_test:
      # The expression selects ONLY the pending node — the operator needs to know which box.
      - expr: 'max by (node) (kured_reboot_required) == 1'
        eval_time: 1h
        exp_samples:
          - labels: '{node="k3s-prod-1"}'
            value: 1
      # A NORMAL WAIT IS NOT AN ALERT. At 24h the node is still inside its legitimate
      # once-per-day window wait, so the alert may be pending but must NOT be firing.
      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
        eval_time: 24h
        exp_samples: []
      # Still not firing just short of the boundary.
      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
        eval_time: 35h30m
        exp_samples: []
      # Firing past it, carrying the node label through to the notification.
      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
        eval_time: 36h30m
        exp_samples:
          - labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
            value: 1
      # The two healthy nodes must NOT be alerted on. A rule written with `kured_reboot_required`
      # unaggregated, or compared with `>= 0`, would raise all three and bury the real one.
      - expr: 'count(ALERTS{alertname="KuredRebootPending", alertstate="firing"})'
        eval_time: 36h30m
        exp_samples:
          - labels: '{}'
            value: 1
      # Liveness rules stay quiet: kured is present and covering all three nodes. A pending reboot
      # is NOT a wiring fault and must not be reported as one.
      - expr: 'ALERTS{alertname="KuredNotReporting"}'
        eval_time: 36h30m
        exp_samples: []
      - expr: 'ALERTS{alertname="KuredNodeCoverageGap"}'
        eval_time: 36h30m
        exp_samples: []

  # ─────────────────────────────────────────────────────────────────────────────────────────────
  # 5. THE RULE MUST STOP. A reboot that SUCCEEDS clears the sentinel, and the alert must clear
  #    with it — the other half of the failure space this harness exists for (a permanently-firing
  #    rule is how VPNTunnelDown taught this repo the lesson). k3s-prod-1 is pending for ~40h,
  #    long enough to fire, then the reboot happens and the gauge returns to 0.
  # ─────────────────────────────────────────────────────────────────────────────────────────────
  - interval: 5m
    name: reboot completes after firing - KuredRebootPending resolves
    input_series:
      # 1 for 480 samples (40h), then 0 for the remainder.
      - series: 'kured_reboot_required{node="k3s-prod-1"}'
        values: '1x480 0x120'
      - series: 'kured_reboot_required{node="k3s-prod-2"}'
        values: '0x600'
      - series: 'kured_reboot_required{node="k3s-prod-3"}'
        values: '0x600'
      - series: 'kube_node_info{node="k3s-prod-1"}'
        values: '1x600'
      - series: 'kube_node_info{node="k3s-prod-2"}'
        values: '1x600'
      - series: 'kube_node_info{node="k3s-prod-3"}'
        values: '1x600'
    promql_expr_test:
      # Firing while the reboot is still outstanding.
      - expr: 'ALERTS{alertname="KuredRebootPending", alertstate="firing"}'
        eval_time: 39h
        exp_samples:
          - labels: 'ALERTS{alertname="KuredRebootPending", alertstate="firing", node="k3s-prod-1", severity="warning"}'
            value: 1
      # Cleared once the sentinel is gone. If this assertion ever fails, the rule has become
      # self-sustaining and the operator will start ignoring it.
      - expr: 'ALERTS{alertname=~"Kured.*"}'
        eval_time: 45h
        exp_samples: []