feat(updates): alert on failed / rolled-back updates (R-UPD-02 + R-OBS)

Add platform-health.automated-updates PrometheusRule group carried by platform-config: ImageUpdateAutomationFailing (write-back broken -> advances silently stop), ImagePolicyStalled (registry/range can't resolve a stable tag), and UpdateHealthGateFailed (a bad update health-gated and auto-rolling-back via HelmRelease remediation). Routes to Alertmanager email so a bad update is visible. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:14:54 -05:00
1 parent f538f8ff38
commit bfec9d869e
@@ -369,3 +369,68 @@ spec:
            summary: "More than 10% of Prometheus targets are down"
            description: "Over 10% of scrape targets have been down for 15m — a broad monitoring blind spot."
            runbook_url: "https://home.keithjackson.dev/runbooks/targets-down"
+
+# =====================================================================================
+# Automated updates — Flux image-automation + hands-off update health (docs/11, ADR-0009)
+#   R-UPD-02 (a failed/rolled-back update alerts) + R-OBS. All Flux controllers (incl. the image
+#   reflector/automation + helm controllers) export the shared `gotk_reconcile_condition` /
+#   `gotk_suspend_status` series; these rules add UPDATE-SPECIFIC visibility on top of the generic
+#   platform-health.flux group so a silently-stalled or a rolled-back update is not missed.
+# =====================================================================================
+---
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: platform-health-automated-updates
+  namespace: monitoring
+  labels:
+    release: kube-prometheus-stack
+    platform-health: "true"
+spec:
+  groups:
+    - name: platform-health.automated-updates
+      rules:
+        # The write-back engine (ADR-0009) is broken → hands-off image updates SILENTLY stop advancing.
+        - alert: ImageUpdateAutomationFailing
+          expr: |
+            max by (namespace, name) (
+              gotk_reconcile_condition{kind="ImageUpdateAutomation", type="Ready", status="False"}
+            ) == 1
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Flux image-update-automation failing: {{ $labels.name }}"
+            description: "ImageUpdateAutomation {{ $labels.namespace }}/{{ $labels.name }} cannot reconcile (git write-back) for 30m — stable image advances have stopped (R-UPD-02). Check the git write credential."
+            runbook_url: "https://home.keithjackson.dev/runbooks/image-automation"
+        # An ImagePolicy/ImageRepository can't resolve the latest stable tag (registry rate-limit /
+        # auth / bad range) → that service stops receiving updates even though automation is healthy.
+        - alert: ImagePolicyStalled
+          expr: |
+            max by (namespace, name, kind) (
+              gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}
+            ) == 1
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "Image scan/policy stalled: {{ $labels.kind }}/{{ $labels.name }}"
+            description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} cannot resolve its stable tag for 1h (registry rate-limit / creds / range) — no longer receiving updates (R-UPD-01)."
+            runbook_url: "https://home.keithjackson.dev/runbooks/image-policy-stalled"
+        # A bad update was health-gated and AUTO-ROLLED-BACK (or is stuck mid-remediation). helm-controller
+        # holds Ready=False while a failed upgrade remediates/rolls back to the last-good revision
+        # (HelmRelease.spec.upgrade.remediation + .spec.rollback). This is the R-UPD-03 safety rail firing.
+        # TODO(bring-up): tighten with the `reason` label (e.g. UpgradeFailed / RolledBack) once confirmed
+        # against the deployed helm-controller metrics.
+        - alert: UpdateHealthGateFailed
+          expr: |
+            max by (namespace, name) (
+              gotk_reconcile_condition{kind="HelmRelease", type="Ready", status="False"}
+            ) == 1
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Update health-gate failed / rolled back: {{ $labels.name }}"
+            description: "HelmRelease {{ $labels.namespace }}/{{ $labels.name }} failed its health-gated upgrade and is rolling back to the last-good revision (bad-update auto-rollback — R-UPD-03). Consider an `update.pin` escape hatch (docs/11)."
+            runbook_url: "https://home.keithjackson.dev/runbooks/update-rollback"