Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
09265e03819c088fcc58642bcd20763ff7a9be03
home-network-k3s-impl/platform/monitoring/alerts.yaml
T
jakman85andClaude Opus 5 6a8953e288 feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing 
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 00:59:22 -05:00

544 lines
28 KiB
YAML
Raw Blame History

---
# Platform-health alert rules (docs/10, R-OBS-04). These complement the default node/kubernetes
# rules bundled by kube-prometheus-stack and add FIRST-CLASS alerts for the platform's own moving
# parts: nodes, Longhorn, Flux, cert-manager, networking/VPN, backups, identity, and the monitoring
# stack's own capacity ("don't let the monitor blind itself").
#
# Each rule maps to a docs/10 platform-health bullet and carries a severity (critical = page-worthy
# vs warning = degraded/capacity) plus enough annotation context (summary + runbook) to act.
#
# The `release: kube-prometheus-stack` label makes the operator's Prometheus select these rules
# (ruleSelector). Some expressions depend on exporters/ServiceMonitors owned by other workstreams
# (Flux, cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble ship their own metrics); the rules
# are authored to fire as soon as those series arrive. Metric names flagged TODO need confirming at
# bring-up against the deployed exporter.
#
# runbook_url points at a docs base; swap for the real runbook location when it exists.
# =====================================================================================
# Nodes
# =====================================================================================
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-nodes
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.nodes
rules:
- alert: NodeExporterDown
expr: up{job="node-exporter"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Node exporter down on {{ $labels.instance }}"
description: "Prometheus cannot scrape node-exporter on {{ $labels.instance }} for 5m — the node may be down or unreachable."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-down"
- alert: NodeMemoryPressureHigh
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
for: 10m
labels:
severity: warning
annotations:
summary: "Memory pressure high on {{ $labels.instance }}"
description: "Available memory below 10% for 10m on {{ $labels.instance }} — the ~22 GiB box is under pressure."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-memory"
- alert: NodeDiskFillingUp
expr: |
(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
/ node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}) < 0.10
for: 15m
labels:
severity: warning
annotations:
summary: "Disk filling up on {{ $labels.instance }} ({{ $labels.mountpoint }})"
description: "Less than 10% free on {{ $labels.mountpoint }} of {{ $labels.instance }} for 15m."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-disk"
- alert: NodeCPUPressureHigh
# PSI "some" for CPU = fraction of wall-clock time at least one task was stalled waiting to
# run. <5% is healthy; sustained >15% means bursty workloads are queueing against the vCPU
# cap (measured 15-23% at 6 vCPU before the 8-vCPU resize). Averaged CPU% (kubectl top) hides
# this — PSI is the metric that actually surfaces the "spikes cause blips of delay" symptom.
expr: rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15
for: 15m
labels:
severity: warning
annotations:
summary: "CPU pressure high on {{ $labels.instance }}"
description: "CPU PSI (some) above 15% for 15m on {{ $labels.instance }} — tasks are being delayed waiting for CPU (bursts exceeding the node's vCPU headroom). Add vCPUs or trim CPU-heavy workloads."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
- alert: NodeCPULoadHigh
# 5-min load average per core. Sustained >1 means the run queue exceeds the core count (more
# runnable/blocked tasks than CPUs). Core count = number of idle-mode cpu series per node.
# Pair with NodeCPUPressureHigh to confirm the contention is CPU (vs I/O, which inflates load
# but not CPU-PSI).
expr: |
node_load5
/ count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1
for: 15m
labels:
severity: warning
annotations:
summary: "Load per core high on {{ $labels.instance }}"
description: "5-min load average per core above 1.0 for 15m on {{ $labels.instance }} — the node's run queue exceeds its core count."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
# =====================================================================================
# Storage — Longhorn
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-longhorn
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.longhorn
rules:
# longhorn_volume_robustness: 0=unknown 1=healthy 2=degraded 3=faulted
- alert: LonghornVolumeDegraded
expr: longhorn_volume_robustness == 2
for: 10m
labels:
severity: warning
annotations:
summary: "Longhorn volume degraded: {{ $labels.volume }}"
description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
- alert: LonghornVolumeFaulted
expr: longhorn_volume_robustness == 3
for: 1m
labels:
severity: critical
annotations:
summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
- alert: LonghornVolumeCapacityHigh
expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85
for: 30m
labels:
severity: warning
annotations:
summary: "Longhorn volume near capacity: {{ $labels.volume }}"
description: "Volume {{ $labels.volume }} is over 85% used for 30m."
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
- alert: LonghornNodeStorageHigh
expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85
for: 30m
labels:
severity: warning
annotations:
summary: "Longhorn node storage near capacity: {{ $labels.node }}"
description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m."
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
# =====================================================================================
# GitOps — Flux
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-flux
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.flux
rules:
- alert: FluxReconciliationFailure
expr: |
max by (namespace, name, kind) (
gotk_reconcile_condition{type="Ready", status="False"}
) == 1
for: 15m
labels:
severity: warning
annotations:
summary: "Flux resource not ready: {{ $labels.kind }}/{{ $labels.name }}"
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has failed to reconcile (Ready=False) for 15m — GitOps drift/failure (R-GITOPS-03)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-reconcile"
- alert: FluxResourceSuspended
expr: |
max by (namespace, name, kind) (gotk_suspend_status) == 1
for: 1h
labels:
severity: warning
annotations:
summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
# =====================================================================================
# Certificates — cert-manager
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-certificates
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.certificates
rules:
- alert: CertificateExpiringSoon
expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (14 * 24 * 3600)
for: 1h
labels:
severity: warning
annotations:
summary: "Certificate expiring soon: {{ $labels.name }}"
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 14 days."
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
- alert: CertificateExpiryImminent
expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (3 * 24 * 3600)
for: 1h
labels:
severity: critical
annotations:
summary: "Certificate expiry imminent: {{ $labels.name }}"
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 3 days and has not renewed."
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
- alert: CertificateNotReady
expr: certmanager_certificate_ready_status{condition="False"} == 1
for: 1h
labels:
severity: warning
annotations:
summary: "Certificate not ready / issuance failing: {{ $labels.name }}"
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} has been in a not-ready state for 1h (ACME issuance failure?)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-issuance"
# =====================================================================================
# Networking — VPN / egress-gateway / denied flows (Hubble)
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-networking
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.networking
rules:
# VPN tunnel health (ADR-0015: node-level host OpenVPN tunnel on the egress-gateway node, steered
# by Cilium Egress Gateway — there is NO in-cluster gateway pod). OpenVPN creates/destroys the tun0
# device with the connection, so the tun0 netdev series exists iff the tunnel is up. node_network_up
# is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent()
# also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering
# gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24.
#
# FIXED 2026-09-06 — the bare `absent(...)` was evaluated CLUSTER-WIDE, so on a cluster with NO
# egress-gateway node it could never clear. Prod has no gateway node, and this alert fired
# continuously from 2026-09-05T21:23Z into `email-critical` — unbounded noise on the one channel
# that must stay trustworthy. The expression is now GATED on the cluster actually being supposed
# to have a tunnel, via two independent guards OR'd together:
#
# 1. A node carries `homelab.keithjackson.dev/egress-gateway=true` — the same label the
# CiliumEgressGatewayPolicy nodeSelector uses, so the alert is armed exactly when the
# steering datapath has a target. Requires the kube-state-metrics node-label allowlist
# enabled in kube-prometheus-stack.yaml (KSM exposes no custom labels by default). This is
# what makes a gateway-less cluster SELF-ARM the moment a gateway node is labelled — no
# manual re-enable to forget. The Node object survives a NotReady node, so this stays true
# while the gateway node is DOWN: gateway-node-down is still alerted.
# 2. `tun0` was observed at all in the last hour — holds the alert armed through a
# kube-state-metrics outage, and keeps dev's behaviour unchanged even before/independently
# of the KSM rollout.
#
# `and on()` joins on the empty label set deliberately: absent() carries `device="tun0"` while the
# guards carry no labels, so an unqualified `and` would never match. Both clauses verified live
# against dev and prod Prometheus (read-only) on 2026-09-06: empty on prod (cannot fire), empty on
# dev (tunnel up), and clause 2 already true on dev.
- alert: VPNTunnelDown
expr: |
absent(node_network_receive_bytes_total{device="tun0"})
and on()
(
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
or
(count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
)
for: 5m
labels:
severity: critical
annotations:
summary: "Egress VPN tunnel down"
description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
# above (tun0 presence, which also goes absent if the gateway node is down); Cilium datapath
# health is covered by the Cilium platform alerts.
- alert: HubbleDeniedFlowSpike
# Cilium/Hubble denied (policy-drop) flows. Requires hubble metrics enabled (docs/03).
expr: sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5
for: 10m
labels:
severity: warning
annotations:
summary: "Spike in policy-denied network flows"
description: "Policy-denied flow rate is elevated for 10m — a NetworkPolicy regression or an intrusion probe (security audit, docs/07)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/denied-flows"
# =====================================================================================
# Backups — Velero
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-backups
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.backups
rules:
- alert: VeleroBackupFailure
expr: increase(velero_backup_failure_total[1h]) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "Velero backup failed: schedule {{ $labels.schedule }}"
description: "One or more Velero backups failed in the last hour (R-STATE-03 — off-cluster DR at risk)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-failure"
- alert: VeleroBackupPartialFailure
expr: increase(velero_backup_partial_failure_total[1h]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Velero backup partially failed"
description: "A Velero backup completed with partial failures in the last hour — some resources/volumes were skipped."
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-partial"
- alert: VeleroNoRecentBackup
expr: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)
for: 1h
labels:
severity: critical
annotations:
summary: "No successful Velero backup in 24h: {{ $labels.schedule }}"
description: "Schedule {{ $labels.schedule }} has not produced a successful backup in over 24h (missed schedule)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
# =====================================================================================
# Identity — Keycloak (SSO is a critical dependency — docs/15)
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-identity
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.identity
rules:
- alert: KeycloakDown
# Scope to the Keycloak SERVER StatefulSet pods (keycloak-0, keycloak-1). The keycloak
# namespace also holds the operator (keycloak-operator-*) and DB (keycloak-db-*) pods, so an
# un-scoped max() would be masked by those being ready — this must reflect the IdP itself.
expr: |
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"})
for: 5m
labels:
severity: critical
annotations:
summary: "Keycloak IdP down"
description: "No ready Keycloak pod for 5m — SSO is unavailable and dependent services cannot authenticate (docs/15)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-down"
- alert: KeycloakDatabaseDown
# Keycloak's PostgreSQL runs as a CloudNativePG Cluster `keycloak-db` (pods keycloak-db-N).
# DR-critical, Longhorn-backed. (CNPG also exports cnpg_* metrics via its PodMonitor.)
expr: |
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*", condition="true"}) == 0
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*"})
for: 5m
labels:
severity: critical
annotations:
summary: "Keycloak database unavailable"
description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
# --- The client-secret reconciler (platform/keycloak/client-secret-reconciler.yaml) ---
# WHY THESE EXIST: on 2026-09-06 every OIDC login on prod failed because the realm import
# had stored the LITERAL `(env:...)` placeholder as all 12 client secrets — and every
# signal the cluster offered was GREEN (the import Job succeeded, the CR read Done=True,
# KeycloakDown/KeycloakDatabaseDown were quiet, the conformance test that asserts
# Done=True passed). The only observable was a human failing to log in. The CronJob is
# the healer; these two rules make its SILENCE observable, so the class cannot sit
# undetected again. They are the runtime counterpart of the L2 audit test.
#
# UNVERIFIED-ON-CLUSTER at authoring time: no CronJob existed on either cluster yet, so
# these series had no samples to check against. `kube_job_status_failed` /
# `kube_job_status_succeeded` and the `job_name` label WERE confirmed present (the realm
# import Job exports them); the reconciler's own series appear with its first run.
# Confirm with: kube_job_status_succeeded{namespace="keycloak",
# job_name=~"keycloak-client-secret-reconciler-.*"}
- alert: KeycloakClientSecretReconcilerFailing
# A reconciler run failed. Either it found drift it could not repair, or it could not
# reach the admin API. Fires alongside KeycloakDown during an IdP outage — that is
# deliberate (the reconciler genuinely is not running), so treat KeycloakDown as the
# cause when both are lit. Self-clears when the failed Jobs age out (TTL 1h).
expr: |
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "Keycloak client-secret reconciler failing"
description: >-
A keycloak-client-secret-reconciler Job has been failing for 15m — an OIDC client
secret may differ from git (unresolved import placeholder, or an unrepairable
drift) and SSO logins for that client will fail. Read the Job log: it names the
client and never prints a secret.
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
- alert: KeycloakClientSecretReconcilerStale
# No SUCCESSFUL run in scope. Catches the failure modes the rule above cannot see: the
# CronJob suspended, never applied by Flux, or unable to schedule a pod at all — i.e.
# the healer being quietly absent, which is exactly the "looks applied and isn't"
# failure this whole change exists to end.
expr: |
absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)
for: 2h
labels:
severity: warning
annotations:
summary: "Keycloak client-secret reconciler has not succeeded"
description: >-
No successful keycloak-client-secret-reconciler run in 2h (schedule is every 15m).
The CronJob may be suspended, unapplied, or unable to schedule — client-secret
drift would go uncorrected and unnoticed (docs/15).
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
# =====================================================================================
# The monitor's own capacity — don't let the monitor blind itself (docs/10)
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-monitoring-self
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.monitoring-self
rules:
- alert: MonitoringVolumeFillingUp
expr: |
(kubelet_volume_stats_available_bytes{namespace="monitoring"}
/ kubelet_volume_stats_capacity_bytes{namespace="monitoring"}) < 0.15
for: 15m
labels:
severity: warning
annotations:
summary: "Monitoring PVC filling up: {{ $labels.persistentvolumeclaim }}"
description: "PVC {{ $labels.persistentvolumeclaim }} (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
- alert: PrometheusTargetsMissing
expr: |
(count(up == 0) / count(up)) > 0.10
for: 15m
labels:
severity: warning
annotations:
summary: "More than 10% of Prometheus targets are down"
description: "Over 10% of scrape targets have been down for 15m — a broad monitoring blind spot."
runbook_url: "https://${BASE_DOMAIN}/runbooks/targets-down"
# =====================================================================================
# Automated updates — Flux image-automation + hands-off update health (docs/11, ADR-0009)
# R-UPD-02 (a failed/rolled-back update alerts) + R-OBS. All Flux controllers (incl. the image
# reflector/automation + helm controllers) export the shared `gotk_reconcile_condition` /
# `gotk_suspend_status` series; these rules add UPDATE-SPECIFIC visibility on top of the generic
# platform-health.flux group so a silently-stalled or a rolled-back update is not missed.
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-automated-updates
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.automated-updates
rules:
# The write-back engine (ADR-0009) is broken → hands-off image updates SILENTLY stop advancing.
- alert: ImageUpdateAutomationFailing
expr: |
max by (namespace, name) (
gotk_reconcile_condition{kind="ImageUpdateAutomation", type="Ready", status="False"}
) == 1
for: 30m
labels:
severity: warning
annotations:
summary: "Flux image-update-automation failing: {{ $labels.name }}"
description: "ImageUpdateAutomation {{ $labels.namespace }}/{{ $labels.name }} cannot reconcile (git write-back) for 30m — stable image advances have stopped (R-UPD-02). Check the git write credential."
runbook_url: "https://${BASE_DOMAIN}/runbooks/image-automation"
# An ImagePolicy/ImageRepository can't resolve the latest stable tag (registry rate-limit /
# auth / bad range) → that service stops receiving updates even though automation is healthy.
- alert: ImagePolicyStalled
expr: |
max by (namespace, name, kind) (
gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}
) == 1
for: 1h
labels:
severity: warning
annotations:
summary: "Image scan/policy stalled: {{ $labels.kind }}/{{ $labels.name }}"
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} cannot resolve its stable tag for 1h (registry rate-limit / creds / range) — no longer receiving updates (R-UPD-01)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/image-policy-stalled"
# A bad update was health-gated and AUTO-ROLLED-BACK (or is stuck mid-remediation). helm-controller
# holds Ready=False while a failed upgrade remediates/rolls back to the last-good revision
# (HelmRelease.spec.upgrade.remediation + .spec.rollback). This is the R-UPD-03 safety rail firing.
# TODO(bring-up): tighten with the `reason` label (e.g. UpgradeFailed / RolledBack) once confirmed
# against the deployed helm-controller metrics.
- alert: UpdateHealthGateFailed
expr: |
max by (namespace, name) (
gotk_reconcile_condition{kind="HelmRelease", type="Ready", status="False"}
) == 1
for: 10m
labels:
severity: critical
annotations:
summary: "Update health-gate failed / rolled back: {{ $labels.name }}"
description: "HelmRelease {{ $labels.namespace }}/{{ $labels.name }} failed its health-gated upgrade and is rolling back to the last-good revision (bad-update auto-rollback — R-UPD-03). Consider an `update.pin` escape hatch (docs/11)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/update-rollback"
Reference in new issue View Git Blame Copy Permalink