jakman85andClaude Opus 5.5
09f219fe7f
servicemonitor-ports-resolve could not audit a PodMonitor with no in-repo Service, and no-plaintext-secrets requires a template's secret value to be a bare placeholder. Co-Authored-By: Claude Opus 5.5 <[email protected]>
65 lines
3.5 KiB
YAML
65 lines
3.5 KiB
YAML
---
|
|
# Alert rules for the Cloudflare Tunnel connector.
|
|
#
|
|
# THE DESIGN CONSTRAINT: NEITHER RULE MAY GO QUIET WHEN ITS OWN DATA DISAPPEARS. The repo has
|
|
# shipped self-silencing rules before (a bare `up{}` that clears exactly when scraping breaks), so:
|
|
#
|
|
# * CloudflaredTunnelDown reads `cloudflared_tunnel_ha_connections` — the per-replica gauge of
|
|
# live edge connections — and falls back to `vector(0)` when NO such series exists. A dead
|
|
# tunnel, zero pods, a broken ServiceMonitor, or a NetworkPolicy that drops the scrape all make the
|
|
# series absent, and all of them FIRE rather than read as healthy.
|
|
# * CloudflaredReplicasDegraded reads kube-state-metrics, which does not depend on cloudflared
|
|
# being scrapable at all, and has the same `vector(0)` fallback.
|
|
#
|
|
# Both directions are pinned by the companion alerts.promtool-test.yaml (healthy is quiet; absent
|
|
# series fires; working-then-broken re-fires).
|
|
#
|
|
# NOTE FOR PHASE 1: until the operator completes runbooks/cloudflare-tunnel.md steps 1-5 the
|
|
# Deployment cannot connect and CloudflaredTunnelDown fires. That is the honest state — the tunnel
|
|
# is not serving — and nothing routes through it yet, which is why both rules are `warning`.
|
|
# Raise TunnelDown to `critical` in the change that points public DNS at the tunnel.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: cloudflared
|
|
namespace: cloudflared
|
|
labels:
|
|
app.kubernetes.io/name: cloudflared
|
|
spec:
|
|
groups:
|
|
- name: platform-health.cloudflare-tunnel
|
|
rules:
|
|
- alert: CloudflaredTunnelDown
|
|
# Sum of live edge connections across every replica. `or on() vector(0)` turns an EMPTY
|
|
# result (nothing scraped) into 0 so it fires instead of evaluating to no series.
|
|
expr: (sum(cloudflared_tunnel_ha_connections{namespace="cloudflared"}) or on() vector(0)) < 1
|
|
# 10m rides out a rolling update and a Cloudflare edge reconnect, not a real outage.
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: Cloudflare Tunnel has no healthy edge connections
|
|
description: >-
|
|
No cloudflared replica reports a live connection to the Cloudflare edge (or none is
|
|
being scraped at all). Every hostname routed through the tunnel is unreachable from
|
|
the internet. Before the operator has enrolled the tunnel this is expected.
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cloudflare-tunnel"
|
|
|
|
- alert: CloudflaredReplicasDegraded
|
|
# Fewer than the 2 replicas deployment.yaml asks for are Available — the tunnel has lost
|
|
# its HA, and the next node drain or reboot takes it down. Keep `2` in step with
|
|
# spec.replicas. Absent series (kube-state-metrics gone, Deployment deleted) -> 0 -> fires.
|
|
expr: >-
|
|
(max(kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"})
|
|
or on() vector(0)) < 2
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: cloudflared has fewer than 2 available replicas
|
|
description: >-
|
|
The cloudflared Deployment has fewer than two Available replicas, so the tunnel has
|
|
no redundancy (or is down entirely). Check for unenrolled credentials, a node that
|
|
cannot schedule the hard-spread pod, or an image pull failure.
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cloudflare-tunnel"
|