home-network-k3s-impl/runbooks/targets-down.md

targets-down — More than 10% of Prometheus scrape targets are down

Alert: PrometheusTargetsMissing (warning, 15m) File: platform/monitoring/alerts.yaml, group platform-health.monitoring-self Expression: (count(up == 0) / count(up)) > 0.10

What this means here

A tenth of everything Prometheus is supposed to scrape is unreachable. This is a blind-spot alert: its purpose is to tell you that other alerts may be silent because their data is missing, not because the thing they watch is healthy.

Live baseline on prod, measured 2026-09-13: 55 targets across 19 jobs — apiserver, cdi-prometheus-metrics, coredns, keycloak-discovery, keycloak-service, keycloak/keycloak-db, kube-prometheus-stack-alertmanager, kube-prometheus-stack-grafana, kube-prometheus-stack-operator, kube-prometheus-stack-prometheus, kube-state-metrics, kubelet, kubevirt-prometheus-metrics, nextcloud-db/nextcloud-db, node-exporter, oauth2-proxy, semaphore-db/semaphore-db, velero, velero/node-agent. So 10% is roughly 6 targets — one node's worth of exporters, or one CNPG cluster's three pods.

The far more dangerous failure this alert CANNOT see: a target that was never configured. up only counts targets Prometheus knows about. A missing ServiceMonitor produces no up == 0 series and no alert at all. This cluster has been bitten by that repeatedly, and these are the known live gaps as of 2026-09-13 — every one of them means the corresponding alerts are inert:

Missing scrape job	Alerts that cannot fire
Longhorn	LonghornVolumeDegraded, LonghornVolumeFaulted, LonghornDiskAllocationHigh, LonghornDiskUnschedulable
cert-manager	CertificateExpiringSoon, CertificateExpiryImminent, CertificateNotReady
Flux controllers (gotk_*)	FluxReconciliationFailure, FluxResourceSuspended, ImageUpdateAutomationFailing, ImagePolicyStalled, UpdateHealthGateFailed
Hubble / Cilium (hubble_drop_total)	HubbleDeniedFlowSpike
etcd (kubeEtcd disabled for k3s, deliberately)	no etcd fsync alerting exists at all

Precedent for how costly this is: Velero shipped with three backup alerts and metrics.serviceMonitor.enabled: false, so "a nightly backup could fail — or silently stop running — for weeks with nothing to say so." That ServiceMonitor now exists. The rest of the table does not.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git. main is a protected branch — a direct push is rejected. A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which targets are down, grouped by job:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=up == 0" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "\"job\"|\"instance\"|\"namespace\"" '

The scrape-error text, which usually names the cause outright:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s "localhost:19090/api/v1/targets?state=active" | python3 -c "
import json,sys
for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
    if t[\"health\"]!=\"up\": print(t[\"labels\"].get(\"job\"), t[\"scrapeUrl\"], t[\"lastError\"][:120])"'

Which monitors exist at all (the gap this alert cannot see):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get servicemonitors,podmonitors -A'

Likely causes, most likely first

1. A node is down or NotReady

One node takes out its node-exporter and kubelet targets together — roughly the 10% threshold on its own. Go to /runbooks/node-down; this alert is the echo.

2. A workload was rolled, scaled to zero, or crashlooped

Targets follow pods. A CNPG cluster restarting takes three targets with it. Transient; if it persists, the workload is the problem, not the monitoring.

3. A NetworkPolicy blocks the scrape

Default-deny is the baseline everywhere (R-NET-01) and a new component's monitor will be scraped the moment it exists — serviceMonitorSelectorNilUsesHelmValues: false means Prometheus selects every monitor in the cluster — so a newly added workload can produce a permanently-down target purely because nothing allows ingress from monitoring. The lastError will be a timeout. Fix is an explicit allow in the component's CiliumNetworkPolicy, in git.

4. TLS or auth on the scrape endpoint

lastError will say so (certificate or 401/403). Note a related trap from this cluster: after the Cilium upgrade, an in-cluster call to a *.home hostname is re-checked against the destination's real backend identity, so toEntities: world no longer covers it — see /runbooks/denied-flows.

5. The exporter is genuinely broken

CDI once had a broken HTTP ServiceMonitor that caused a false CDIOperatorDown, and KubeVirtControlPlaneDown once used a non-existent metric with the wrong job label. Both were fixed. An exporter that is up but publishing the wrong series is the hardest case — see /runbooks/cnpg-exporter for the canonical example (the CNPG exporter can be up=1 while its whole cnpg_pg_stat_* family is missing).

When to escalate or stop

• Treat this alert as "trust nothing else for now". Before acting on the silence of any other alert, confirm its series exists (count(<metric>)), because the table above shows five whole rule groups are currently evaluating against nothing.
• Escalate if the down targets are apiserver or kubelet — that is a control-plane problem, not a monitoring one.
• Adding a missing ServiceMonitor is a git change (a new ServiceMonitor next to the component, or a chart value). Do not "fix" a blind spot by silencing the alert.
• The Longhorn and cert-manager scrapes were enabled on 2026-09-13 (the Longhorn row above is now history). The Flux-controller and Hubble/Cilium monitor gaps are still open work. Note them; do not try to hand-apply monitors into the cluster — every durable change goes through git.

---

Source of truth: runbooks/targets-down.md. Index: /runbooks/ Records: DECISIONS-LOG.md / commit 2d5171e (Velero had no ServiceMonitor, so three critical backup alerts could never fire); commits 6a7e9d1, b57b3e9 (the false CDI/KubeVirt alerts).