home-network-k3s-impl/runbooks/targets-down.md

# targets-down — More than 10% of Prometheus scrape targets are down

**Alert:** `PrometheusTargetsMissing` (warning, 15m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.monitoring-self`
**Expression:** `(count(up == 0) / count(up)) > 0.10`

## What this means here

A tenth of everything Prometheus is supposed to scrape is unreachable. This is a **blind-spot** alert:
its purpose is to tell you that other alerts may be silent because their data is missing, not because
the thing they watch is healthy.

Live baseline on prod, measured 2026-09-13: **55 targets across 19 jobs** —
`apiserver`, `cdi-prometheus-metrics`, `coredns`, `keycloak-discovery`, `keycloak-service`,
`keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, `kube-prometheus-stack-grafana`,
`kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, `kube-state-metrics`, `kubelet`,
`kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, `node-exporter`, `oauth2-proxy`,
`semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. So 10% is roughly **6 targets** — one
node's worth of exporters, or one CNPG cluster's three pods.

**The far more dangerous failure this alert CANNOT see: a target that was never configured.** `up`
only counts targets Prometheus knows about. A missing ServiceMonitor produces no `up == 0` series and
no alert at all. This cluster has been bitten by that repeatedly, and these are the known live gaps as
of 2026-09-13 — every one of them means the corresponding alerts are **inert**:

| Missing scrape job | Alerts that cannot fire |
|---|---|
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornVolumeCapacityHigh`, `LonghornNodeStorageHigh` |
| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` |
| Flux controllers (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
| Hubble / Cilium (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |

Precedent for how costly this is: Velero shipped with three backup alerts and
`metrics.serviceMonitor.enabled: false`, so *"a nightly backup could fail — or silently stop running —
for weeks with nothing to say so."* That ServiceMonitor now exists. The rest of the table does not.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
defaults its current-context to k3s-**dev**, so a command without `--context` silently answers
about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.)
Do not run two `dev.sh` invocations at once — they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

Which targets are down, grouped by job:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=up == 0" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "\"job\"|\"instance\"|\"namespace\"" '
```

The scrape-error text, which usually names the cause outright:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s "localhost:19090/api/v1/targets?state=active" | python3 -c "
import json,sys
for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
    if t[\"health\"]!=\"up\": print(t[\"labels\"].get(\"job\"), t[\"scrapeUrl\"], t[\"lastError\"][:120])"'
```

Which monitors exist at all (the gap this alert cannot see):

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get servicemonitors,podmonitors -A'
```

## Likely causes, most likely first

### 1. A node is down or NotReady
One node takes out its `node-exporter` and `kubelet` targets together — roughly the 10% threshold on
its own. Go to `/runbooks/node-down`; this alert is the echo.

### 2. A workload was rolled, scaled to zero, or crashlooped
Targets follow pods. A CNPG cluster restarting takes three targets with it. Transient; if it persists,
the workload is the problem, not the monitoring.

### 3. A NetworkPolicy blocks the scrape
Default-deny is the baseline everywhere (R-NET-01) and a new component's monitor will be scraped the
moment it exists — `serviceMonitorSelectorNilUsesHelmValues: false` means Prometheus selects every
monitor in the cluster — so a newly added workload can produce a permanently-down target purely
because nothing allows ingress from `monitoring`. The `lastError` will be a timeout. Fix is an
explicit allow in the component's CiliumNetworkPolicy, in git.

### 4. TLS or auth on the scrape endpoint
`lastError` will say so (certificate or 401/403). Note a related trap from this cluster: after the
Cilium upgrade, an in-cluster call to a `*.home` hostname is re-checked against the destination's real
backend identity, so `toEntities: world` no longer covers it — see `/runbooks/denied-flows`.

### 5. The exporter is genuinely broken
CDI once had a broken HTTP ServiceMonitor that caused a false `CDIOperatorDown`, and
`KubeVirtControlPlaneDown` once used a non-existent metric with the wrong job label. Both were fixed.
An exporter that is up but publishing the wrong series is the hardest case — see
`/runbooks/cnpg-exporter` for the canonical example (the CNPG exporter can be `up=1` while its whole
`cnpg_pg_stat_*` family is missing).

## When to escalate or stop

- **Treat this alert as "trust nothing else for now".** Before acting on the *silence* of any other
  alert, confirm its series exists (`count(<metric>)`), because the table above shows five whole rule
  groups are currently evaluating against nothing.
- Escalate if the down targets are `apiserver` or `kubelet` — that is a control-plane problem, not a
  monitoring one.
- Adding a missing ServiceMonitor is a git change (a new `ServiceMonitor` next to the component, or a
  chart value). Do not "fix" a blind spot by silencing the alert.
- The Longhorn, cert-manager and Flux monitor gaps are known open work. Note them; do not try to
  hand-apply monitors into the cluster — every durable change goes through git.

---
Source of truth: `runbooks/targets-down.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` / commit `2d5171e` (Velero had no ServiceMonitor, so three critical backup
alerts could never fire); commits `6a7e9d1`, `b57b3e9` (the false CDI/KubeVirt alerts).