jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
142 lines
8.3 KiB
Markdown
142 lines
8.3 KiB
Markdown
# targets-down — More than 10% of Prometheus scrape targets are down
|
|
|
|
**Alert:** `PrometheusTargetsMissing` (warning, 15m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.monitoring-self`
|
|
**Expression:** `(count(up == 0) / count(up)) > 0.10`
|
|
|
|
## What this means here
|
|
|
|
A tenth of everything Prometheus is supposed to scrape is unreachable. This is a **blind-spot** alert:
|
|
its purpose is to tell you that other alerts may be silent because their data is missing, not because
|
|
the thing they watch is healthy.
|
|
|
|
Live baseline on prod, measured 2026-09-13: **55 targets across 19 jobs** —
|
|
`apiserver`, `cdi-prometheus-metrics`, `coredns`, `keycloak-discovery`, `keycloak-service`,
|
|
`keycloak/keycloak-db`, `kube-prometheus-stack-alertmanager`, `kube-prometheus-stack-grafana`,
|
|
`kube-prometheus-stack-operator`, `kube-prometheus-stack-prometheus`, `kube-state-metrics`, `kubelet`,
|
|
`kubevirt-prometheus-metrics`, `nextcloud-db/nextcloud-db`, `node-exporter`, `oauth2-proxy`,
|
|
`semaphore-db/semaphore-db`, `velero`, `velero/node-agent`. So 10% is roughly **6 targets** — one
|
|
node's worth of exporters, or one CNPG cluster's three pods.
|
|
|
|
**The far more dangerous failure this alert CANNOT see: a target that was never configured.** `up`
|
|
only counts targets Prometheus knows about. A missing ServiceMonitor produces no `up == 0` series and
|
|
no alert at all. This cluster has been bitten by that repeatedly, and these are the known live gaps as
|
|
of 2026-09-13 — every one of them means the corresponding alerts are **inert**:
|
|
|
|
| Missing scrape job | Alerts that cannot fire |
|
|
|---|---|
|
|
| Longhorn | `LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `LonghornDiskAllocationHigh`, `LonghornDiskUnschedulable` |
|
|
| cert-manager | `CertificateExpiringSoon`, `CertificateExpiryImminent`, `CertificateNotReady` |
|
|
| Flux controllers (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
|
|
| Hubble / Cilium (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
|
|
| etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
|
|
|
|
Precedent for how costly this is: Velero shipped with three backup alerts and
|
|
`metrics.serviceMonitor.enabled: false`, so *"a nightly backup could fail — or silently stop running —
|
|
for weeks with nothing to say so."* That ServiceMonitor now exists. The rest of the table does not.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Which targets are down, grouped by job:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
|
|
curl -s -G --data-urlencode "query=up == 0" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "\"job\"|\"instance\"|\"namespace\"" '
|
|
```
|
|
|
|
The scrape-error text, which usually names the cause outright:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
|
|
curl -s "localhost:19090/api/v1/targets?state=active" | python3 -c "
|
|
import json,sys
|
|
for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
|
|
if t[\"health\"]!=\"up\": print(t[\"labels\"].get(\"job\"), t[\"scrapeUrl\"], t[\"lastError\"][:120])"'
|
|
```
|
|
|
|
Which monitors exist at all (the gap this alert cannot see):
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get servicemonitors,podmonitors -A'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. A node is down or NotReady
|
|
One node takes out its `node-exporter` and `kubelet` targets together — roughly the 10% threshold on
|
|
its own. Go to `/runbooks/node-down`; this alert is the echo.
|
|
|
|
### 2. A workload was rolled, scaled to zero, or crashlooped
|
|
Targets follow pods. A CNPG cluster restarting takes three targets with it. Transient; if it persists,
|
|
the workload is the problem, not the monitoring.
|
|
|
|
### 3. A NetworkPolicy blocks the scrape
|
|
Default-deny is the baseline everywhere (R-NET-01) and a new component's monitor will be scraped the
|
|
moment it exists — `serviceMonitorSelectorNilUsesHelmValues: false` means Prometheus selects every
|
|
monitor in the cluster — so a newly added workload can produce a permanently-down target purely
|
|
because nothing allows ingress from `monitoring`. The `lastError` will be a timeout. Fix is an
|
|
explicit allow in the component's CiliumNetworkPolicy, in git.
|
|
|
|
### 4. TLS or auth on the scrape endpoint
|
|
`lastError` will say so (certificate or 401/403). Note a related trap from this cluster: after the
|
|
Cilium upgrade, an in-cluster call to a `*.home` hostname is re-checked against the destination's real
|
|
backend identity, so `toEntities: world` no longer covers it — see `/runbooks/denied-flows`.
|
|
|
|
### 5. The exporter is genuinely broken
|
|
CDI once had a broken HTTP ServiceMonitor that caused a false `CDIOperatorDown`, and
|
|
`KubeVirtControlPlaneDown` once used a non-existent metric with the wrong job label. Both were fixed.
|
|
An exporter that is up but publishing the wrong series is the hardest case — see
|
|
`/runbooks/cnpg-exporter` for the canonical example (the CNPG exporter can be `up=1` while its whole
|
|
`cnpg_pg_stat_*` family is missing).
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Treat this alert as "trust nothing else for now".** Before acting on the *silence* of any other
|
|
alert, confirm its series exists (`count(<metric>)`), because the table above shows five whole rule
|
|
groups are currently evaluating against nothing.
|
|
- Escalate if the down targets are `apiserver` or `kubelet` — that is a control-plane problem, not a
|
|
monitoring one.
|
|
- Adding a missing ServiceMonitor is a git change (a new `ServiceMonitor` next to the component, or a
|
|
chart value). Do not "fix" a blind spot by silencing the alert.
|
|
- The Longhorn and cert-manager scrapes were enabled on 2026-09-13 (the Longhorn row above is now
|
|
history). The Flux-controller and Hubble/Cilium monitor gaps are still open work. Note them; do not
|
|
try to hand-apply monitors into the cluster — every durable change goes through git.
|
|
|
|
---
|
|
Source of truth: `runbooks/targets-down.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` / commit `2d5171e` (Velero had no ServiceMonitor, so three critical backup
|
|
alerts could never fire); commits `6a7e9d1`, `b57b3e9` (the false CDI/KubeVirt alerts).
|