jakman85andClaude Opus 5
44983095e1
Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names, across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute serving that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that reached a human handed them a dead link at the exact moment the documentation is read, and the annotation made it look solved — the same green-surface-over-absent-mechanism class as the import Job that reported success while storing placeholder secrets. No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is made to resolve instead. WHAT LANDS - runbooks/<34>.md — one per name, each grounded in this cluster's real topology and, where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG 38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the 2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12 clients held literal placeholder secrets behind an entirely green cluster, the Velero R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap re-enabling itself across reboots via systemd-gpt-auto-generator. Each page states what the alert means HERE, the first diagnostic commands, likely causes ordered by probability with a remediation each, and when to escalate or stop. - Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss. - Every page states that kubectl only works via `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`, because dev.sh defaults its current-context to k3s-dev. - Pages also record which alert groups are INERT today (measured 2026-09-13: no Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules evaluate against nothing) so nobody reads their silence as health. SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier (dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file. - The service catalog cannot express this route: ingressRoutes[].host is pattern-bound to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is no path field. - Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is broken, and Gitea is an app-tier workload whose own dependencies several of these runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier, so it would need 34 hand-maintained ReplaceFullPath rules. - configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md), so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not load anyway. - The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01 emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the https-apex listener's, so nothing needs substituting. AUTH — PUBLIC, deliberately auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make runbooks/keycloak-down unreadable when Keycloak is down and runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode. An operator who cannot log in is this route's primary reader. Content is operational procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute so the route is left OUT of the Gateway ext-authz attachment when forward-auth enforcement lands. PROVEN — in-cluster probe through the Gateway on prod /runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8 /runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8 /runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8 /runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8 /runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8 /runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8 HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the `/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence. GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04) 363 offline assertions. It checks BOTH halves, because the regression is silent in both directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its configMapGenerator entry is in git, reviewed, merged, and still 404s in the cluster. Also asserts a quality floor so a stub cannot make the link look fixed, and that route/listener/nginx prefix agree with the annotations. Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the generator (1 FAIL), and a 31-byte stub (7 FAIL). Co-Authored-By: Claude Opus 5 <[email protected]>
7.7 KiB
targets-down — More than 10% of Prometheus scrape targets are down
Alert: PrometheusTargetsMissing (warning, 15m)
File: platform/monitoring/alerts.yaml, group platform-health.monitoring-self
Expression: (count(up == 0) / count(up)) > 0.10
What this means here
A tenth of everything Prometheus is supposed to scrape is unreachable. This is a blind-spot alert: its purpose is to tell you that other alerts may be silent because their data is missing, not because the thing they watch is healthy.
Live baseline on prod, measured 2026-09-13: 55 targets across 19 jobs —
apiserver, cdi-prometheus-metrics, coredns, keycloak-discovery, keycloak-service,
keycloak/keycloak-db, kube-prometheus-stack-alertmanager, kube-prometheus-stack-grafana,
kube-prometheus-stack-operator, kube-prometheus-stack-prometheus, kube-state-metrics, kubelet,
kubevirt-prometheus-metrics, nextcloud-db/nextcloud-db, node-exporter, oauth2-proxy,
semaphore-db/semaphore-db, velero, velero/node-agent. So 10% is roughly 6 targets — one
node's worth of exporters, or one CNPG cluster's three pods.
The far more dangerous failure this alert CANNOT see: a target that was never configured. up
only counts targets Prometheus knows about. A missing ServiceMonitor produces no up == 0 series and
no alert at all. This cluster has been bitten by that repeatedly, and these are the known live gaps as
of 2026-09-13 — every one of them means the corresponding alerts are inert:
| Missing scrape job | Alerts that cannot fire |
|---|---|
| Longhorn | LonghornVolumeDegraded, LonghornVolumeFaulted, LonghornVolumeCapacityHigh, LonghornNodeStorageHigh |
| cert-manager | CertificateExpiringSoon, CertificateExpiryImminent, CertificateNotReady |
Flux controllers (gotk_*) |
FluxReconciliationFailure, FluxResourceSuspended, ImageUpdateAutomationFailing, ImagePolicyStalled, UpdateHealthGateFailed |
Hubble / Cilium (hubble_drop_total) |
HubbleDeniedFlowSpike |
etcd (kubeEtcd disabled for k3s, deliberately) |
no etcd fsync alerting exists at all |
Precedent for how costly this is: Velero shipped with three backup alerts and
metrics.serviceMonitor.enabled: false, so "a nightly backup could fail — or silently stop running —
for weeks with nothing to say so." That ServiceMonitor now exists. The rest of the table does not.
Running the commands in this runbook
kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's k3s-oidc context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass --context k3s-prod explicitly. dev.sh merges both clusters into one kubeconfig and
defaults its current-context to k3s-dev, so a command without --context silently answers
about the wrong cluster. (CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... is equivalent.)
Do not run two dev.sh invocations at once — they share one container and interfere.
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup:
longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to
Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io,
settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.
Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for
agents here: every durable change goes through git (git push origin main is the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.
First commands
Which targets are down, grouped by job:
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
curl -s -G --data-urlencode "query=up == 0" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "\"job\"|\"instance\"|\"namespace\"" '
The scrape-error text, which usually names the cause outright:
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
curl -s "localhost:19090/api/v1/targets?state=active" | python3 -c "
import json,sys
for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
if t[\"health\"]!=\"up\": print(t[\"labels\"].get(\"job\"), t[\"scrapeUrl\"], t[\"lastError\"][:120])"'
Which monitors exist at all (the gap this alert cannot see):
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod get servicemonitors,podmonitors -A'
Likely causes, most likely first
1. A node is down or NotReady
One node takes out its node-exporter and kubelet targets together — roughly the 10% threshold on
its own. Go to /runbooks/node-down; this alert is the echo.
2. A workload was rolled, scaled to zero, or crashlooped
Targets follow pods. A CNPG cluster restarting takes three targets with it. Transient; if it persists, the workload is the problem, not the monitoring.
3. A NetworkPolicy blocks the scrape
Default-deny is the baseline everywhere (R-NET-01) and a new component's monitor will be scraped the
moment it exists — serviceMonitorSelectorNilUsesHelmValues: false means Prometheus selects every
monitor in the cluster — so a newly added workload can produce a permanently-down target purely
because nothing allows ingress from monitoring. The lastError will be a timeout. Fix is an
explicit allow in the component's CiliumNetworkPolicy, in git.
4. TLS or auth on the scrape endpoint
lastError will say so (certificate or 401/403). Note a related trap from this cluster: after the
Cilium upgrade, an in-cluster call to a *.home hostname is re-checked against the destination's real
backend identity, so toEntities: world no longer covers it — see /runbooks/denied-flows.
5. The exporter is genuinely broken
CDI once had a broken HTTP ServiceMonitor that caused a false CDIOperatorDown, and
KubeVirtControlPlaneDown once used a non-existent metric with the wrong job label. Both were fixed.
An exporter that is up but publishing the wrong series is the hardest case — see
/runbooks/cnpg-exporter for the canonical example (the CNPG exporter can be up=1 while its whole
cnpg_pg_stat_* family is missing).
When to escalate or stop
- Treat this alert as "trust nothing else for now". Before acting on the silence of any other
alert, confirm its series exists (
count(<metric>)), because the table above shows five whole rule groups are currently evaluating against nothing. - Escalate if the down targets are
apiserverorkubelet— that is a control-plane problem, not a monitoring one. - Adding a missing ServiceMonitor is a git change (a new
ServiceMonitornext to the component, or a chart value). Do not "fix" a blind spot by silencing the alert. - The Longhorn, cert-manager and Flux monitor gaps are known open work. Note them; do not try to hand-apply monitors into the cluster — every durable change goes through git.
Source of truth: runbooks/targets-down.md. Index: /runbooks/
Records: DECISIONS-LOG.md / commit 2d5171e (Velero had no ServiceMonitor, so three critical backup
alerts could never fire); commits 6a7e9d1, b57b3e9 (the false CDI/KubeVirt alerts).