jakman85andClaude Opus 5
b9f3143f42
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
Alert runbooks — home-network-k3s prod
This is the index page served at https://home.keithjackson.dev/runbooks/.
Every alert rule in this platform carries an annotation of the form
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
46 of them, over the 35 names below. Until this directory existed, that path returned HTTP 404 on the live cluster: every alert that reached a human handed them a dead link at the exact moment they needed help. These pages are what makes the link resolve.
How to run the commands in these runbooks
kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang
is not the cluster being down.
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass --context k3s-prod. dev.sh merges BOTH clusters into one kubeconfig and picks a
current-context per invocation, and that default has already changed once — k3s-dev until dev was
decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing
next month; one that relies on the default does not, and the failure is silent. CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner
naming the cluster — read it. Do not run two dev.sh invocations at once; they share one container.
Use fully-qualified resource names, always. Three CRDs on this cluster register the kind Backup —
longhorn.io/v1beta2, postgresql.cnpg.io/v1, velero.io/v1 — and the bare short name resolves to
Longhorn's, which holds zero objects. kubectl get backup -A therefore prints "No resources found" on
a cluster holding 6 Velero and 27 CNPG backups (verified on prod 2026-09-13). Acting on that false
negative is how a CLI quirk becomes real data loss. Write backups.velero.io,
backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io,
volumes.longhorn.io, nodes.longhorn.io.
git push origin main is the deploy; there is no other apply step. Durable changes go through git.
Alerts you should not trust the silence of (measured 2026-09-13)
The Longhorn and cert-manager scrapes were ENABLED on 2026-09-13, so the Longhorn row below is no
longer inert — and re-keying the two robustness rules the same day fixed a SECOND inertness the
scrape had left in place (the metric is one-hot over a state label, not a 0–3 enum). Take the
Longhorn row as history; the rest of this table still holds, and Flux-controller and Hubble/Cilium
rule groups still evaluate against nothing and cannot fire:
| Missing scrape job | Inert alerts |
|---|---|
| Longhorn | LonghornVolumeDegraded, LonghornVolumeFaulted, LonghornDiskAllocationHigh, LonghornDiskUnschedulable |
| cert-manager | CertificateExpiringSoon, CertificateExpiryImminent, CertificateNotReady |
Flux (gotk_*) |
FluxReconciliationFailure, FluxResourceSuspended, ImageUpdateAutomationFailing, ImagePolicyStalled, UpdateHealthGateFailed |
Hubble (hubble_drop_total) |
HubbleDeniedFlowSpike |
etcd (kubeEtcd disabled for k3s, deliberately) |
no etcd fsync alerting exists at all |
Each affected runbook says so on its own page and gives the by-hand equivalent. See targets-down.
The runbooks
Nodes
node-down— a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have stopped posting status while the host stays up (the 2026-09-07 shape).node-io— disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas; this is the alert that would have called the 2026-09-07 outage and did not exist at the time.node-memory— memory pressure. Includes whySwapTotal == 0is not evidence swap is off.node-disk— a node filesystem filling up./datais the one that matters.node-cpu— CPU PSI and run-queue depth, and how to tell CPU contention from I/O contention.
Storage
longhorn-degraded— a volume has lost redundancy. Normal after any reboot, and the rebuild is the expensive part. Freeze/thaw commands here.longhorn-faulted— no usable replica. Data at risk; stop before anything destructive.longhorn-capacity— a node's Longhorn disk running out of room for replicas (allocation, not PVC fill; for PVC fill seepvc-capacity/monitoring-capacity).pvc-capacity— PVC filling up / almost full / predicted to fill. These rules work.monitoring-capacity— Prometheus or Loki running out of disk; don't let the monitor blind itself.
GitOps and updates
flux-reconcile— a Flux resource stuck at Ready=False. Which tier decides the blast radius.flux-suspended— reconciliation turned off and forgotten. Drift becomes silent.update-rollback— a HelmRelease failed its health gate and is rolling back.image-automation— the git write-back engine is failing (feature is disabled here).image-policy-stalled— an ImagePolicy/ImageRepository cannot resolve a tag.
Certificates
cert-issuance— ACME issuance failing. The DNS-01 self-check is the usual cause.cert-expiry— a certificate is expiring. Almost always a late symptom ofcert-issuance.
Networking
vpn-down— the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert cannot see, and what the kill-switch actually does.denied-flows— policy-denied flow spike. The Cilium hairpin/identity regression class.
Backups and DR
backup-failure— a Velero backup failed. The R2 object-tagging incompatibility is documented here.backup-partial— a Velero backup completed with partial failures, and which skips are deliberate.backup-missed— no successful Velero backup in 24h.cnpg-archiving— WAL archiving stuck / no recoverability point. An availability incident, not just a backup one.cnpg-backup-missed— no recent CNPG base backup. Velero does not cover these databases.cnpg-wal-growth—pg_walnot being recycled. Never delete WAL by hand.cnpg-exporter— the CNPG SQL exporter erroring: a blind spot that looks exactly like health.cnpg-primary-spread— two or more CNPG primaries on one node. Blast radius, not an outage, and the rebalance costs ~9s of writes per database — one cluster at a time.
Identity
keycloak-down— the IdP is down. Blast radius, and the break-glass path that still works.keycloak-db— Keycloak's CloudNativePG database is unavailable.keycloak-client-secrets— the client/URL reconciler failing or silent. The 2026-09-06 lockout that every green signal missed.
Virtualisation
kubevirt-control-plane— VM lifecycle operations will fail; running VMs keep running.kubevirt-launch-failed— a VMI reached the Failed phase. The three k3s-specific launch traps.kubevirt-crashloop— a virt-launcher pod restarting repeatedly.cdi-import-stuck— a DataVolume import not progressing. Check node I/O before restarting it.
Editing a runbook
Each page is runbooks/<name>.md in the home-network-k3s-impl repo. Edit it, push to main, and Flux
re-renders the ConfigMap and rolls the two-replica server; kustomize's content hash on the generated
ConfigMap name is what triggers the roll.
Adding a new alert with a runbook_url requires the matching runbooks/<name>.md and its
<name>=<name>.md entry in runbooks/kustomization.yaml. Both are enforced by
conformance/tests/l0-static/alert-runbooks-resolvable, which runs unconditionally in the merge gate —
"the file exists in git" and "the URL resolves in the cluster" are different claims, and it checks both.
Every page must carry the four operator sections (What this means here, First commands, Likely
causes, most likely first, When to escalate or stop), mention dev.sh, and pass --context k3s-prod.
The check enforces that too, because a stub that makes the link resolve while helping nobody is worse than
the 404 it replaced — it looks fixed.
Why this route has no login
Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr)
really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on
:4180 rather than the app port. If
these pages did too, keycloak-down would be unreadable exactly when Keycloak is down,
keycloak-db unreadable when its database is down, and keycloak-client-secrets unreadable during the
precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An
operator who cannot log in is this route's primary reader. The content is operational procedure — metric
names, thresholds, kubectl invocations, topology — and no secrets. See
runbooks/serving/httproute.yaml for the full reasoning, including how forward-auth is actually
enforced here and the instruction to leave this route out of any Gateway-level ext-authz attachment
if one is ever introduced.
The running narrative of real incidents on this cluster is DECISIONS-LOG.md.