jakman85andClaude Opus 5
79d9ab2c8c
Closes the gap `feat/io-pressure-diagnostic-context` deliberately left open. That
change enabled node-exporter's `--collector.mountstats` and recorded its own honest
limit: "an NFS export that is merely SLOW cannot be distinguished today", because
node_mountstats_* did not exist on prod until the HelmRelease rolled and the L0 guard
alert-metrics-have-scrape-targets rightly refuses a rule on an unscraped metric.
The rollout has landed. Verified on prod 2026-09-15: 1,842 node_mountstats_* series
(~0.4% of a 451,650-series head), six mounts, 426 per-operation series in each of the
five operation families and 6 in each transport family. Every metric referenced below
returns series under `promtool query instant`.
FIVE RULES in platform-health.node-io-cause, each naming its cause in `summary`:
NFSExportServerSlow response_time > 50 ms/op, our op rate > 20/s,
queue_time < 5 ms/op -> the server is slow BECAUSE OF US
NFSExportServerSlowNotOurLoad same, but our op rate 1-20/s -> the server is slow and
the load is NOT OURS. This is the operator's Proxmox
hypothesis made testable.
NFSClientQueueTimeHigh queue_time > 5 ms/op -> WE are saturating the transport
NFSTransportBacklogDeep backlog depth > 8 per send -> even the grown slot table is full
NFSExportMajorTimeouts any major timeout in 15m -> the export is HANGING (critical)
THE DISCRIMINATOR. mountstats splits a slow NFS op into the half the server owns
(response_time) and the half we own (queue_time). High response with low queue means the
SERVER; high queue means US. The two server rules then partition that state by our own op
rate at 20/s, so exactly one of them can fire and it names which reading applies. The
`unless queue_time > 5ms` clause is what licenses reading response_time as a statement
about 192.168.2.102 rather than about us, and it is pinned by fixture 5 (a client queueing
30 ms/op while the server appears to take 60 ms/op: both server rules must stay quiet).
MEASURED BASELINE, prod 2026-09-15, re-measured for this change:
server response time 4.40-4.81 ms/op (30m window), uniform across all six mounts
client queue time 0.013-0.067 ms/op; snapshot maximum 0.333 ms/op
our own op rate 0.019-0.163 ops/s — STATFS keepalive and nothing else
major timeouts 0 on every mount
backlog depth 0 on every mount
slot-table high-water 586 (prod-1), 162 (prod-3), 2 (prod-2)
THREE LIMITS ON THAT BASELINE, recorded in the rules and the runbook rather than papered
over:
(a) the history is MINUTES, not days — the series begin when the DaemonSet rolled
(count_over_time(...[24h]) returned 18, and time() - process_start_time_seconds read
558-561 s on all three nodes), so no rule uses a range longer than 15m;
(b) it is an IDLE baseline — STATFS keepalive only, no READ/WRITE traffic, so the 50 ms
threshold is argued from the link and the array (one outstanding 1 MB read at 50 ms is
20 MB/s) and must be re-cut after the first real media event. The runbook carries the
query;
(c) prod-1's /mnt/Storage/plex and /mnt/Storage/plex/downloads report byte-identical
per-operation counters (0.12443893333333335 on both, to every digit), as do prod-3's two
nextcloud mounts — mounts sharing one NFS superblock share the kernel's RPC statistics.
One slow export on prod-1 therefore raises two alerts. Documented, not aggregated away.
PROOF, not assertion. platform/monitoring/tests/nfs-export-latency.test.yaml: seven fixtures
(measured healthy; healthy under 60 ops/s of real load; server-slow-ours; server-slow-not-ours;
client-backed-up; backlog-deep; export-failing), and every fixture asserts all five alertnames
in the fires or the quiet direction. The measured baseline is pinned as a band, the
shared-superblock duplication is pinned, and both hazards the sibling recorded are asserted:
the PromQL precedence trap (`(ratio) and rate(ops) > 1 > 0.050` returns a series on a healthy
60 ops/s mount; the shipped form returns none) and the op-rate guards (a mount doing half an
operation a second at 1 s/op response, 1 s/op queue and 20 deep in the backlog — 20x, 200x and
2.5x the three thresholds — must stay silent).
MUTATION-CHECKED, 14 mutations, every one of them making the test file FAIL: each of the five
thresholds moved in both directions where the fixtures allow it, both `unless` clauses deleted,
and all four rate guards loosened. The two clause deletions fail with an exp/got alert diff,
not a parse error. No threshold here is pinned only by a promql_expr_test restating it.
Live on prod, all five shipped expressions return NO series while all four of their
sub-expressions return six each — so the silence is a verdict, not missing data.
runbooks/node-io.md: the triage table grows from 10 rows to 15 (per-export server response
time, client queue time, our own demand, major timeouts, backlog depth), with real measured
HEALTHY values and an explicit "not collected" in both incident columns, because the collector
postdates both events and that can never be backfilled. A new four-way verdict table reads rows
11-13 together and maps each combination to the alert that says it.
HONEST REMAINING GAP: these rules rule US out; they cannot rule Proxmox IN. The per-client view
of an export exists only on the server. What would settle it is one read on 192.168.2.102 while
NFSExportServerSlowNotOurLoad is firing — `nfsstat -s` and /proc/fs/nfsd/clients/*/info — which
is out of scope from this repo. Recorded in the runbook.
Co-Authored-By: Claude Opus 5 <[email protected]>
Alert runbooks — home-network-k3s prod
This is the index page served at https://home.keithjackson.dev/runbooks/.
Every alert rule in this platform carries an annotation of the form
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
46 of them, over the 35 names below. Until this directory existed, that path returned HTTP 404 on the live cluster: every alert that reached a human handed them a dead link at the exact moment they needed help. These pages are what makes the link resolve.
How to run the commands in these runbooks
kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang
is not the cluster being down.
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass --context k3s-prod. dev.sh merges BOTH clusters into one kubeconfig and picks a
current-context per invocation, and that default has already changed once — k3s-dev until dev was
decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing
next month; one that relies on the default does not, and the failure is silent. CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner
naming the cluster — read it. Do not run two dev.sh invocations at once; they share one container.
Use fully-qualified resource names, always. Three CRDs on this cluster register the kind Backup —
longhorn.io/v1beta2, postgresql.cnpg.io/v1, velero.io/v1 — and the bare short name resolves to
Longhorn's, which holds zero objects. kubectl get backup -A therefore prints "No resources found" on
a cluster holding 6 Velero and 27 CNPG backups (verified on prod 2026-09-13). Acting on that false
negative is how a CLI quirk becomes real data loss. Write backups.velero.io,
backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io,
volumes.longhorn.io, nodes.longhorn.io.
git push origin main is the deploy; there is no other apply step. Durable changes go through git.
Alerts you should not trust the silence of (measured 2026-09-13)
The Longhorn and cert-manager scrapes were ENABLED on 2026-09-13, so the Longhorn row below is no
longer inert — and re-keying the two robustness rules the same day fixed a SECOND inertness the
scrape had left in place (the metric is one-hot over a state label, not a 0–3 enum). Take the
Longhorn row as history; the rest of this table still holds, and Flux-controller and Hubble/Cilium
rule groups still evaluate against nothing and cannot fire:
| Missing scrape job | Inert alerts |
|---|---|
| Longhorn | LonghornVolumeDegraded, LonghornVolumeFaulted, LonghornDiskAllocationHigh, LonghornDiskUnschedulable |
| cert-manager | CertificateExpiringSoon, CertificateExpiryImminent, CertificateNotReady |
Flux (gotk_*) |
FluxReconciliationFailure, FluxResourceSuspended, ImageUpdateAutomationFailing, ImagePolicyStalled, UpdateHealthGateFailed |
Hubble (hubble_drop_total) |
HubbleDeniedFlowSpike |
etcd (kubeEtcd disabled for k3s, deliberately) |
no etcd fsync alerting exists at all |
Each affected runbook says so on its own page and gives the by-hand equivalent. See targets-down.
The runbooks
Nodes
node-down— a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have stopped posting status while the host stays up (the 2026-09-07 shape).node-io— disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas; this is the alert that would have called the 2026-09-07 outage and did not exist at the time.node-memory— memory pressure. Includes whySwapTotal == 0is not evidence swap is off.node-disk— a node filesystem filling up./datais the one that matters.node-cpu— CPU PSI and run-queue depth, and how to tell CPU contention from I/O contention.
Storage
longhorn-degraded— a volume has lost redundancy. Normal after any reboot, and the rebuild is the expensive part. Freeze/thaw commands here.longhorn-faulted— no usable replica. Data at risk; stop before anything destructive.longhorn-capacity— a node's Longhorn disk running out of room for replicas (allocation, not PVC fill; for PVC fill seepvc-capacity/monitoring-capacity).pvc-capacity— PVC filling up / almost full / predicted to fill. These rules work.monitoring-capacity— Prometheus or Loki running out of disk; don't let the monitor blind itself.
GitOps and updates
flux-reconcile— a Flux resource stuck at Ready=False. Which tier decides the blast radius.flux-suspended— reconciliation turned off and forgotten. Drift becomes silent.update-rollback— a HelmRelease failed its health gate and is rolling back.image-automation— the git write-back engine is failing (feature is disabled here).image-policy-stalled— an ImagePolicy/ImageRepository cannot resolve a tag.
Certificates
cert-issuance— ACME issuance failing. The DNS-01 self-check is the usual cause.cert-expiry— a certificate is expiring. Almost always a late symptom ofcert-issuance.
Networking
vpn-down— the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert cannot see, and what the kill-switch actually does.denied-flows— policy-denied flow spike. The Cilium hairpin/identity regression class.
Backups and DR
backup-failure— a Velero backup failed. The R2 object-tagging incompatibility is documented here.backup-partial— a Velero backup completed with partial failures, and which skips are deliberate.backup-missed— no successful Velero backup in 24h.cnpg-archiving— WAL archiving stuck / no recoverability point. An availability incident, not just a backup one.cnpg-backup-missed— no recent CNPG base backup. Velero does not cover these databases.cnpg-wal-growth—pg_walnot being recycled. Never delete WAL by hand.cnpg-exporter— the CNPG SQL exporter erroring: a blind spot that looks exactly like health.cnpg-primary-spread— two or more CNPG primaries on one node. Blast radius, not an outage, and the rebalance costs ~9s of writes per database — one cluster at a time.
Identity
keycloak-down— the IdP is down. Blast radius, and the break-glass path that still works.keycloak-db— Keycloak's CloudNativePG database is unavailable.keycloak-client-secrets— the client/URL reconciler failing or silent. The 2026-09-06 lockout that every green signal missed.
Virtualisation
kubevirt-control-plane— VM lifecycle operations will fail; running VMs keep running.kubevirt-launch-failed— a VMI reached the Failed phase. The three k3s-specific launch traps.kubevirt-crashloop— a virt-launcher pod restarting repeatedly.cdi-import-stuck— a DataVolume import not progressing. Check node I/O before restarting it.
Editing a runbook
Each page is runbooks/<name>.md in the home-network-k3s-impl repo. Edit it, push to main, and Flux
re-renders the ConfigMap and rolls the two-replica server; kustomize's content hash on the generated
ConfigMap name is what triggers the roll.
Adding a new alert with a runbook_url requires the matching runbooks/<name>.md and its
<name>=<name>.md entry in runbooks/kustomization.yaml. Both are enforced by
conformance/tests/l0-static/alert-runbooks-resolvable, which runs unconditionally in the merge gate —
"the file exists in git" and "the URL resolves in the cluster" are different claims, and it checks both.
Every page must carry the four operator sections (What this means here, First commands, Likely
causes, most likely first, When to escalate or stop), mention dev.sh, and pass --context k3s-prod.
The check enforces that too, because a stub that makes the link resolve while helping nobody is worse than
the 404 it replaced — it looks fixed.
Why this route has no login
Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr)
really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on
:4180 rather than the app port. If
these pages did too, keycloak-down would be unreadable exactly when Keycloak is down,
keycloak-db unreadable when its database is down, and keycloak-client-secrets unreadable during the
precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An
operator who cannot log in is this route's primary reader. The content is operational procedure — metric
names, thresholds, kubectl invocations, topology — and no secrets. See
runbooks/serving/httproute.yaml for the full reasoning, including how forward-auth is actually
enforced here and the instruction to leave this route out of any Gateway-level ext-authz attachment
if one is ever introduced.
The running narrative of real incidents on this cluster is DECISIONS-LOG.md.