jakman85andClaude Opus 5
f663e7c801
`runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd
I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and
`snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on
prod today: both read **5**, and neither name appeared anywhere in this
repository — so nothing declared them and nothing would have restored them.
Fourth instance of config that looked applied and was not.
Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux
reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` —
a hand-applied Setting is exactly how this gap arose and Helm would silently
rewrite it on its next upgrade.
concurrentVolumeBackupRestorePerNodeLimit 5 -> 1
snapshotHeavyTaskConcurrentLimit 5 -> 1
replicaReplenishmentWaitInterval 600 PINNED, unchanged
backupConcurrentLimit / restoreConcurrentLimit 2 / 2 PINNED, unchanged
concurrentAutomaticEngineUpgradePerNodeLimit 0 PINNED, unchanged
nodeDrainPolicy block-if-contains-last-replica PINNED — it was in
force only as Longhorn's OWN DEFAULT, declared in no manifest and in no
rendered ConfigMap, so an upstream default change would have moved it
silently while two runbooks reasoned about drain safety from it.
EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn
--version 1.12.1` (the version this chart range resolves to, per the
HelmRelease's own lastAttemptedRevision) on this branch's spec.values,
substituted as clusters/prod substitutes them, emits all eight into the rendered
`longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE
ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost.
The chain to the Settings is verified read-only on prod — all 115
settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917`
(= that ConfigMap's resourceVersion) with `status.applied: true`, and every key
in the ConfigMap equals its Setting's value. The two targets sit at 5 with the
key absent, i.e. Longhorn defaults, as measured.
`orphanAutoDeletion: true` REMOVED, verified inert two independent ways against
the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed
it `orphanResourceAutoDeletion`, a list, and the chart drops unknown
defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is
NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT
replaced — orphan deletion is extra writes on the constrained partition and buys
only space. Orphan cleanup stays manual by decision; the two runbooks that cite
the key by name are corrected rather than left to go stale.
BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at.
`concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has
no backup target (the `backup-target` Setting does not exist; still the
WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup —
`snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia
file-system backup through velero's node-agent, which this Setting does not
gate; CNPG base backups are barman-level and ungated too. Its only consumer is
an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd
mid-restore. Revisit when a Longhorn backup target lands.
`snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is
generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO
RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to
1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain
at the price of a second heavy writer during the exact rebuild that starved
etcd. The cost: space reclaims serially after a rebuild storm — real here (a
37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by
LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not.
Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a
full 18-replica rebuild after node 1's drive replacement, against a 5% warning
threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one
ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on.
NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration
(`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32,
systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on
2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node
Ready throughout; the role was never run against any cluster, its author
recorded it as unproven with a node-3-first sequenced rollout, and its own
README concedes it does not separate the two tenants. Node-wide blast radius,
low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2).
It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in
September, we would now be told about. Nothing under infra/ansible/ is touched.
ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the
dashboard I/O row are on main already (efc6db6) with runbooks/node-io.md.
Verified live, not by reading the file: both rules loaded in
platform-health.nodes, health ok, state inactive, and
ALERTS{alertname=~"NodeIOPressure.*"} returns no samples.
Runbooks corrected in both directions: longhorn-degraded.md's prominent "DO NOT
RELY ON THAT: it was never applied" block becomes an accurate statement of what
is now declared, what to verify before trusting it, what it cannot do, and what
it costs; longhorn-capacity.md / node-disk.md stop citing a key that no longer
exists; node-io.md gains the one line an operator needs mid-incident.
No new ${...} substitution variables and not one dollar sign added anywhere in
the diff, so neither StrictPostBuildSubstitutions nor the L0 doubled-dollar guard
is affected.
Co-Authored-By: Claude Opus 5 <[email protected]>
Alert runbooks — home-network-k3s prod
This is the index page served at https://home.keithjackson.dev/runbooks/.
Every alert rule in this platform carries an annotation of the form
runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
46 of them, over the 35 names below. Until this directory existed, that path returned HTTP 404 on the live cluster: every alert that reached a human handed them a dead link at the exact moment they needed help. These pages are what makes the link resolve.
How to run the commands in these runbooks
kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang
is not the cluster being down.
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass --context k3s-prod. dev.sh merges BOTH clusters into one kubeconfig and picks a
current-context per invocation, and that default has already changed once — k3s-dev until dev was
decommissioned on 2026-09-13, k3s-prod now. A command that names its context still means the same thing
next month; one that relies on the default does not, and the failure is silent. CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner
naming the cluster — read it. Do not run two dev.sh invocations at once; they share one container.
Use fully-qualified resource names, always. Three CRDs on this cluster register the kind Backup —
longhorn.io/v1beta2, postgresql.cnpg.io/v1, velero.io/v1 — and the bare short name resolves to
Longhorn's, which holds zero objects. kubectl get backup -A therefore prints "No resources found" on
a cluster holding 6 Velero and 27 CNPG backups (verified on prod 2026-09-13). Acting on that false
negative is how a CLI quirk becomes real data loss. Write backups.velero.io,
backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io,
volumes.longhorn.io, nodes.longhorn.io.
git push origin main is the deploy; there is no other apply step. Durable changes go through git.
Alerts you should not trust the silence of (measured 2026-09-13)
The Longhorn and cert-manager scrapes were ENABLED on 2026-09-13, so the Longhorn row below is no
longer inert — and re-keying the two robustness rules the same day fixed a SECOND inertness the
scrape had left in place (the metric is one-hot over a state label, not a 0–3 enum). Take the
Longhorn row as history; the rest of this table still holds, and Flux-controller and Hubble/Cilium
rule groups still evaluate against nothing and cannot fire:
| Missing scrape job | Inert alerts |
|---|---|
| Longhorn | LonghornVolumeDegraded, LonghornVolumeFaulted, LonghornDiskAllocationHigh, LonghornDiskUnschedulable |
| cert-manager | CertificateExpiringSoon, CertificateExpiryImminent, CertificateNotReady |
Flux (gotk_*) |
FluxReconciliationFailure, FluxResourceSuspended, ImageUpdateAutomationFailing, ImagePolicyStalled, UpdateHealthGateFailed |
Hubble (hubble_drop_total) |
HubbleDeniedFlowSpike |
etcd (kubeEtcd disabled for k3s, deliberately) |
no etcd fsync alerting exists at all |
Each affected runbook says so on its own page and gives the by-hand equivalent. See targets-down.
The runbooks
Nodes
node-down— a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have stopped posting status while the host stays up (the 2026-09-07 shape).node-io— disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas; this is the alert that would have called the 2026-09-07 outage and did not exist at the time.node-memory— memory pressure. Includes whySwapTotal == 0is not evidence swap is off.node-disk— a node filesystem filling up./datais the one that matters.node-cpu— CPU PSI and run-queue depth, and how to tell CPU contention from I/O contention.
Storage
longhorn-degraded— a volume has lost redundancy. Normal after any reboot, and the rebuild is the expensive part. Freeze/thaw commands here.longhorn-faulted— no usable replica. Data at risk; stop before anything destructive.longhorn-capacity— a node's Longhorn disk running out of room for replicas (allocation, not PVC fill; for PVC fill seepvc-capacity/monitoring-capacity).pvc-capacity— PVC filling up / almost full / predicted to fill. These rules work.monitoring-capacity— Prometheus or Loki running out of disk; don't let the monitor blind itself.
GitOps and updates
flux-reconcile— a Flux resource stuck at Ready=False. Which tier decides the blast radius.flux-suspended— reconciliation turned off and forgotten. Drift becomes silent.update-rollback— a HelmRelease failed its health gate and is rolling back.image-automation— the git write-back engine is failing (feature is disabled here).image-policy-stalled— an ImagePolicy/ImageRepository cannot resolve a tag.
Certificates
cert-issuance— ACME issuance failing. The DNS-01 self-check is the usual cause.cert-expiry— a certificate is expiring. Almost always a late symptom ofcert-issuance.
Networking
vpn-down— the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert cannot see, and what the kill-switch actually does.denied-flows— policy-denied flow spike. The Cilium hairpin/identity regression class.
Backups and DR
backup-failure— a Velero backup failed. The R2 object-tagging incompatibility is documented here.backup-partial— a Velero backup completed with partial failures, and which skips are deliberate.backup-missed— no successful Velero backup in 24h.cnpg-archiving— WAL archiving stuck / no recoverability point. An availability incident, not just a backup one.cnpg-backup-missed— no recent CNPG base backup. Velero does not cover these databases.cnpg-wal-growth—pg_walnot being recycled. Never delete WAL by hand.cnpg-exporter— the CNPG SQL exporter erroring: a blind spot that looks exactly like health.cnpg-primary-spread— two or more CNPG primaries on one node. Blast radius, not an outage, and the rebalance costs ~9s of writes per database — one cluster at a time.
Identity
keycloak-down— the IdP is down. Blast radius, and the break-glass path that still works.keycloak-db— Keycloak's CloudNativePG database is unavailable.keycloak-client-secrets— the client/URL reconciler failing or silent. The 2026-09-06 lockout that every green signal missed.
Virtualisation
kubevirt-control-plane— VM lifecycle operations will fail; running VMs keep running.kubevirt-launch-failed— a VMI reached the Failed phase. The three k3s-specific launch traps.kubevirt-crashloop— a virt-launcher pod restarting repeatedly.cdi-import-stuck— a DataVolume import not progressing. Check node I/O before restarting it.
Editing a runbook
Each page is runbooks/<name>.md in the home-network-k3s-impl repo. Edit it, push to main, and Flux
re-renders the ConfigMap and rolls the two-replica server; kustomize's content hash on the generated
ConfigMap name is what triggers the roll.
Adding a new alert with a runbook_url requires the matching runbooks/<name>.md and its
<name>=<name>.md entry in runbooks/kustomization.yaml. Both are enforced by
conformance/tests/l0-static/alert-runbooks-resolvable, which runs unconditionally in the merge gate —
"the file exists in git" and "the URL resolves in the cluster" are different claims, and it checks both.
Every page must carry the four operator sections (What this means here, First commands, Likely
causes, most likely first, When to escalate or stop), mention dev.sh, and pass --context k3s-prod.
The check enforces that too, because a stub that makes the link resolve while helping nobody is worse than
the 404 it replaced — it looks fixed.
Why this route has no login
Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr)
really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on
:4180 rather than the app port. If
these pages did too, keycloak-down would be unreadable exactly when Keycloak is down,
keycloak-db unreadable when its database is down, and keycloak-client-secrets unreadable during the
precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An
operator who cannot log in is this route's primary reader. The content is operational procedure — metric
names, thresholds, kubectl invocations, topology — and no secrets. See
runbooks/serving/httproute.yaml for the full reasoning, including how forward-auth is actually
enforced here and the instruction to leave this route out of any Gateway-level ext-authz attachment
if one is ever introduced.
The running narrative of real incidents on this cluster is DECISIONS-LOG.md.